Audit Logs

• Cloud Audit Logs help answer the question, "Who did what, where, and when?”

  It maintains four audit logs for each Google Cloud project, folder, and organization:

  • Admin Activity audit logs
  • Data Access audit logs
  • System Event audit logs
  • Policy Denied audit logs

  All Google Cloud services will eventually provide audit logs
• Admin Activity audit logs:

  • Admin Activity audit logs contain log entries for API calls or other administrative actions that modify the configuration or metadata of resources.
  • For example, these logs record when users create VM instances or change Identity and Access Management permissions.
  • They are always on, are retained for 400 days, and are available at no charge.
  • To view these logs, you must have the IAM role Logging/Logs Viewer or Project/Viewer.
• System Event audit logs:

  • System Event audit logs contain log entries for Google Cloud administrative actions that modify the configuration of resources.
  • System Event audit logs are generated by Google systems; they are not driven by direct user action.
  • They are always enabled, free, and retained for 400 days.
  • To view these logs, you must have the IAM role Logging/Logs Viewer or Project/Viewer.
• Data Access audit logs:

  • Data Access audit logs contain API calls that read the configuration or metadata of resources. Also, user-driven API calls that create, modify, or read user-provided resource data.
  • Data Access audit logs don’t record the data-access operations on resources that are publicly shared (available to All Users or All Authenticated Users). Data Access audit logs also don’t record the data-access operations on resources that can be accessed without logging into Google Cloud.
• Data Access audit logs:

  • They are disabled by default (except for BigQuery), and when enabled, the default retention is 30 days.
  • To view these logs, you must have the IAM roles Logging/Private Logs Viewer or Project/Owner.
• Policy Denied audit logs:

  • When a security policy is violated, this type of audit logs records when access to a user or service account is denied by Google Cloud service.
  • Policy Denied audit logs are generated by default, and your Google Cloud project is charged for the log storage.
  • You can't disable Policy Denied audit logs. However, you can use exclusion filters to prevent Policy Denied audit logs from being ingested and stored in Cloud Logging.
• To view and filter audit logs:

  1. Navigate to Logs Explorer
  2. Filter by using the Log name drop-down menu. Note: typing cloud audit into the filter box is frequently quicker than scrolling

  • You can even use the query builder to filter audit logs. This query is auto populated in the query section when using the UI.
• Access Transpartency Logs

  Whether it's a hardware support engineer or a rep working on a ticket, having dedicated experts manage parts of the infrastructure is a key benefit of operating in Google Cloud.

  • Very similar to Cloud Audit logs, Access Transparency logs help by providing logs of accesses to your data by human Googlers (as opposed to automated systems).
  • Enterprises with appropriate support packages can enable the logs and receive the log events in near-real time. The log events are surfaced through the APIs, Cloud Logging, and Security Command Center.
• Access Transparency logs give you different information than Cloud Audit Logs. Cloud Audit Logs record the actions that members of your Google Cloud organization have taken in your Google Cloud resources, whereas Access Transparency logs record the actions taken by Google personnel.
• Data Access audit logs can be enabled at various levels in the resource hierarchy. These levels include:

  • Organization
  • Folder
  • Project
  • Resources
  • Billing accounts

  You can also exempt principals from recording data access logs.
• • The final configuration of Data Access audit logs is the union of the configurations. For example, at a project level, you can enable logs for a Google Cloud service. But you can't disable logs for a Google Cloud service that is enabled in a parent organization or folder.
  • The added logging does add to the cost, currently: $0.50 per gigabyte for ingestion.
• • Data Access audit logs are disabled by default, for everything but BigQuery. They may be enabled and configured at the organization, folder, project, or service level. You can control what type of information is kept in the audit logs.
• There are three types of Data Access audit logs information:

  • Admin-read: Records operations that read metadata or configuration information. For example, you looked at the configurations for your bucket.
  • Data-read: Records operations that read user-provided data. For example, you listed files and then downloaded one from Cloud Storage.
  • Data-write: Records operations that write user-provided data. For example, you created a new Cloud Storage file.
• • You can exempt specific users or groups from having their data accesses recorded. This functionality comes in handy when you want reduce the cost and noise associated with the volume of logs that are not of your interest. Data access audit logs can be of high volume, so cost associated is directly proportional to the volume of data logs.
• You can also use the Google Cloud CLI or the API to enable Data Access audit logs.

  1. If you're using the gcloud CLI frequently, the easiest way is to get the current IAM policies, as seen in step 1, and write them to a file.
  2. Then you can edit the /tmp/policy.yaml file to add or edit the auditLogConfigs. You can also add the log details per service, like this example is enabling logging for Cloud Run. You can even enable logging on all services.
  3. Then, as seen in step 3, you would set that as the new IAM policy.
• • Every audit log entry in Cloud Logging is an object of type LogEntry. What distinguishes an audit log entry from other log entries is the protoPayload field, which contains an AuditLog object that stores the audit logging data.
  • Note the log name, which tells us that we’re looking at an example from Data Access audit logs.
  • Identify the principal generating log by looking at the principalEmail.
  • The operation field only exists for large or long-running audit log entries.
  • Google has a standard list of official service names. You can use this list as a handy reference.
• • On this slide, you can tell we’re looking at a query that was run in BigQuery. If you expanded the serviceData field, you could actually see the query itself. So, when someone at your organization runs that unexpected, $40,000 query, you can figure out who ran it and what the query was. Then you can go learn more about price controls and BigQuery.

  
• • Like anything in the cloud, start by planning first. Spend time and create a solid plan for Data Access audit logs. Think organization, folder, then project. Like most organizations, some of your projects will be very specialized, but usually, they do break down into common organizational types. Then, create a test project and experiment to see if the logging works the way you expect. Then roll out the plan, and don't forget automation (Infrastructure as Code, coming soon).
• • Remember that Data Access audit logs can be enabled as high as the organization. The pro would be detailed information on exactly who accessed, edited, and deleted what, and when. The con is that Data Access logs can grow to be large, and are billed per gigabyte. This also results in higher Queries Per Second based on the number of data access requests.
• • Infrastructure as Code (IaC) is essentially the process of automating the creation and modifications to your infrastructure using a platform. The platform supports configuration files, which can be put through a CI/CD pipeline, like with code. Terraform is an open source package from HashiCorp or paid for enterprise version. It isn't hosted directly in Google Cloud, though it’s installed by default in Cloud Shell. State management is a decision point for your organization. It can be remote or local.
• IaC:

  • Remote options include using Cloud Storage or Terraform Cloud. Local storage involves setting up something local to your organization, or using the pay-to-use HashiCorp Enterprise service. Audit logs also keep you informed of the resources provisioned using an IaC tool. It is really useful if you want to control log sink filters at the project level. With terraform you can set default include/exclude filters and have them applied to every project.
• To manage your Google Cloud organization's logs, you can aggregate them from across your organization into a single Cloud Logging bucket. It is recommended to create user-defined buckets to centralize or subdivide your log storage. Based on compliance and usage requirements, customize your logs storage by choosing where your logs are stored and defining the data retention period. Some organization might have latency, compliance, and availability requirement in specific regions. Configure a default storage location to automatically apply a region in which buckets are create for log data.
• Aggregate and store your organization's logs

  • By default, Cloud Logging encrypts customer content stored at rest. Your organization might have advanced encryption requirements that the default encryption at rest doesn't provide. To meet your organization's requirements, instead of Google managing the key encryption keys that protect your data, configure customer-managed encryption keys (CMEK) to control and manage your own encryption.
• Plan and configure exports

  • Start by deciding what, if anything, you will export from Aggregated Exports at the organization level. Next, decide what options you will use, project by project, folder by folder, and so on. Then, carefully consider your filters—both what they leave in, and what they leave out. Filters apply to all logging, not just to exports. Lastly, carefully consider what, if anything, you will fully exclude from logging. Remember that excluded entries will be gone forever.

  
• Principle of least privilege

  • Side-channel leakage of data through logs is a common issue. You need to be careful who gets what kind of access, to which logs. Are you monitoring project by project, or are you selectively grouping work projects into higher-level monitored projects?
  • Use appropriate IAM controls on both Google Cloud-based and exported logs, only allowing the minimal access required to get the job done. Especially scrutinize the Data Access audit log permissions, as they will often contain Personally Identifiable Information (PII).
• Configure log views

  • Log buckets store logs, including audit logs. Log views control access to logs in a log bucket. Custom log views can be created to control access to logs from specific projects or users. This helps protect sensitive data and ensures only authorized users have access.
• By job: CTO: resourcemanager.organizationAdmin, so they can assign permissions to the security team and service accounts. The CTO can then give the security team logging.viewer so they can view the Admin Activity audit logs. Also, logging.privateLogViewer, so they can view the Data Access audit logs. The view permissions are assigned at the organization level, so they are global. Access control to data exported to Cloud Storage or BigQuery will be secured selectively with IAM.
• You might also want to explore Cloud DLP to redact the PII. Data in the Data Access audit logs is deemed as personally identifiable information (PII) for this organization. Integrating the application with Cloud DLP gives the ability to redact sensitive PII data when viewing Data Access logs whether they are in the Data Access audit logs or from the historical archive in Cloud Storage.
• Security team, same:

  • logging.viewer, logging.privateLogViewer

  Dev team:

  • logging.viewer at folder level
  • See Admin Activity audit logs by dev projects in folder.

  Dev team:

  • logging.privateLogViewer at folder
  • See Data Access audit logs.

  Use Cloud Storage or BigQuery IAM to control access to exported logs:

  • Providing a Dashboard might be helpful.
• Scenarios: External Auditors:

  • For external auditors, provide pre-created dashboards where possible. If they need broad access, you might make them logging.viewer at the org level. For BigQuery, they could be bigquery.dataViewer on the exported dataset. For Cloud Storage, again, you could use IAM, but also remember the temporary access URLs that Cloud Storage supports.