Logging & Analysis

• • Cloud Logging allows you to store, search, analyze, monitor, and alert on log data and events from Google Cloud. Cloud Logging is a fully managed service that performs at scale and can ingest application and system log data from thousands of VMs. Even better, you can analyze all that log data in real time. Finally we explore a new feature, Log analytics.
• Cloud Logging helps to:

  • Gather data from various workloads: Gathers the information required to troubleshoot and understand the workload and application needs.
  • Analyze large volumes of data: Tools like Error Reporting, Log Explorer, and Log Analytics let you focus from large sets of data.
  • Route and store logs: Route your logs to the region or service of your choice for additional compliance or business benefits.
  • Get Compliance Insights: Leverage audit and app logs for compliance patterns and issues.
• Cloud Logging Architecture:

  
• Cloud Logging architecture consists of the following components:

  • Log Collections: These are the places where log data originates. Log sources can be Google Cloud services, such as Compute Engine, App Engine, and Kubernetes Engine, or your own applications.
  • Log Routing: The Log Router is responsible for routing log data to its destination. The Log Router uses a combination of inclusion filters and exclusion filters to determine which log data is routed to each destination.
• Log sinks: Log sinks are destinations where log data is stored. Cloud Logging supports a variety of log sinks, including:

  • ○  Cloud Logging log buckets: These are storage buckets that are specifically designed for storing log data.
  • ○  Cloud Pub/Sub topics: These topics can be used to route log data to other services, such as third-party logging solutions.
  • ○  BigQuery: This is a fully-managed, petabyte-scale analytics data warehouse that can be used to store and analyze log data.
  • ○  Cloud Storage buckets: Provides storage of log data in Cloud Storage. Log entries are stored as JSON files.
• Cloud Logging architecture consists of the following components:

  • Log Analysis: Cloud Logging provides several tools to analyze logs.
  • Logs Explorer is optimized for troubleshooting use cases with features like log streaming, a log resource explorer and a histogram for visualization.
  • Error Reporting help users react to critical application errors through automated error grouping and notifications.
  • Logs-based metrics, dashboards and alerting provide other ways to understand and make logs actionable.
  • Log Analytics feature expands the toolset to include ad hoc log analysis capabilities.
• • Platform logs are logs written by Google Cloud services. These logs can help you debug and troubleshoot issues, and help you better understand the Google Cloud services you're using. For example, VPC Flow Logs record a sample of network flows sent from and received by VM instances.
• Component logs are similar to platform logs, but they are generated by Google-provided software components that run on your systems. For example, GKE provides software components that users can run on their own VM or in their own data center. Logs are generated from the user's GKE instances and sent to a user's Cloud project. GKE uses the logs or their metadata to provide user support.
• Security logs help you answer "who did what, where, and when.":Cloud Audit Logs provide information about administrative activities and accesses within your Google Cloud resources.

  • Access Transparency provides you with logs of actions taken by Google staff when accessing your Google Cloud content.
• User-written logs are logs written by custom applications and services. Typically, these logs are written to Cloud Logging by using one of the following methods:

  • Ops Agent
  • Cloud Logging API
  • Cloud Logging client libraries
• Multi-cloud logs and Hybrid-cloud logs: These refer to logs from other cloud providers like Microsoft Azure and logs from on-premises infrastructure.
• • You can programmatically send application logs to Cloud Logging by using client libraries or by using one of the Logging agents. When you can't use them, or when you only want to experiment, you can write logs by using the gcloud logging write command or by sending HTTP commands to the Cloud Logging API endpoint entries.write.
• If you're using one of the agents, then your applications can use any established logging framework to emit logs. For example, in container environments like Google Kubernetes Engine or Container-Optimized OS, the agents automatically collect logs from stdout and stderr.
• • On virtual machines (VMs), the agents collect logs from known file locations or logging services like the Windows Event Log, journald, or syslogd.
  • Serverless compute services like Cloud Run and Cloud Functions, include simple runtime logging by default. Logs written to stdout or stderr will appear automatically in the Google Cloud console.
• Log Routing & Storage

  
• log router: Entries are passed through the API and fed to Log Router. Log Router is optimized for processing streaming data, reliably buffering it, and sending it to any combination of log storage and sink (export) locations.

  • By default, log entries are fed into one of the default logs storage buckets. Exclusion filters might be created to partially or totally prevent this behavior.
• Log sinks run in parallel with the default log flow and might be used to direct entries to external locations.

  Log storage: Locations might include additional Cloud Logging buckets, Cloud Storage, BigQuery, Pub/Sub, or external projects.

  Inclusion and exclusion filters can control exactly which logging entries end up at a particular destination, and which are ignored completely.

  • Cloud Storage buckets are different storage entities than Cloud Logging buckets.
• For each Google Cloud project, Logging automatically creates two logs buckets: _Required and _Default, and corresponding log sinks with the same names. All logs generated in the project are stored in one of these two locations

  
• • _Required: This bucket holds Admin Activity audit logs, System Event audit logs, and Access Transparency logs, and retains them for 400 days. You aren't charged for the logs stored in _Required, and the retention period of the logs stored here cannot be modified. You cannot delete or modify this bucket.
• • _Default: This bucket holds all other ingested logs in a Google Cloud project, except for the logs held in the _Required bucket. Standard Cloud Logging pricing applies to these logs. Log entries held in the _Default bucket are retained for 30 days, unless you apply custom retention rules. You can't delete this bucket, but you can disable the _Default log sink that routes logs to this bucket.
• • You can also use the gcloud CLI to adjust the retention:gcloud beta logging buckets update _Default --location=global--retention-days=[RETENTION_DAYS]
• The Logs Storage page displays a summary of statistics for the logs that your project is receiving, including:

  • Current total volume: The amount of logs your project has received since the first date of the current month.
  • Previous month volume: The amount of logs your project received in the last calendar month.
  • Projected volume by EOM: The estimated amount of logs your project will receive by the end of the current month, based on current usage.

  
• Logging Flow

  
• • Log Router sinks can be used to forward copies of some or all of your log entries to non-default locations. Use cases include storing logs for extended periods, querying logs with SQL, and access control.
• There are several log sink locations, depending on need:

  • Cloud Logging bucket works well to help pre-separate log entries into a distinct log storage bucket.
  • BigQuery dataset allows the SQL query power of BigQuery to be brought to bear on large and complex log entries.
  • Cloud Storage bucket is a simple external Cloud Storage location, perhaps for long-term storage or processing with other systems.
• There are several log sink locations, depending on need:

  • Pub/Sub topic can export log entries to message handling third-party applications or systems created with code and running somewhere like Dataflow or Cloud Functions.
  • Splunk is used to integrate logs into existing Splunk-based system.
  • The Other project option is useful to help control access to a subset of logentries.
• • The process for creating log sinks mimics that of creating log exclusions.It involves writing a query that selects the log entries you want to export in Logs
  • Use Logs Explorer to build a query that selects the logs you want to exclude. Save the query to use when building the exclusion.
  • Explorer, and choosing a destination of Cloud Storage, BigQuery, or Pub/Sub. The query and destination are held in an object called a sink.
  • Sinks can be created in Google Cloud projects, organizations, folders, and billing accounts.
• Use the Log Explorer query to create an exclusion filter that filters the unwanted entries out of the sink. Give the exclusion a name and add the filter for log entries to exclude.

  • It might be helpful to leave some representative events, depending on the exclusion.
  • Create the exclusion and it will go into effect immediately. Use the Navigation menu to initiate editing of that entity.
  • Take care here, because excluded log events will be lost forever.
• • Log Sink pipelines targeting Cloud Storage tend to work best when your needs align with Cloud Storage strengths. For example, long-term retention, reduced storage costs, and configurable object lifecycles.
  • Cloud Storage features include automated storage class changes, auto-delete, and guaranteed retention.
• • Aggregated sinks allow for easy exporting of logging entries without a one-to-one setup. The sink destination can be any of the destinations discussed up to now.
  • Project Level Sinks, Folder Level Sinks, Organizational Level Sinks
• There are three available Google Cloud Logging aggregation levels.

  • We've discussed a project-level log sink. It exports all the logs for a specific project and a log filter can be specified in the sink definition to include/exclude certain log types.
  • A folder-level log sink aggregates logs on the folder level and can include logs from children resources (subfolders, projects).
  • And for a global view, an organization-level log sink can aggregate logs on the organization level and can also include logs from children resources (subfolders, projects).
• One of the steps in security log analytics workflow is to create aggregate sinks and route those logs to a single destination depending on the choice of security analytics tool, such as Log Analytics, BigQuery, Chronicle, or a third-party security information and event management (SIEM) technology. Logs logs are aggregated from your organization, including any contained folders, projects, and billing accounts.
• Log Entry Fields

  
• The Logs Explorer interface lets you retrieve logs, parse and analyze log data, and refine your query parameters. The Logs Explorer contains the following panes:

  1. Action Toolbar:

  • Action toolbar to refine logs to projects or storage views, share a link, and learn about Logs Explorer.

  1. Query Pane:

  • Query pane is where you can build queries, view recently viewed and saved queries, and more.

  1. Results Toolbar:

  • This can be used to quickly show or hide logs and histogram pane and create a log-based metric or alert. Jump to now option helps query and view the current time results.
• The Logs Explorer interface lets you retrieve logs, parse and analyze log data, and refine your query parameters. The Logs Explorer contains the following panes:

  1. Query Results:

  • Details of results with a summary and timestamp that helps troubleshoot further.

  1. Log Fields:

  • Log fields pane is used to filter your options based on various factors such as resource type, log name, project ID, etc.

  1. Histogram:

  • Histogram is where the query result is visualized with histogram bars, where each bar is a time range and is color-coded based on severity.
• • Ultimately, it’s the query that selects the entries displayed by Logs Explorer. Queries may be created directly with the Logging Query Language (LQL), using the drop-down menus, the logs field explorer, or by clicking fields in the results themselves. Start with what you know about the entry you’re trying to find. If it belongs to a resource, a particular log file, or has a known severity, use the query builder drop-down menus.
• Logs Explorer:

  • Resource:
  • Lets you specify resource.type. You can select a single resource at a time to add to the Query builder. Entries use the logical operator AND.
  • Log Name:
  • Lets you specify logName. You can select multiple log names at once to add to the Query builder. When selecting multiple entries, the logical operator OR is used.
  • Severity:
  • Lets you specify severity. You can select multiple severity levels at once to add to the Query builder. When selecting multiple entries, the logical operator OR is used.
• Logs Explorer:

  • Simple Text Search:
  • Lets you use a simple text search box for global text searches.
  • Advanced Query:
  • Lets you toggle to show/hide the Logging query language for the query.
  • Date/Time Picker:
  • Lets you pick the date/time range in the query builder.
  • Dropdown Selectors:
  • Lets you prominently display the resource, logName, and severity dropdown selectors.
  • Default Summary Fields:
  • Lets you disable default summary fields for a more basic log view.
• Logs Explorer

  
• Logs Explorer:

  Advanced queries support multiple comparison operators, as seen here:

  • Equal and Not Equal Operators:
  • The equal and not equal operators help filter values that match or do not match a value assigned to a field name. These are useful when you search for a specific resource type or ID that you want to evaluate.
  • Numeric Ordering Operators:
  • The numeric ordering operators are handy when searching for logs filtering a timestamp or duration.