Google Cloud Storage Logs

ยท 579 words ยท 3 minute read

Google Cloud Storage is the equivalent of AWS S3 on GCP. This is the Google Cloud’s blob object storage service.

Inside each bucket, the Observability tab shows key metrics about the bucket, such as request rates, network error and client errors. However, detailed logs are not enabled by default and must be manually activated.

The two types of GCS logs ๐Ÿ”—

GCS offers two types of logs. You must enable the correct log type depending on the use case.

  1. Audit logs keeps log of read, write actions performed by GCS users. This log can be enabled easily using the GCP console UI.
  2. Usage and storage logs keep logs of all read and write requests. This is suitable for tracking log of a public bucket. However, this log needs to be set up using the terminal as of writing this article.

Enabling audit logs for Google Cloud Storage ๐Ÿ”—

Go to the Audit logs page. Search for Storage. Enable the logs needed, and click save.

GCP GCS Audit Logs

Reading Audit logs for Google Cloud Storage ๐Ÿ”—

Go to Logs, search and filter by Storage. Learn more about viewing audit logs here

Enabling usage and storage logs for Google Cloud Storage ๐Ÿ”—

Both usage and storage logs are generated for a bucket when log delivery is enabled. Below are the steps to enable log delivery for a Google Cloud Storage bucket.

Step 1: Create a bucket for storing logs ๐Ÿ”—

This command will create a storage bucket.

gsutil mb gs://example-logs-bucket

Step 2: Update permissions ๐Ÿ”—

Cloud storage needs the role roles/storage.objectCreator to be able to create objects in the bucket.

gsutil iam ch group:cloud-storage-analytics@google.com:objectCreator gs://example-logs-bucket

Step 3: Enable logging ๐Ÿ”—

Enable logging using the logging command

gsutil logging set on -b gs://example-logs-bucket [-o log_object_prefix ] gs://example-bucket

Step 4: Confirm logging status ๐Ÿ”—

The following command will show whether the logging is enabled or not, and where the log files are stored.

gsutil logging get gs://example-bucket

When logging is set up properly, it will respond with the details.

{"logBucket": "example-logs-bucket", "logObjectPrefix": "log_object_prefix"}

When logging is not enabled, the following is returned.

gs://example-bucket/ has no logging configuration.

Analyze logs in BigQuery ๐Ÿ”—

The logs can be uploaded to BigQuery for faster analysis.

Step 1: Create a dataset ๐Ÿ”—

This command will create a dataset in BigQuery.

bq mk storagelogs

Step 2: Get schema ๐Ÿ”—

Data schema are available here:

  1. Usage logs schema
  2. Storage logs schema

Download the schema files in the working directory.

Step 3: Get the list of log files ๐Ÿ”—

Run this on terminal to create a list of all log files stored in the bucket.

gsutil ls gs://example-logs-bucket > logs_files.txt

Step 4: Upload the files to BigQuery ๐Ÿ”—

This bash script will loop over all logs files and upload them to BigQuery.

Please make sure that all log files are of the same type. If, for example, the log_files.txt file created in the previous step contains both usage and storage logs, make sure to run this step twice - once for each type of logs.

for log in $(cat logs_files.txt); do
bq load --skip_leading_rows=1 --replace=false storagelogs.usage $log ./usage_schema_v0.json
done

Step 5: Analyze in BigQuery ๐Ÿ”—

Enable bq shell

bq shell

Run Query

SELECT sc_status, COUNT(*) AS counted FROM storagelogs.usage GROUP BY 1

Exit the BigQuery shell

exit

Disable usage and storage logs ๐Ÿ”—

Run this command in terminal to disable logging.

gsutil logging set off gs://example-bucket

Validate that it is disabled.

gsutil logging get gs://example-bucket

If logging is disabled, this is returned.

gs://example-bucket/ has no logging configuration.

Further reading ๐Ÿ”—

Usage logs and Access logs of Google Cloud Storage