Google Cloud Storage is the equivalent of AWS S3 on GCP. This is the Google Cloud’s blob object storage service.
Inside each bucket, the Observability tab shows key metrics about the bucket, such as request rates, network error and client errors. However, detailed logs are not enabled by default and must be manually activated.
The two types of GCS logs ๐
GCS offers two types of logs. You must enable the correct log type depending on the use case.
- Audit logs keeps log of read, write actions performed by GCS users. This log can be enabled easily using the GCP console UI.
- Usage and storage logs keep logs of all read and write requests. This is suitable for tracking log of a public bucket. However, this log needs to be set up using the terminal as of writing this article.
Enabling audit logs for Google Cloud Storage ๐
Go to the Audit logs page. Search for Storage. Enable the logs needed, and click save.
Reading Audit logs for Google Cloud Storage ๐
Go to Logs, search and filter by Storage. Learn more about viewing audit logs here
Enabling usage and storage logs for Google Cloud Storage ๐
Both usage and storage logs are generated for a bucket when log delivery is enabled. Below are the steps to enable log delivery for a Google Cloud Storage bucket.
Step 1: Create a bucket for storing logs ๐
This command will create a storage bucket.
gsutil mb gs://example-logs-bucket
Step 2: Update permissions ๐
Cloud storage needs the role roles/storage.objectCreator
to be able to create objects in the bucket.
gsutil iam ch group:cloud-storage-analytics@google.com:objectCreator gs://example-logs-bucket
Step 3: Enable logging ๐
Enable logging using the logging
command
gsutil logging set on -b gs://example-logs-bucket [-o log_object_prefix ] gs://example-bucket
Step 4: Confirm logging status ๐
The following command will show whether the logging is enabled or not, and where the log files are stored.
gsutil logging get gs://example-bucket
When logging is set up properly, it will respond with the details.
{"logBucket": "example-logs-bucket", "logObjectPrefix": "log_object_prefix"}
When logging is not enabled, the following is returned.
gs://example-bucket/ has no logging configuration.
Analyze logs in BigQuery ๐
The logs can be uploaded to BigQuery for faster analysis.
Step 1: Create a dataset ๐
This command will create a dataset in BigQuery.
bq mk storagelogs
Step 2: Get schema ๐
Data schema are available here:
Download the schema files in the working directory.
Step 3: Get the list of log files ๐
Run this on terminal to create a list of all log files stored in the bucket.
gsutil ls gs://example-logs-bucket > logs_files.txt
Step 4: Upload the files to BigQuery ๐
This bash script will loop over all logs files and upload them to BigQuery.
Please make sure that all log files are of the same type. If, for example, the log_files.txt
file created in the previous step contains both usage and storage logs, make sure to run this step twice - once for each type of logs.
for log in $(cat logs_files.txt); do
bq load --skip_leading_rows=1 --replace=false storagelogs.usage $log ./usage_schema_v0.json
done
Step 5: Analyze in BigQuery ๐
Enable bq shell
bq shell
Run Query
SELECT sc_status, COUNT(*) AS counted FROM storagelogs.usage GROUP BY 1
Exit the BigQuery shell
exit
Disable usage and storage logs ๐
Run this command in terminal to disable logging.
gsutil logging set off gs://example-bucket
Validate that it is disabled.
gsutil logging get gs://example-bucket
If logging is disabled, this is returned.
gs://example-bucket/ has no logging configuration.