How to use the command gsutil (with examples)

Google Cloud Storage is a powerful storage service provided by Google Cloud Platform. gsutil is a command-line tool that allows users to access, manage, and manipulate data in Google Cloud Storage. It provides a wide range of functionalities for working with buckets and objects in Google Cloud Storage.

Use case 1: List all buckets in a project you are logged into

Code:

gsutil ls

Motivation: By listing all the buckets in a project, users can get an overview of the available storage resources. This can be helpful for managing and organizing the data stored in Google Cloud Storage.

Explanation: The command gsutil ls lists all the buckets in the project you are logged into. It does not require any additional arguments.

Example output:

gs://bucket1/
gs://bucket2/
gs://bucket3/

Use case 2: List the objects in a bucket

Code:

gsutil ls -r 'gs://bucket_name/prefix**'

Motivation: Sometimes, it is important to get a list of all the objects within a specific bucket. This can be useful for verifying the contents of a bucket or performing further operations on the objects within it.

Explanation: The command gsutil ls -r 'gs://bucket_name/prefix**' lists all the objects in the specified bucket. The -r flag is used to perform a recursive listing, which includes objects within subdirectories. The 'gs://bucket_name/prefix**' argument specifies the bucket name and an optional prefix to filter the results.

Example output:

gs://bucket_name/object1.txt
gs://bucket_name/dir/object2.txt
gs://bucket_name/dir/subdir/object3.txt

Use case 3: Download an object from a bucket

Code:

gsutil cp gs://bucket_name/object_name path/to/save_location

Motivation: Sometimes, it is necessary to download a specific object from a bucket for local processing or analysis. This command allows users to easily retrieve objects from Google Cloud Storage.

Explanation: The command gsutil cp gs://bucket_name/object_name path/to/save_location downloads the specified object from the bucket to the local filesystem. The gs://bucket_name/object_name argument specifies the object to be downloaded, and the path/to/save_location argument specifies the destination on the local filesystem.

Example output:

Copying gs://bucket_name/object_name [Content-Type=application/octet-stream]...
/ [1 files][  4.0 KiB/  4.0 KiB]
Operation completed over 1 objects/4.0 KiB.

Use case 4: Upload an object to a bucket

Code:

gsutil cp object_location gs://destination_bucket_name/

Motivation: Uploading objects to a bucket is essential for storing data in Google Cloud Storage. This command simplifies the process of uploading local files or data to a specified bucket.

Explanation: The command gsutil cp object_location gs://destination_bucket_name/ uploads the specified object or file to the destination bucket. The object_location argument specifies the local object or file to be uploaded, and the gs://destination_bucket_name/ argument specifies the target bucket.

Example output:

Copying file.txt [Content-Type=text/plain]...
/ [1 files][  4.0 KiB/  4.0 KiB]
Operation completed over 1 objects/4.0 KiB.

Use case 5: Rename or move objects in a bucket

Code:

gsutil mv gs://bucket_name/old_object_name gs://bucket_name/new_object_name

Motivation: Renaming or moving objects within a bucket is often required for organization or updating purposes. This command makes it easy to rename or move objects without the need for manual file transfers.

Explanation: The command gsutil mv gs://bucket_name/old_object_name gs://bucket_name/new_object_name renames or moves the specified object within the same bucket. The gs://bucket_name/old_object_name argument specifies the original object, and the gs://bucket_name/new_object_name argument specifies the new name or location of the object within the bucket.

Example output:

Copying gs://bucket_name/old_object_name [Content-Type=application/octet-stream]...
Removing gs://bucket_name/old_object_name...
/ [1 files][  4.0 KiB/  4.0 KiB]
Operation completed over 1 objects/4.0 KiB.

Use case 6: Create a new bucket in the project you are logged into

Code:

gsutil mb gs://bucket_name

Motivation: Creating a new bucket is the first step in starting to use Google Cloud Storage. This command allows users to easily create buckets in the project they are logged into.

Explanation: The command gsutil mb gs://bucket_name creates a new bucket with the specified name. The gs://bucket_name argument specifies the name and location of the bucket to be created.

Example output:

Creating gs://bucket_name/...
ServiceException: 409 Bucket bucket_name already exists.

Use case 7: Delete a bucket and remove all the objects in it

Code:

gsutil rm -r gs://bucket_name

Motivation: Deleting a bucket is necessary when the data within it is no longer needed or when cleaning up resources. This command simplifies the deletion process and ensures that all the objects within the bucket are removed as well.

Explanation: The command gsutil rm -r gs://bucket_name deletes the specified bucket and removes all the objects within it. The -r flag is used to perform a recursive deletion, ensuring that all the objects within the bucket are also deleted.

Example output:

Removing gs://bucket_name/object1.txt...
Removing gs://bucket_name/dir/object2.txt...
Removing gs://bucket_name/dir/subdir/object3.txt...
Removing gs://bucket_name/...

Conclusion:

gsutil is a versatile command-line tool that provides powerful features for managing and manipulating data within Google Cloud Storage. By understanding and utilizing its various use cases, users can effectively interact with Google Cloud Storage, making it easier to manage and work with large amounts of data.