How to use the command 's3cmd' (with examples)

The ‘s3cmd’ command line tool is used for uploading, retrieving, and managing data in S3 compatible object storage. It provides a simple and convenient way to interact with S3 storage, allowing users to perform various operations such as configuring the tool, listing buckets and objects, creating buckets and folders, downloading and uploading files, moving objects between buckets, and deleting objects.

Use case 1: Invoke configuration/reconfiguration tool

Code:

s3cmd --configure

Motivation: The “–configure” option is used to invoke the configuration/reconfiguration tool. This allows users to set up their access credentials and S3 endpoint in order to connect to the S3 compatible object storage.

Explanation: The “–configure” option is used after the “s3cmd” command to initiate the configuration process. It prompts the user to enter their access key, secret key, default S3 endpoint, and other settings required to establish a connection with the S3 storage.

Example output:

Enter new values or accept defaults in parentheses.
Access Key: <enter your access key>
Secret Key: <enter your secret key>
Default Region [US]: <enter your default region>
S3 Endpoint [s3.amazonaws.com]: <enter your S3 endpoint>
DNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.s3.amazonaws.com]: 
Encryption password: <enter your encryption password>
Path to GPG program [/usr/bin/gpg]: 
Use HTTPS protocol [Yes]: 
HTTP Proxy server name: 
New settings:
  Access Key: <your access key>
  Secret Key: <your secret key>
  Default Region: <your default region>
  S3 Endpoint: <your S3 endpoint>
  DNS-style bucket+hostname:port template for accessing a bucket: %(bucket)s.s3.amazonaws.com
  Encoding: UTF-8
  Encryption Password: <your encryption password>
  Path to GPG program: /usr/bin/gpg
  Use HTTPS protocol: True
  HTTP Proxy server name: 
  HTTP Proxy server port: 0

Use case 2: List Buckets/Folders/Objects

Code:

s3cmd ls s3://bucket|path/to/file

Motivation: The “s3cmd ls” command is used to list the buckets, folders, or objects in the S3 storage. This can be useful to check the contents of the storage and verify if a specific bucket, folder, or object exists.

Explanation: The “s3cmd ls” command is followed by the S3 URI of the bucket, folder, or object that needs to be listed. The S3 URI starts with “s3://” followed by the bucket name or path to the file. It can be used to list the buckets by omitting the bucket name, list the folders inside a bucket by specifying the bucket name, or list the objects inside a folder by specifying the path to the folder.

Example output:

2022-01-01 12:00  s3://bucket/
2022-01-02 10:30  s3://bucket/path/to/folder/
2022-01-03 09:45  s3://bucket/path/to/file1.txt
2022-01-03 10:15  s3://bucket/path/to/file2.txt

Use case 3: Create Bucket/Folder

Code:

s3cmd mb s3://bucket

Motivation: The “s3cmd mb” command is used to create a new bucket or folder in the S3 storage. This is useful when you want to organize your data by creating new buckets or folders to store files.

Explanation: The “s3cmd mb” command is followed by the S3 URI of the bucket or folder that needs to be created. The S3 URI starts with “s3://” followed by the desired bucket name or path to the folder. Only the bucket or folder specified at the end of the S3 URI will be created, while any parent buckets or folders should already exist.

Example output:

Bucket 's3://bucket/' created

Use case 4: Download a specific file from a bucket

Code:

s3cmd get s3://bucket_name/path/to/file path/to/local_file

Motivation: The “s3cmd get” command is used to download a specific file from a bucket in the S3 storage. This allows users to retrieve files from the S3 storage for local use or further processing.

Explanation: The “s3cmd get” command is followed by the S3 URI of the file that needs to be downloaded and the local file path where the file should be saved. The S3 URI starts with “s3://” followed by the bucket name and the path to the file. The local file path specifies the location on the local system where the file should be downloaded.

Example output:

File 's3://bucket_name/path/to/file' saved as 'path/to/local_file'

Use case 5: Upload a file to a bucket

Code:

s3cmd put local_file s3://bucket/file

Motivation: The “s3cmd put” command is used to upload a local file to a bucket in the S3 storage. This allows users to transfer files from their local system to the remote S3 storage for storage or sharing purposes.

Explanation: The “s3cmd put” command is followed by the local file path that needs to be uploaded and the S3 URI specifying the bucket and the destination filename. The local file path is the path to the file on the local system that should be uploaded. The S3 URI starts with “s3://” followed by the bucket name and the desired filename for the uploaded file.

Example output:

upload: 'local_file' -> 's3://bucket/file'  [1 of 1]
1 file(s) copied.

Use case 6: Move an object to a specific bucket location

Code:

s3cmd mv s3://src_bucket/src_object s3://dst_bucket/dst_object

Motivation: The “s3cmd mv” command is used to move an object from a source bucket/object location to a destination bucket/object location in the S3 storage. This allows users to reorganize their data and move objects between different buckets or locations within the same bucket.

Explanation: The “s3cmd mv” command is followed by the S3 URI of the source object that needs to be moved and the S3 URI of the destination object location. Both source and destination URIs start with “s3://” followed by the bucket name and the path to the object. The source object will be moved to the destination object location.

Example output:

File 's3://src_bucket/src_object' moved to 's3://dst_bucket/dst_object'

Use case 7: Delete a specific object

Code:

s3cmd rm s3://bucket/object

Motivation: The “s3cmd rm” command is used to delete a specific object from a bucket in the S3 storage. This allows users to remove unwanted files or objects from the S3 storage to free up storage space or clean up the storage.

Explanation: The “s3cmd rm” command is followed by the S3 URI of the object that needs to be deleted. The S3 URI starts with “s3://” followed by the bucket name and the path to the object. The specified object will be permanently deleted from the bucket.

Example output:

File 's3://bucket/object' deleted

Conclusion:

The ‘s3cmd’ command line tool provides a versatile and efficient way to interact with S3 compatible object storage. By utilizing the various use cases of the command, users can easily configure the tool, list and organize buckets and objects, download and upload files, move objects between locations, and delete unwanted objects. The examples provided in this article demonstrate how to use the ‘s3cmd’ command for these different purposes, allowing users to effectively manage their data in the S3 storage.