How to Use the AWS S3 Command (with Examples)

Amazon Web Services (AWS) Simple Storage Service (S3) provides scalable, high-speed, web-based cloud storage services. AWS S3 enables developers to store and retrieve any amount of data, at any time, from anywhere on the web. Using the AWS Command Line Interface (CLI), users can interact with AWS services, such as S3, using terminal commands, facilitating automation and management of cloud resources. The aws s3 command is a part of the AWS CLI and offers an interface to manage S3 buckets and objects.

Use case 1: Show Files in a Bucket

Code:

aws s3 ls bucket_name

Motivation:

Listing the contents of an S3 bucket is a fundamental task for managing stored data. It allows users to quickly verify the contents of a bucket, review the names of stored files, their sizes, and timestamps indicating when they were last modified. This is particularly useful for auditing, ensuring the right files are in place, and diagnosing any potential issues with expected data.

Explanation:

• aws s3: The main AWS CLI command for interacting with S3 services.
• ls: The subcommand used here resembles the UNIX ls command, listing files and directories.
• bucket_name: This argument specifies the target bucket within S3 from which to list contents. It is the unique identifier for the resource containing stored objects.

Example Output:

2023-09-25 10:30:00       34012 sample-file.txt
2023-09-24 14:45:00      209715 image1.png
2023-09-23 09:15:00          512 empty-directory/

Use case 2: Sync Files and Directories from Local to Bucket

Code:

aws s3 sync path/to/file1 path/to/file2 ... s3://bucket_name

Motivation:

Synchronizing files from a local machine to an S3 bucket is crucial for data backup, sharing, or replication across systems. Syncing ensures that only changed or new files are copied, saving bandwidth and reducing execution time, which is more efficient than “uploading” every file indiscriminately. This way, users keep their cloud storage updated with minimal effort.

Explanation:

• aws s3: The AWS CLI command establishing interaction with S3.
• sync: A powerful subcommand that compares the source and destination paths and transfers only those files that differ.
• path/to/file1 path/to/file2 ...: These paths represent local files or directories slated for synchronization with the bucket.
• s3://bucket_name: The destination path where files will be copied, representing a bucket in the S3 storage.

Example Output:

Assuming file1 and file2 are being copied, and no prior existing copies in the bucket:

upload: path/to/file1 to s3://bucket_name/file1
upload: path/to/file2 to s3://bucket_name/file2

Use case 3: Sync Files and Directories from Bucket to Local

Code:

aws s3 sync s3://bucket_name path/to/target

Motivation:

Syncing files from a bucket to a local machine is vital for retrieving backups, migrating data, or maintaining a local copy for offline access. This utility is just as important as uploading, ensuring local data reflects the latest from cloud storage without unnecessary duplication of unchanged files.

Explanation:

• aws s3: The CLI command to access AWS S3.
• sync: The subcommand intelligently updating local data to mirror the cloud version.
• s3://bucket_name: The source S3 bucket from which files will be synchronized.
• path/to/target: The local directory where files will be downloaded and saved.

Example Output:

download: s3://bucket_name/file1 to path/to/target/file1
download: s3://bucket_name/file2 to path/to/target/file2

Use case 4: Sync Files and Directories with Exclusions

Code:

aws s3 sync path/to/file1 path/to/file2 ... s3://bucket_name --exclude path/to/file --exclude path/to/directory/*

Motivation:

There are scenarios when only specific files or directories should be part of the synchronization process. Excluding certain files can help when sensitive information should not be uploaded, when dealing with large but unnecessary files, or optimizing storage costs and processing time.

Explanation:

• aws s3: Command line access to S3 storage.
• sync: Synchronizes files/directories, as in previous use cases.
• path/to/file1 path/to/file2 ...: Local files or directories intended for synchronization.
• s3://bucket_name: Destination S3 bucket path for storage.
• --exclude path/to/file: This flag and argument pair specifies files to avoid during the sync. Multiple occurrences are possible to define various files or patterns.
• --exclude path/to/directory/*: This pattern is an example to exclude a complete directory. The asterisk wild card means all content under this directory will not be transferred.

Example Output:

For demonstration, assuming exclusions apply to file3:

upload: path/to/file1 to s3://bucket_name/file1
upload: path/to/file2 to s3://bucket_name/file2

Use case 5: Remove File from Bucket

Code:

aws s3 rm s3://bucket/path/to/file

Motivation:

Removing files from an S3 bucket is part of data lifecycle management, security, and cost control. At times, files need to be deleted either due to reaching retention thresholds, contractual obligations, or simple housekeeping to ensure storage costs do not balloon with unnecessary data.

Explanation:

• aws s3: CLI command to interact with AWS S3.
• rm: Subcommand resembling the UNIX rm command; removes specified objects.
• s3://bucket/path/to/file: Complete path to the specific file object within a bucket targeted for deletion.

Example Output:

delete: s3://bucket/path/to/file

Use case 6: Preview Changes Only

Code:

aws s3 any_command --dryrun

Motivation:

The --dryrun option is excellent for evaluating the outcome of potentially destructive or costly operations without executing them. This preview is invaluable for debugging scripts, verifying commands, learning CLI actions, and ensuring there are no unintended consequences.

Explanation:

• aws s3: The base command indicating an interaction with S3.
• any_command: Placeholder for commands such as cp, sync, rm, allowing dry-run on its given operation.
• --dryrun: The flag that triggers preview mode, displaying actions without applying changes to actual data.

Example Output:

(dryrun) upload: path/to/file1 to s3://bucket_name/file1
(dryrun) upload: path/to/file2 to s3://bucket_name/file2

Conclusion

The AWS S3 CLI offers powerful flexibility and control for managing cloud-stored data through a straightforward command-line interface. From basic listing of objects to advanced sync operations both locally and on the cloud, these use cases illustrate various functionalities provided by the aws s3 command. These features can streamline operations, support data integrity, and enhance productivity within your cloud-based workflows.