How to Use the Command 'aws batch' (with examples)

AWS Batch is a fully managed service that helps developers, scientists, and engineers run batch computing workloads on the Amazon Web Services (AWS) Cloud. The service takes care of provisioning, managing, and scaling the underlying compute resources. It efficiently handles work ranging from simple shell scripts to complex multistep workflows. In this article, we illustrate common use cases of the aws batch command with practical examples.

Use Case 1: List Running Batch Jobs

Code:

aws batch list-jobs --job-queue queue_name

Motivation:

Listing running batch jobs helps users keep track of their workloads in the cloud. It provides a quick overview of active tasks, enabling users to monitor job statuses and resource consumption effectively. This capability is essential for project management, troubleshooting, and optimizing resource allocation.

Explanation:

• aws batch list-jobs: This command invokes AWS Batch to list jobs.
• --job-queue queue_name: The --job-queue flag specifies the job queue name for which you want to list the jobs. The queue_name represents the specific queue in your AWS Batch setup where the jobs are running.

Example Output:

{
  "jobSummaryList": [
    {
      "jobId": "abcdefgh-1234-5678-abcd-1234567890ab",
      "jobName": "process-data-job",
      "status": "RUNNING",
      "createdAt": 1622547800000
    },
    ...
  ]
}

Use Case 2: Create Compute Environment

Code:

aws batch create-compute-environment --compute-environment-name compute_environment_name --type type

Motivation:

A compute environment defines the compute resources that AWS Batch can utilize to execute jobs. Creating a compute environment is vital for configuring the type and amount of resources necessary for batch processing. This flexibility allows users to tailor the infrastructure to their specific workload needs, balancing cost and performance.

Explanation:

• aws batch create-compute-environment: This command initializes a new compute environment in AWS Batch.
• --compute-environment-name compute_environment_name: Indicates the name of the compute environment. This name is a label for reference and management.
• --type type: Specifies the type of compute environment. It can be either MANAGED (AWS manages the compute resources) or UNMANAGED (users manage the resources).

Example Output:

{
  "computeEnvironmentArn": "arn:aws:batch:us-east-1:123456789012:compute-environment/compute_environment_name",
  "computeEnvironmentName": "compute_environment_name"
}

Use Case 3: Create Batch Job Queue

Code:

aws batch create-job-queue --job-queue-name queue_name --priority priority --compute-environment-order compute_environment

Motivation:

Job queues serve as the link between users’ submitted jobs and compute environments. Creating a batch job queue is crucial to arrange and prioritize job execution. Job queues can influence the order of job processing depending on the assigned priority, significantly impacting resource utilization and scheduling efficiency.

Explanation:

• aws batch create-job-queue: This command sets up a new job queue in AWS Batch.
• --job-queue-name queue_name: Specifies the name of the job queue, which you will refer to when submitting jobs.
• --priority priority: Determines the order in which jobs are processed if they are in different job queues.
• --compute-environment-order compute_environment: Dictates the list of compute environments to consider for the queue. The order influences which environments are utilized first.

Example Output:

{
  "jobQueueArn": "arn:aws:batch:us-east-1:123456789012:job-queue/queue_name",
  "jobQueueName": "queue_name"
}

Use Case 4: Submit Job

Code:

aws batch submit-job --job-name job_name --job-queue job_queue --job-definition job_definition

Motivation:

Submitting a job is the core function of using AWS Batch. By submitting a job, users trigger the execution of computational tasks defined in the job definition. This action effectively represents the initiation of workflow processes, making it integral for data analysis, computation, and various application workloads.

Explanation:

• aws batch submit-job: This command submits a new job for execution in AWS Batch.
• --job-name job_name: Assigns a unique name to the job, for identification and management purposes.
• --job-queue job_queue: Specifies the job queue to which this job is submitted. The queues define which compute environments are used.
• --job-definition job_definition: Indicates the job definition, which includes the details of what the job executes and its resource requirements.

Example Output:

{
  "jobId": "abcdefgh-1234-5678-abcd-1234567890ab",
  "jobName": "job_name"
}

Use Case 5: Describe the List of Batch Jobs

Code:

aws batch describe-jobs --jobs jobs

Motivation:

Describing a list of batch jobs provides detailed information about each job’s status, environment, and configuration settings. This is essential for tracking the progress, diagnosing issues, and gaining insights into resource utilization. It aids in comprehensive job management and fine-tuning future submissions.

Explanation:

• aws batch describe-jobs: This function retrieves details about specific batch jobs.
• --jobs jobs: Specifies a list of job IDs that you need information about. These IDs should correspond to submitted jobs within the AWS Batch.

Example Output:

{
  "jobs": [
    {
      "jobName": "data-analysis",
      "jobId": "abcdefgh-0001-2345-abcd-1234567890ab",
      "status": "SUCCEEDED",
      "createdAt": 1622547800000,
      "container": {
        "image": "my-docker-image",
        "memory": 1024,
        "vcpus": 1
      }
    },
    ...
  ]
}

Use Case 6: Cancel Job

Code:

aws batch cancel-job --job-id job_id --reason reason

Motivation:

In situations where job execution is no longer necessary or needs to be halted due to changes in project goals, job cancellation becomes a useful tool. It allows users to manage costs effectively by stopping unnecessary resource use. Canceling jobs also aids in queue management by freeing up resources for higher-priority tasks.

Explanation:

• aws batch cancel-job: This command cancels a specified job within AWS Batch.
• --job-id job_id: Identifies the job to cancel through its unique ID.
• --reason reason: Provides a user-defined reason for canceling the job. This is helpful for logging and tracking purposes.

Example Output:

{
  "jobArn": "arn:aws:batch:us-east-1:123456789012:job/job_id",
  "jobName": "job_name",
  "status": "CANCELLING"
}

Conclusion:

AWS Batch commands provide robust functionality for managing batch computing workloads in the cloud. From creating environments and job queues to monitoring and controlling job execution, these examples illustrate the flexibility and control users have over their computational tasks. By understanding these commands, users can optimize workflow management, enhance productivity, and make efficient use of cloud resources.