AWS Batch: A Comprehensive Guide to Batch Computing Workloads (with examples)

AWS Batch is a service provided by Amazon Web Services (AWS) that allows users to run batch computing workloads efficiently. Batch computing involves performing a series of tasks, known as jobs, in parallel to save time and resources. In this guide, we will explore the different use cases of AWS Batch and provide code examples for each command.

List Running Batch Jobs

To list the running batch jobs in a specific job queue, you can use the list-jobs command.

aws batch list-jobs --job-queue queue_name

Motivation: This command is useful to get an overview of all the running jobs in a particular job queue. It allows you to monitor the progress and status of your jobs, ensuring efficient resource utilization.

Arguments:

• queue_name: The name of the job queue for which you want to list the running jobs.

Example Output:

{
    "jobSummaryList": [
        {
            "jobId": "87654321-abcd-1234-5678-0123456789ab",
            "jobName": "example-job",
            "status": "RUNNING",
            "createdAt": 1632356712
        },
        {
            "jobId": "12345678-abcd-5678-1234-0987654321ab",
            "jobName": "another-job",
            "status": "SUCCEEDED",
            "createdAt": 1632356802
        }
    ]
}

Create Compute Environment

To create a compute environment where your batch jobs will run, you can use the create-compute-environment command.

aws batch create-compute-environment --compute-environment-name compute_environment_name --type type

Motivation: This command is necessary to set up the infrastructure required to execute your batch jobs. By creating a compute environment, you define the resources and configuration for your jobs’ execution.

Arguments:

• compute_environment_name: The name of the compute environment you want to create.
• type: The type of compute environment. It can be either “MANAGED” or “UNMANAGED”. A managed environment is optimized for simplicity, while an unmanaged environment provides more control over resources.

Example Output:

{
    "computeEnvironmentName": "example-compute-env",
    "computeEnvironmentArn": "arn:aws:batch:us-west-2:123456789012:compute-environment/example-compute-env",
    "state": "ENABLED"
}

Create Batch Job Queue

To create a batch job queue, which manages the scheduling and execution of jobs, you can use the create-job-queue command.

aws batch create-job-queue --job-queue-name queue_name --priority priority --compute-environment-order compute_environment

Motivation: A job queue determines the order in which jobs are executed and the compute resources they use. Creating a batch job queue helps organize and prioritize your batch jobs effectively.

Arguments:

• queue_name: The name of the job queue you want to create.
• priority: The priority of the job queue. A higher value means higher priority.
• compute_environment_order: The order in which compute environments are tried for job execution. You can specify multiple compute environments and their relative order.

Example Output:

{
    "jobQueueName": "example-job-queue",
    "jobQueueArn": "arn:aws:batch:us-west-2:123456789012:job-queue/example-job-queue",
    "state": "ENABLED"
}

Submit Job

To submit a batch job to a specific job queue, you can use the submit-job command.

aws batch submit-job --job-name job_name --job-queue job_queue --job-definition job_definition

Motivation: This command is essential to submit batch jobs for execution. By providing the job name, job queue, and job definition, you can ensure proper execution and configuration of your job.

Arguments:

• job_name: The name of the job you want to submit.
• job_queue: The name or ARN of the job queue to which you want to submit the job.
• job_definition: The name or ARN of the job definition. This defines the Docker container image and command used for job execution.

Example Output:

{
    "jobId": "87654321-abcd-1234-5678-0123456789ab",
    "jobName": "example-job",
    "jobArn": "arn:aws:batch:us-west-2:123456789012:job/example-job",
    "jobQueue": "example-job-queue"
}

Describe List of Batch Jobs

To get detailed information about a list of batch jobs, you can use the describe-jobs command.

aws batch describe-jobs --jobs jobs

Motivation: This command allows you to retrieve detailed information about the specified batch jobs, such as their state, start time, and job definition. It helps in monitoring and troubleshooting purposes.

Arguments:

• jobs: A space-separated list of job IDs or ARNs that you want to describe.

Example Output:

{
    "jobs": [
        {
            "jobId": "87654321-abcd-1234-5678-0123456789ab",
            "jobName": "example-job",
            "jobQueue": "example-job-queue",
            "status": "SUCCEEDED",
            "createdAt": 1632356712,
            "startedAt": 1632356715,
            "stoppedAt": 1632356720,
            "jobDefinition": "example-job-definition"
        }
    ]
}

Cancel Job

To cancel a running or queued batch job, you can use the cancel-job command.

aws batch cancel-job --job-id job_id --reason reason

Motivation: This command is useful when you need to terminate a job prematurely or stop the execution of a job that is no longer required. It allows you to manage and control your batch jobs effectively.

Arguments:

• job_id: The ID of the job you want to cancel.
• reason: The reason for cancelling the job.

Example Output:

{
    "jobId": "87654321-abcd-1234-5678-0123456789ab",
    "status": "CANCELED"
}

By using these different AWS Batch commands, you can efficiently manage, schedule, and execute your batch computing workloads, saving valuable time and resources. Whether it’s creating job queues, submitting jobs, or monitoring their progress, AWS Batch offers a powerful suite of tools to streamline and optimize your batch computing workflows.