AWS Glue CLI Examples (with examples)

List jobs

aws glue list-jobs

Motivation: The list-jobs command allows you to retrieve a list of jobs in your AWS Glue environment. This is useful for monitoring and managing your jobs programmatically.

Arguments:

• None

Example Output:

{
    "Jobs": [
        {
            "Name": "example_job_1",
            "Id": "j_1234567890123456",
            "Type": "ETL",
            "CreatedOn": 1592968263,
            ...
        },
        {
            "Name": "example_job_2",
            "Id": "j_7890123456789012",
            "Type": "Python Shell",
            "CreatedOn": 1592968291,
            ...
        },
        ...
    ]
}

Start a job

aws glue start-job-run --job-name job_name

Motivation: The start-job-run command allows you to start a job run for a specific job in AWS Glue. This is useful for triggering job execution programmatically.

Arguments:

• --job-name: (Required) The name or ARN of the job to start the run for.

Example Output: None

Start running a workflow

aws glue start-workflow-run --name workflow_name

Motivation: The start-workflow-run command allows you to start a workflow run in AWS Glue. Workflows enable you to create a sequence of jobs and triggers, allowing for complex ETL pipeline orchestration.

Arguments:

• --name: (Required) The name or ARN of the workflow to start the run for.

Example Output: None

List triggers

aws glue list-triggers

Motivation: The list-triggers command allows you to retrieve a list of triggers in your AWS Glue environment. Triggers can be used to automatically start jobs based on events or schedules.

Arguments:

• None

Example Output:

{
    "Triggers": [
        {
            "Name": "example_trigger_1",
            "Id": "tt_1234567890123456",
            "Type": "SCHEDULED",
            "State": "CREATED",
            ...
        },
        {
            "Name": "example_trigger_2",
            "Id": "tt_7890123456789012",
            "Type": "CONDITIONAL",
            "State": "ENABLED",
            ...
        },
        ...
    ]
}

Start a trigger

aws glue start-trigger --name trigger_name

Motivation: The start-trigger command allows you to manually start a specific trigger in AWS Glue. This is useful for triggering jobs based on specific events or conditions.

Arguments:

• --name: (Required) The name or ARN of the trigger to start.

Example Output: None

Create a dev endpoint

aws glue create-dev-endpoint --endpoint-name name --role-arn role_arn_used_by_endpoint

Motivation: The create-dev-endpoint command allows you to set up a development endpoint in AWS Glue. Development endpoints are used for interactive development and debugging of ETL scripts.

Arguments:

• --endpoint-name: (Required) The name of the dev endpoint to create.
• --role-arn: (Required) The Amazon Resource Name (ARN) of the IAM role to be used by the endpoint.

Example Output:

{
    "EndpointName": "example_dev_endpoint",
    "Status": "READY",
    ...
}

By utilizing the AWS Glue CLI commands, you can automate various tasks in your AWS Glue environment such as managing jobs, workflows, triggers, and dev endpoints. These examples demonstrate how to perform common operations programmatically, providing flexibility and efficiency in your ETL pipeline management.