How to Manage Amazon MSK Clusters Using 'aws kafka' (with examples)

The aws kafka command is a valuable tool for users of Amazon Managed Streaming for Apache Kafka (Amazon MSK). It provides a command-line interface to manage MSK clusters, configurations, and related resources on AWS. This allows users to create, describe, list, update, and delete Kafka clusters and configurations, which are essential for building reliable and scalable streaming data pipelines. Below are several practical use cases demonstrating the power and versatility of the aws kafka command.

Use Case 1: Create a New MSK Cluster

Code:

aws kafka create-cluster --cluster-name myCluster --broker-node-group-info instanceType=m5.large,clientSubnets=subnet-abc123 subnet-def456 --kafka-version 2.8.1 --number-of-broker-nodes 3

Motivation:

Creating a new MSK cluster is often the first step in utilizing Amazon MSK for stream processing tasks. You might want to set up a new cluster to handle a specific workload or separate multiple environments, such as development and production.

Explanation:

• --cluster-name myCluster: Specifies the name of the new Kafka cluster.
• --broker-node-group-info instanceType=m5.large,clientSubnets=subnet-abc123 subnet-def456: Defines the properties for the broker nodes. Here, instanceType is set to m5.large, and two subnets are specified for client connections.
• --kafka-version 2.8.1: Indicates the version of Apache Kafka you wish to run on the cluster.
• --number-of-broker-nodes 3: Sets the number of broker nodes in the cluster, determining its capacity and resilience.

Example Output:

{
    "ClusterArn": "arn:aws:kafka:us-west-2:123456789012:cluster/myCluster/uuid",
    "ClusterName": "myCluster",
    "State": "CREATING",
    ...
}

Use Case 2: Describe a MSK Cluster

Code:

aws kafka describe-cluster --cluster-arn arn:aws:kafka:us-west-2:123456789012:cluster/myCluster/uuid

Motivation:

Describing a Kafka cluster is crucial when you need to gather details about its status, configuration, and operational metrics. This information can be used for monitoring, troubleshooting, or auditing purposes.

Explanation:

• --cluster-arn arn:aws:kafka:us-west-2:123456789012:cluster/myCluster/uuid: Provides the Amazon Resource Name (ARN) of the Kafka cluster you wish to describe. The ARN uniquely identifies a resource in AWS.

Example Output:

{
    "ClusterInfo": {
        "ClusterArn": "arn:aws:kafka:us-west-2:123456789012:cluster/myCluster/uuid",
        "ClusterName": "myCluster",
        "ClusterState": "ACTIVE",
        ...
    }
}

Use Case 3: List All MSK Clusters in the Current Region

Code:

aws kafka list-clusters

Motivation:

Listing all Kafka clusters in your current AWS region helps you quickly discover available clusters for management or monitor any changes in deployment.

Explanation:

This command doesn’t require any arguments since it lists all existing clusters in the current region based on the configured AWS CLI profile and region settings.

Example Output:

{
    "ClusterInfoList": [
        {
            "ClusterArn": "arn:aws:kafka:us-west-2:123456789012:cluster/myCluster/uuid",
            "ClusterName": "myCluster",
            ...
        },
        ...
    ]
}

Use Case 4: Create a New MSK Configuration

Code:

aws kafka create-configuration --name myNewConfig --server-properties file://path/to/configuration_file.txt

Motivation:

Creating a new Kafka configuration allows you to customize broker settings, such as log retention policies or connection limits, tailored to your application’s requirements.

Explanation:

• --name myNewConfig: Sets the name for the new configuration.
• --server-properties file://path/to/configuration_file.txt: Specifies a local file containing broker properties written as key-value pairs formatted for Kafka configuration.

Example Output:

{
    "Arn": "arn:aws:kafka:us-west-2:123456789012:configuration/myNewConfig/uuid",
    "LatestRevision": {
        "CreationTime": "2023-01-01T12:00:00Z",
        "Revision": 1
    }
}

Use Case 5: Describe a MSK Configuration

Code:

aws kafka describe-configuration --arn arn:aws:kafka:us-west-2:123456789012:configuration/myNewConfig/uuid

Motivation:

Describing a Kafka configuration is useful to verify its settings or ensure consistency across different environments.

Explanation:

• --arn arn:aws:kafka:us-west-2:123456789012:configuration/myNewConfig/uuid: The ARN specifies which configuration you want to describe, providing a direct link to the resource.

Example Output:

{
    "Arn": "arn:aws:kafka:us-west-2:123456789012:configuration/myNewConfig/uuid",
    "Name": "myNewConfig",
    "ServerProperties": "log.retention.hours=168\nnum.io.threads=8\n...",
    ...
}

Use Case 6: List All MSK Configurations in the Current Region

Code:

aws kafka list-configurations

Motivation:

Listing all Kafka configurations gives an overview of available configurations, aiding in resource organization and management.

Explanation:

This command requires no additional arguments, and it simply returns a list of all configurations within the region set in your AWS CLI profile.

Example Output:

{
    "Configurations": [
        {
            "Arn": "arn:aws:kafka:us-west-2:123456789012:configuration/myNewConfig/uuid",
            "Name": "myNewConfig",
            ...
        },
        ...
    ]
}

Use Case 7: Update the MSK Cluster Configuration

Code:

aws kafka update-cluster-configuration --cluster-arn arn:aws:kafka:us-west-2:123456789012:cluster/myCluster/uuid --configuration-info arn=arn:aws:kafka:us-west-2:123456789012:configuration/myNewConfig/uuid,revision=1

Motivation:

Updating a cluster’s configuration is necessary when your data processing requirements change, ensuring optimal performance and resource utilization.

Explanation:

• --cluster-arn arn:aws:kafka:us-west-2:123456789012:cluster/myCluster/uuid: Identifies the cluster to update.
• --configuration-info arn=arn:aws:kafka:us-west-2:123456789012:configuration/myNewConfig/uuid,revision=1: Specifies the new configuration ARN and its revision number.

Example Output:

{
    "ClusterArn": "arn:aws:kafka:us-west-2:123456789012:cluster/myCluster/uuid",
    "ConfigurationState": "UPDATE_IN_PROGRESS",
    ...
}

Use Case 8: Delete the MSK Cluster

Code:

aws kafka delete-cluster --cluster-arn arn:aws:kafka:us-west-2:123456789012:cluster/myCluster/uuid

Motivation:

Deleting a cluster is necessary when it’s no longer needed, freeing up resources and potentially reducing costs.

Explanation:

• --cluster-arn arn:aws:kafka:us-west-2:123456789012:cluster/myCluster/uuid: The ARN of the cluster you wish to delete, ensuring the correct resource is removed.

Example Output:

{
    "ClusterArn": "arn:aws:kafka:us-west-2:123456789012:cluster/myCluster/uuid",
    "State": "DELETING",
    ...
}

Conclusion:

By understanding these use cases, users can effectively manage Apache Kafka clusters through Amazon’s MSK service, making use of the full range of capabilities provided by the aws kafka CLI command. From creating and configuring clusters to describing, updating, and deleting them, this tool supports robust and efficient data streaming applications on AWS.