Using AWS Kinesis CLI Command (with examples)

Introduction

Amazon Kinesis is a managed streaming data service provided by AWS. It allows you to collect, process, and analyze real-time streaming data at a large scale. The AWS CLI provides a convenient way to interact with Amazon Kinesis and perform various operations. In this article, we will explore different use cases of AWS Kinesis CLI commands with code examples.

Use Case 1: Show all streams in the account

Code

aws kinesis list-streams

Motivation

This use case is helpful when you want to get an overview of all the Kinesis streams present in your AWS account.

Explanation

The list-streams command is used to list all the Kinesis streams in the account. It does not require any arguments.

Example Output

{
    "StreamNames": [
        "stream1",
        "stream2",
        "stream3"
    ]
}

Use Case 2: Write one record to a Kinesis stream

Code

aws kinesis put-record --stream-name name --partition-key key --data base64_encoded_message

Motivation

This use case is useful when you want to write a single record to a specific Kinesis stream.

Explanation

• stream-name: The name of the Kinesis stream to which you want to write the record.
• partition-key: A partition key is used to determine the shard to which the record will be written.
• base64_encoded_message: The record data encoded in base64 format.

Example Output

This command does not produce any output.

Use Case 3: Write a record to a Kinesis stream with inline base64 encoding

Code

aws kinesis put-record --stream-name name --partition-key key --data "$( echo "my raw message" | base64 )"

Motivation

If you have a raw message that is not already base64 encoded, this use case allows you to perform inline base64 encoding and write the record to the Kinesis stream.

Explanation

• stream-name: The name of the Kinesis stream to which you want to write the record.
• partition-key: A partition key used to determine the shard.
• my raw message: The raw message that needs to be encoded in base64 format.

Example Output

This command does not produce any output.

Use Case 4: List the shards available on a stream

Code

aws kinesis list-shards --stream-name name

Motivation

If you want to obtain information about the shards present in a Kinesis stream, this use case will help you list all the shards associated with the specified stream.

Explanation

• stream-name: The name of the Kinesis stream for which you want to list the shards.

Example Output

{
    "Shards": [
        {
            "ShardId": "shardId-000000000002",
            "HashKeyRange": { "StartingHashKey": "0", "EndingHashKey": "113427455640312821154458202477256070484" },
            "SequenceNumberRange": { "StartingSequenceNumber": "49545115243490985018280067714973144582180062593244260354" }
        },
        {
            "ShardId": "shardId-000000000001",
            "HashKeyRange": {"StartingHashKey": "113427455640312821154458202477256070485", "EndingHashKey": "226854911280625642308916404954512140969" },
            "SequenceNumberRange": { "StartingSequenceNumber": "49545115243490985018280067763927949869370339199244260354" }
        }
    ]
}

Use Case 5: Get a shard iterator for reading from the oldest message in a stream’s shard

Code

aws kinesis get-shard-iterator --shard-iterator-type TRIM_HORIZON --stream-name name --shard-id id

Motivation

When you want to start reading from the oldest available message in a shard, this use case will help in obtaining a shard iterator.

Explanation

• shard-iterator-type: The type of shard iterator to obtain. In this case, TRIM_HORIZON is used to read from the oldest message in the shard.
• stream-name: The name of the Kinesis stream.
• shard-id: The specific shard identifier.

Example Output

{
    "ShardIterator": "AAAAAAAAAAHhItwvILkYLZ/MoscTdRBVZuHsNxHdzj+SOJb2OMy5zmCn4DLcMk84vtHw4vzMC+x79ULFEk5OGNonliyAsKww+j3PlgBKtnmwVXCzByYPvez0tYvJYs7aHi3iOTypqqKf0JevcaLEQrAeLrBpoOS05ucxjljUZot9GdXX9vjUGB9nB07wX3bSjakJyLlJcEhx8QUED2zR5VaYE0kmFjR7QMNk4yZ74j4df7Igm9g6FXbWZO8HcR18J/rvmIa"
}

Use Case 6: Read records from a shard, using a shard iterator

Code

aws kinesis get-records --shard-iterator iterator

Motivation

To retrieve records from a specific shard using a shard iterator obtained previously, this use case can be utilized.

Explanation

• shard-iterator: The shard iterator obtained from Use Case 5.

Example Output

{
    "Records": [
        {
            "SequenceNumber": "49545115243490985018280067714973144582180062593244260354",
            "ApproximateArrivalTimestamp": 1662163036.709,
            "Data": "SGVsbG8gdGhlcmUh"
        },
        {
            "SequenceNumber": "49545115243490985018280067714973144582180062593244260355",
            "ApproximateArrivalTimestamp": 1662163038.309,
            "Data": "VGhpcyBpcyBhIHN0cmluZyBkYXRhIGluIGEgc3RyZWFtJ3Mgc2l0ZSBhcyBhIG5lZWQgdG8gc29tZSBkYXRhIHRvIGJlIHdpdGggdGhhdCB0aGF0IGV2ZW4gd29ya2luZyBkYXRhIHdpdGggdGhlIG5lZWRzIQ=="
        }
    ],
    "NextShardIterator": "AAAAAAAAAAGlEz1iJcRU/4s/hkpHu1621qv86xI9nKDkM4OVHYqhSZ+98Db6GTJsW1Jg6UbYEmPFTMiuKl0jLMNbTqY1bq+bXb0vAJAKi3sq7aRqrry5Jf4YV3hqPH/CMig4H6vZJOLYws5vOCtp8Ryf9HyW3g93PQ1oXDTzfiQeIRr0pav0J5w/aeYD0y5MNMlz2nxSEMoFewYFUOO7Fo6b6imFbMGV2OiVPIJ7+pByD1wAve3nY2lkuSYkN1f8"
}

Conclusion

In this article, we explored different use cases of AWS Kinesis CLI commands. We covered listing streams, writing records, listing shards, obtaining shard iterators, and reading records from a shard. With these examples, you can effectively interact with Amazon Kinesis using the AWS CLI and perform various streaming data operations. Have fun and explore the power of AWS Kinesis!