How to use the command 'kcat' (with examples)

kcat is a versatile and lightweight command-line tool used for interacting with Apache Kafka, a popular distributed event streaming platform. Originally known as kafkacat, kcat serves as both a producer and consumer client for Kafka topics. It allows users to publish and consume messages, as well as inspect metadata about Kafka topics and brokers. kcat provides a user-friendly interface for performing quick operations on Kafka, making it an essential utility for developers and admins working with Kafka systems.

Use case 1: Consume messages starting with the newest offset

Code:

kcat -C -t topic -b brokers

Motivation: This use case is essential for users who want to consume real-time data from a Kafka topic. Starting from the newest offset allows for the collection of the most up-to-date information without going through the older, possibly irrelevant data. It’s useful in scenarios where only fresh data is necessary, such as monitoring systems or alerting services that depend on the latest data available.

Explanation:

• -C denotes that kcat is operating as a consumer.
• -t topic specifies the topic from which messages will be consumed.
• -b brokers provides the Kafka broker addresses, allowing kcat to connect to the correct cluster.

Example output:

{"user_id": 1, "action": "login", "timestamp": "2023-02-15T12:45:00Z"}
{"user_id": 2, "action": "logout", "timestamp": "2023-02-15T12:46:00Z"}

Use case 2: Consume messages starting with the oldest offset and exit after the last message is received

Code:

kcat -C -t topic -b brokers -o beginning -e

Motivation: Consuming messages from the oldest offset is critical when there is a need to process all messages in a topic from the very start. This is often required during data recovery, historical data analysis, or migration processes where all past messages need to be reviewed or processed. Exiting after the last message is controlled aids in automated tasks where script termination without user intervention is needed.

Explanation:

• -C is for consuming messages.
• -t topic points to the specific topic to consume.
• -b brokers indicates the broker addresses for connection.
• -o beginning instructs kcat to start consuming messages from the oldest available offset.
• -e causes the consumer to exit after the last message is received, making it efficient for batch jobs.

Example output:

{"user_id": 100, "action": "purchase", "timestamp": "2021-01-01T09:00:00Z"}
{"user_id": 101, "action": "page_view", "timestamp": "2021-01-01T09:05:00Z"}

Use case 3: Consume messages as a Kafka consumer group

Code:

kcat -G group_id topic -b brokers

Motivation: Consuming messages as part of a Kafka consumer group is beneficial for load balancing and processing parallelism in distributed applications. By joining a consumer group, you ensure that messages are distributed across all members, allowing for scalable consumption patterns. This approach is ideal for applications where multiple processes can work on the data simultaneously, like stream processing systems.

Explanation:

• -G group_id signifies that kcat is consuming as a part of the specified consumer group.
• topic shows the topic to consume.
• -b brokers gives the addresses for the connections.

Example output:

Group [group_id], topic [topic], partition [0]: 1/10 messages
Group [group_id], topic [topic], partition [1]: 2/10 messages

Use case 4: Publish message by reading from stdin

Code:

echo message | kcat -P -t topic -b brokers

Motivation: Publishing a message from standard input is practical for scenarios where messages are dynamically generated or obtained from other commands. This approach is useful for quick testing or scripting where you need to send a single or few messages to a Kafka topic without the need for creating a complex data flow.

Explanation:

• echo message generates or simulates the message to be sent.
• The pipe (|) redirects the output of the echo command into kcat.
• -P flags kcat to act as a producer.
• -t topic indicates the desired Kafka topic for message publication.
• -b brokers provides the broker addresses to connect with.

Example output:

% Sending message: [message]

Use case 5: Publish messages by reading from a file

Code:

kcat -P -t topic -b brokers path/to/file

Motivation: Publishing messages from a file is a straightforward approach for batch message publishing. It’s particularly advantageous in scenarios like data migration or initial topic population where messages are preformatted and saved in files. This allows for efficient bulk message insertion into Kafka topics without reprocessing or reformatting data.

Explanation:

• -P denotes that kcat is publishing messages (acting as a producer).
• -t topic specifies the topic to which the messages should be sent.
• -b brokers supplies the Kafka broker addresses and connects kcat to the topic.
• path/to/file gives the file path from where the messages will be read and published.

Example output:

% Reading messages from file: [path/to/file]

Use case 6: List metadata for all topics and brokers

Code:

kcat -L -b brokers

Motivation: Viewing metadata for all topics and brokers is necessary for administrators and developers to get insights into the Kafka cluster’s current state. This includes information on partition assignments, which topics exist, and which brokers are available. Such information is fundamental for troubleshooting, monitoring, and planning for cluster scaling.

Explanation:

• -L is used to list the topic and broker metadata.
• -b brokers indicates the Kafka broker addresses to fetch the metadata from.

Example output:

Metadata for all topics and brokers 
Broker 0 at 127.0.0.1:9092
 Topic topic1 with 3 partitions 
 Topic topic2 with 2 partitions

Use case 7: List metadata for a specific topic

Code:

kcat -L -t topic -b brokers

Motivation: Investigating metadata for a specific topic helps in understanding the topic’s configuration details such as the number of partitions, replication factor, and partition leaders. This information is valuable for performance tuning, debugging, and understanding the workload distribution across partitions and brokers.

Explanation:

• -L shows that the command is for listing metadata.
• -t topic specifies which topic’s metadata should be retrieved.
• -b brokers connects to the specified brokers for information retrieval.

Example output:

Metadata for topic [topic]:
 Partition 0 leader 1
 Partition 1 leader 0

Use case 8: Get offset for a topic/partition for a specific point in time

Code:

kcat -Q -t topic:partition:unix_timestamp -b brokers

Motivation: Retrieving offsets for a topic and partition at a specific point in time is critical for implementing point-in-time recovery or audit trails. This allows users to pinpoint the exact position in a Kafka log for precise data recovery or theoretical analysis at a specific timestamp, offering a powerful way to manage data retention and access control.

Explanation:

• -Q indicates a query operation to retrieve offsets.
• -t topic:partition:unix_timestamp specifies the partition of the topic and the timestamp at which to retrieve the offset.
• -b brokers is used for connecting to the Kafka brokers.

Example output:

Offset for topic [topic], partition [0] at [unix_timestamp]: 123

Conclusion

kcat demonstrates its capability as a versatile, user-friendly tool for managing Kafka topics with straightforward command-line commands. From consuming and producing messages to analyzing metadata and querying offsets, kcat offers a unified interface for Kafka operations. Its functionality is crucial for developers and administrators who seek efficient, quick interactions with Kafka clusters to achieve tasks like real-time message processing, historical data evaluation, and understanding the Kafka system’s internal state.