15 Kafka CLI Commands For Everyday Programming

Demonstrating using probably the most generally used Kafka Command Line Interface Instructions

Picture by Blake Connally on Unsplash

Apache Kafka is without doubt one of the mostly used applied sciences that facilitate even-streaming architectures. Regardless of typically being seen as *simply* a message dealer (like RabbitMQ for instance), Kafka is unquestionably far more than simply this.

Kafka was an inside LinkedIn venture which was open sourced in 2011 and shortly developed from a message dealer to a whole platform that permits even-streaming in a extremely scalable, fault-tolerant distributed method.

In right this moment’s article, we are going to undergo a number of the mostly used instructions in Kafka’s command line interface. Ensure to bookmark this tutorial as the probabilities are you will have to reference some elements very often when performing sure administration duties over Kafka clusters.

If you wish to checklist the subjects included in a particular dealer, the next command will do the trick:

$ kafka-topics 
--bootstrap-server localhost:9092
--list

Be aware that in older variations, you could possibly additionally use the Zookeeper endpoint as demonstrated beneath:

$ kafka-topics 
--zookeeper localhost:2181
--list

Since current Kafka releases will take away the Zookeeper dependency, I’d advise utilizing the primary command.

Lastly, if you wish to checklist the subjects throughout the entire cluster, then be sure to incorporate all of the brokers in --bootstrap-server:

$ kafka-topics 
--bootstrap-server localhost:9092,localhost:9093,localhost:9094
--list

With the intention to delete a subject from the cluster, you might want to move the --delete flag together with brokers and the subject identify to be deleted.

$ kafka-topics 
--bootstrap-server localhost:9092,localhost:9093,localhost:9094
--delete
--topic topic_for_deletion

When deleting subjects from Kafka brokers, you possibly can even specify regex-like expressions to be able to delete a number of subjects in a single go. For instance, let’s suppose we need to delete all subjects beginning with the prefix test-. The next command ought to do the trick:

$ kafka-topics 
--bootstrap-server localhost:9092,localhost:9093,localhost:9094
--delete
--topic 'test-.*'

Now if you wish to create a brand new subject, you possibly can merely achieve this utilizing the --create choice of kafka-topics runner.

$ kafka-topics 
--bootstrap-server localhost:9092
--create
--topic topic-name

Be aware that when creating a subject, you possibly can present some further configuration choices such because the variety of partitions and/or the replication issue. For example, to be able to create a subject with three partitions and a replication issue of two, the next command would do the trick:

$ kafka-topics 
--bootstrap-server localhost:9092
--create
--topic topic-name
--partitions 3
--replication-factor 2

If you wish to retrieve the configuration particulars of a particular subject you should use the --describe choice with kafka-topics runner.

$ kafka-topics 
--bootstrap-server localhost:9092
--describe
--topic topic-name

The command’s output gives you data similar to partition counts, replication issue, chief, replicas, and ISR in addition to further configurations (similar to retention interval if the default one was modified).

Subject:topic-name  PartitionCount:1    ReplicationFactor:1 Configs:
Subject: topic-name Partition: 0 Chief: 0 Replicas: 0 Isr: 0

Now let’s get to the info era. With the intention to produce some information on a particular subject, all you want is the kafka-console-consumer runner:

$ kafka-console-producer  
--bootstrap-server localhost:9092
--topic topic-name
> One occasion
> Oh, one other occasion!

In the event you additionally need to set keys for every message generated, then you should use some further choices (i.e., allow keys and key separator character) as demonstrated beneath:

$ kafka-console-producer  
--bootstrap-server localhost:9092
--topic topic-name
--property "parse.key=true"
--property "key.separator=:"
> key1:One occasion
> key2:Oh, one other occasion!

Alternatively, if you wish to eat messages within the terminal from a selected Kafka subject then all you want is the kafka-console-consumer:

$ kafka-console-consumer 
--bootstrap-server localhost:9092
--topic topic-name
--from-beginning
One occasion
Oh, one other occasion!

In the event you additionally need to print out the keys, then you might also need to specify the important thing separator character, the flags to allow printing for each the important thing and the worth, in addition to the important thing and worth deserializers.

For instance,

$ kafka-console-consumer 
--bootstrap-server localhost:9092
--topic topic-name
--from-beginning
--property key.separator="-"
--property print.key=true
--property print.worth=true
--property key.deserialzer=org.apache.kafka.widespread.serialization.StringDeserializer
--property worth.deserializer=org.apache.kafka.widespread.serialization.LongDeserializer

Now to be able to purge a Kafka subject and delete all of the messages, you possibly can merely change the retention interval to 1 second briefly after which change it again once more.

To vary the retention to 1 second you possibly can merely use the --add-config flag with the kafka-configs runner:

$ kafka-configs 
--bootstrap-server localhost:9092
--alter
--entity-type subjects
--entity-name topic-name
--add-config retention.ms=1000

And to revert it again, merely delete this config (in order that the default retention is then used):

$ kafka-configs 
--bootstrap-server localhost:9092
--alter
--entity-type subjects
--entity-name topic-name
--delete-config retention.ms

Each Kafka subject might have quite a few client teams. With the intention to checklist all the patron teams throughout all subjects in a Kafka cluster, you possibly can merely use the --list flag with the kafka-consumer-groups runner.

$ kafka-consumer-groups 
--bootstrap-server localhost:9092
--list

The output will include the names of all client teams:

test-consumer-group-1
test-consumer-group-2
test-consumer-group-3
...

Now to be able to get some extra particulars a few particular client group, you should use the --describe flag together with the patron group identify (that may be inferred utilizing the command we demonstrated within the earlier part).

$ kafka-consumer-groups 
--bootstrap-server localhost:9092
--describe
--group my-group

And the output will include the next data:

TOPIC 
PARTITION
CURRENT-OFFSET
LOG-END-OFFSET
LAG
CONSUMER-ID
HOST
CLIENT-ID

The --describe choice may also be mixed with further choices to be able to get much more particulars a few particular client group.

With the intention to checklist all energetic members in a particular client group, then you may also specify the --members flag:

$ kafka-consumer-groups 
--bootstrap-server localhost:9092
--describe
--group my-group
--members

Right here’s an instance output from the above command:

CONSUMER-ID          HOST           CLIENT-ID       #PARTITIONS
consumer1-3fc8d.. /127.0.0.1 consumer1 2
consumer4-117fe.. /127.0.0.1 consumer4 1
consumer2-e76ea.. /127.0.0.1 consumer2 3
consumer3-ecea4.. /127.0.0.1

To get much more particulars you may also use the --verbose choice that will even report the partitions assigned to every member.

$ kafka-consumer-groups 
--bootstrap-server localhost:9092
--describe
--group my-group
--members
--verbose

And the output ought to appear to be the one shared beneath.

CONSUMER-ID HOST       CLIENT-ID  #PARTITIONS ASSIGNMENT
.. /127.0.0.1 consumer1 2 topic1(0), topic2(0)
.. /127.0.0.1 consumer4 1 topic3(2)
.. /127.0.0.1 consumer2 3 topic2(1), topic3(0,1)
.. /127.0.0.1 consumer3 0 -

If for any purpose you might want to enhance the variety of partitions for a particular subject, then you should use the --alter flag to be able to specify the brand new — enhance — variety of partitions.

$ kafka-topics 
--bootstrap-server localhost:9092
--alter
--topic topic-name
--partitions 40

One other widespread factor chances are you’ll have to do is so as to add additional configuration choices to a particular (and even all) dealer.

For example, let’s suppose we need to specify the variety of background threads to make use of for log cleansing. This behaviour will be adjusted by means of the log.cleaner.threads configuration choice.

To take action, you possibly can run the kafka-configs runner together with the dealer particulars and --add-config, as demonstrated beneath.

$ kafka-configs 
--bootstrap-server localhost:9092
--entity-type brokers
--entity-name 0
--alter
--add-config log.cleaner.threads=2

If you wish to add a configuration to all brokers in a cluster, then you should use the --entity-default flag (as an alternative of specifying a particular dealer by means of (--entity-name broker-id), as demonstrated beneath.

$ kafka-configs 
--bootstrap-server localhost:9092
--entity-type brokers
--entity-default
--alter
--add-config log.cleaner.threads=2

It’s necessary to notice right here that some configuration to be efficient might require the brokers to be restarted. You may examine the replace mode of every particular person dealer configuration within the relevant section of the official documentation.

Alternatively, you might also need to take away some configurations from a particular dealer. All you want is to specify the dealer id, (--entity-name) and the --delete-config flag to specify the choice you want to take away.

$ kafka-configs 
--bootstrap-server localhost:9092
--entity-type brokers
--entity-name 0
--alter
--delete-config log.cleaner.threads

Once more, as an alternative of specifying the dealer id by means of --entity-name, you possibly can present the --entity-default flag if you want the removing to be efficient for each dealer throughout the cluster.

Now, to be able to get the present configuration of a particular dealer, you possibly can specify the dealer id (i.e. --entity-name) together with the --describe flag when executing kafka-configs runner.

$ kafka-configs 
--bootstrap-server localhost:9092
--entity-type brokers
--entity-name 0
--describe

More Posts