simple zookeeper shell commands

A few days ago I encountered an issue in a Kafka cluster. Each Kafka instances in the cluster constantly reported the below error logs:

[2021-02-01 21:59:59,952] ERROR [KafkaApi-2] Number of alive brokers ‘0’ does not meet the required replication factor ‘3’ for the offsets topic (configured via ‘offsets.topic.replication.factor’). This error can be ignored if the cluster is starting up and not all brokers are up yet. (kafka.server.KafkaApis)
[2021-02-01 21:59:59,990] ERROR [KafkaApi-2] Number of alive brokers ‘0’ does not meet the required replication factor ‘3’ for the offsets topic (configured via ‘offsets.topic.replication.factor’). This error can be ignored if the cluster is starting up and not all brokers are up yet. (kafka.server.KafkaApis)
[2021-02-01 21:59:59,992] ERROR [KafkaApi-2] Number of alive brokers ‘0’ does not meet the required replication factor ‘3’ for the offsets topic (configured via ‘offsets.topic.replication.factor’). This error can be ignored if the cluster is starting up and not all brokers are up yet. (kafka.server.KafkaApis)

There were four Kafka instances in the cluster. However, the cluster didn’t work correctly. After analyzing the configuration of the Kafka instances stored in the zookeeper, I found the root cause to this issue. The hostname advertised by one Kafka instance could not be resolved by other Kafka instances. To fix this issue, just add the IP address and hostname mapping to the /etc/hosts file.

In this post, I will introduce a few zookeeper shell commands, which I used to debug the Kafka issue, so that I can refer to them in the future.

1. The first one I want to mention is ‘ls /brokers/ids’

# zookeeper-shell localhost:2181 ls /brokers/ids
Connecting to localhost:2181

WATCHER::

WatchedEvent state:SyncConnected type:None path:null

[1, 2, 3, 4]

2. The second one is ‘get /brokers/ids/1’

# zookeeper-shell localhost:2181 get /brokers/ids/1
Connecting to localhost:2181

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
{“listener_security_protocol_map”:{“PLAINTEXT”:”PLAINTEXT”},”endpoints”:[“PLAINTEXT://kafka-instance-1:9092″],”jmx_port”:-1,”host”:”kafka-instance-1″,”timestamp”:”1612233716134″,”port”:9092,”version”:4}

I will not list too many zookeeper commands. For other commands, you can refer to the zookeeper website.