Exporting Confluent Cloud Metrics to Prometheus

At Kafka Summit this year, Confluent announced consumption based billing for their Kafka Cloud offering, making it the cheapest and easiest way to get a Kafka Cluster. However, due to the Kafka cluster being multi-tenanted it comes with some restrictions, ZooKeeper is not exposed and the __consumer_offsets topic is restricted, this means popular tools like Kafka Manager and Prometheus Kafka Consumer Group Exporter won’t work.

kafka_exporter comes as a nice alternative as it uses the Kafka Admin Client to access the metrics. However, due to the authentication process required by Confluent Cloud it doesn’t work as is.

By forking kafka_exporter and upgrading the Kafka client one can get a successful output.

You can try out my build as follows:

$ docker run -p 9308:9308 -it imduffy15/kafka_exporter \
--kafka.server=host.region.gcp.confluent.cloud:9092 \
--sasl.enabled \
--sasl.username=username \
--sasl.password="password" \ 
--sasl.handshake \ 
--tls.insecure-skip-tls-verify \
--tls.enabled

and on querying http://localhost:9308/metrics all metrics documented here will be available. Prometheus can scrape this and alert and graph on the data.