Monitoring

It is a generally recommended practice to monitor the behavior of your applications. When it comes to Kafka, this can be done by logging metrics from your client directly to Prometheus (by using micrometer or another suitable library), or by creating panels using the metrics the streaming platform pushes to Prometheus directly.

Logging metrics with Micrometer

We suggest using Micrometer as described in Metric collection if your application is Spring Boot-based. Micrometer provides a set of binders specifically for Kafka-related metrics. If you are using Spring-Kafka either directly or through entur-kafka-spring-starter, including the spring boot actuator and prometheus registry dependencies (see Metric collection ) will allow your application to automatically expose a number of useful metrics. These include number of records consumed, lag, latency and more.

Accessing Team Data’s Kafka metrics

The streaming platform is set up to publish a number of relevant metrics via Prometheus to Grafana. Team Data maintains and actively uses a dashboard which can be used to gain an idea of which metrics are available. While a number of these are internal performance metrics for Kafka and its related components, metrics like Consumer Lag may be interesting for consumer applications to monitor directly if exposing these through your own application metrics is not feasible.