Streaming (Kafka)

This doc provides best practices and how to use Entur’s data streaming platform. While we try to cover the most common use cases, Kafka’s highly adaptable and configurable nature makes it difficult to cover every eventuality. If you’re unsure about how to do something or feel like any essential information is missing from this handbook, please feel free to reach out to us on Slack at #talk-kafka.

Entur maintains four distinct Kafka clusters (in addition to an unstable internal development cluster). Test and production each have both an internal and an external cluster.

Internal clusters are for unfiltered data. As this data might belong to several different partners, only internal Entur-applications are granted access.

External clusters are for data ready for consumption by external organisations. Such data is typically (but not always) filtered into separate topics per organisation, with only the owning organisation being granted access to consume each topic.

For more information pertaining to cluster bootstrap broker hosts and credentials, please see Kafka admin, and LastPass.