Skip to main content

Keys in Kafka

Kafka records are divided into two parts - a key and a value. Value is used for the record itself. Keys are usually keys in Entur, but keys support the same types of information as value does.

What are keys used for in Kafka?​

For those coming from a RDBMS background, keys in Kafka may be misleading. Keys are not used for looking up specific messages - Kafka is a streaming platform, and only supports consuming messages by their offsets. Instead, keys serve two primary purposes:

  • Partition Assignment and Guaranteed ordering. Keys determine which partition a given message is assigned to. This usually results in a fairly even distribution of messages across partitions, but messages carrying identical keys will be assigned to the same partition. Such messages will always be consumed in the order they were produced, as only one consumer from each group is assigned to each partition.
  • Log compaction. When using log compaction, messages older than the topic’s retention setting are deleted only if a newer message with the same key exists.