Kafka migration to VPC networks

Virtual Private Cloud (VPC) peering is a method of connecting separate cloud(AWS, Google Cloud, or Azure) private networks with each other. It allows virtual machines in different private networks to talk to each other directly without going through the public Internet.

Aiven's VPC peering allows only private networks in the same cloud provider to talk to each other without going through the public internet. Which means that our Azure cloud users can only access Kafka services that are migrated to Google VPC network via only public URLs

Kafka platform changes

1. Team Data Platform and Team Platform have created the necessary VPC resources to migrate existing internal Kafka clusters to VPC networks.

2. Kafka clusters serving external Kafka users are NOT migrated and their usage remains same as before

   1. entur-kafka-test-ext
   2. entur-kafka-prod-ext

3. This migration has no impact on all Entur applications running in GKE clusters in dev, staging and production environments

4. Following are the clusters that will be migrated.

   1. entur-kafka-test-int
   2. entur-kafka-prod-int

5. Following are the Kafka users affected by this migration. Public URLs are created with a public- prefix to the existing bootstrap and schema registry server URLs for these users.

   1. Entur applications running in other cloud networks like Azure Cloud etc
   2. CI/CD applications
   3. Entur developers

   ➡️ Switching to public URLs is mandatory after switching to VPC networks for the above users as the old/existing URLs are assigned to the private networks

6. The same Kafka user credentials should work as before

Kafka services URL lookup

Cluster ID	Bootstrap Server URL	Schema Registry Server URL	Access	VPC
entur-kafka-test-ext	entur-kafka-test-ext-entur-test.aivencloud.com:11877	https://entur-kafka-test-ext-entur-test.aivencloud.com:11867	public	no
entur-kafka-prod-ext	entur-kafka-prod-ext-entur-prod.aivencloud.com:14019	https://entur-kafka-prod-ext-entur-prod.aivencloud.com:14009	public	no
entur-kafka-test-int	entur-kafka-test-int-entur-test.aivencloud.com:11877	https://entur-kafka-test-int-entur-test.aivencloud.com:11867	private	yes
entur-kafka-test-int	public-entur-kafka-test-int-entur-test.aivencloud.com:11878	https://public-entur-kafka-test-int-entur-test.aivencloud.com:11867	public	yes
entur-kafka-prod-int	entur-kafka-prod-int-entur-prod.aivencloud.com:14019	https://entur-kafka-prod-int-entur-prod.aivencloud.com:14009	private	yes
entur-kafka-prod-int	public-entur-kafka-prod-int-entur-prod.aivencloud.com:14020	https://public-entur-kafka-prod-int-entur-prod.aivencloud.com:14009	public	yes

Changes in usage with entur-kafka-spring-starter library

There is no change in default configuration as long as your application is accessing

1. External Kafka clusters that are not migrated to VPC networks
2. Internal Kafka clusters accessing from Entur's GKE clusters

Here is an example configuration change for accessing entur-kafka-test-int cluster where one has to use public URLs with entur-kafka-spring-starter library.

entur:
  kafka:
    bootstrapServer: "public-entur-kafka-test-int-entur-test.aivencloud.com:11878"
    schemaRegistryUrl: "https://public-entur-kafka-test-int-entur-test.aivencloud.com:11867"
    schemaRegistryBasicAuth: "${KAFKA_USER_NAME}:${KAFKA_USER_PASSWORD}"
    sasl:
      username: "${KAFKA_USER_NAME}"
      password: "${KAFKA_USER_PASSWORD}"

Migration strategy

1. All Kafka applications accessing from outside Entur's GKE clusters can start using public URLs
   1. Azure Cloud users
   2. CI/CD pipeline users
   3. Entur developers
2. entur-kafka-test-int cluster will be migrated to VPC network
3. entur-kafka-prod-int cluster will be migrated to VPC network in the end.