Apache Kafka
Apache Kafka is an open-source distributed event streaming platform used by our automated data ingress to exchange messages between instrument (via Ingress Proxy) and NDIP.
Deployment
Apache Kafka runs inside a Docker container on a dedicated ingress proxy VM in the ORC cloud. It is deployed via a CI/CD job using Ansible.
SSH key for the VM can be found in Gitlab CI/CD Variables. You might need to request for access to this page.
Host
The ingress proxy VM is provisioned through our infrastructure monorepo (see the infrastructure section for more details). The VM's IP address can be found in the Ansible inventory (look for the ingress-proxy-vm record).
Deployment Details
The deployment is managed through the ingress-proxy Ansible role. This role includes:
- A configuration template for the config file.
- A task to start a Docker container with Apache Kafka.
Configuration values defined in an Ansible variable file.