Distributed Log Processor (RabbitMQ-based)

Project: Distributed Log Processor (RabbitMQ-based)

GitHub Repository: distributed-log-processor

🎯 Goal

Design a small distributed system where multiple applications send log events to RabbitMQ. A consumer service processes, enriches, and stores those logs (in Elasticsearch, S3, or a DB). This project demonstrates the use of RabbitMQ as a message broker in a distributed architecture and is intended for educational purposes.

Prerequisites

• A Kubernetes cluster (e.g., Minikube, Kind, GKE, EKS)
• kubectl installed and configured to interact with your cluster

For my setup I am using a local K3s cluster.

Installation

The easiest way to deploy RabbitMQ on Kubernetes is by using a Helm chart. This can be found here: https://artifacthub.io/packages/helm/bitnami/rabbitmq However, for this project, we will take the manual approach using the RabbitMQ Cluster Operator and a custom resource definition to gain more understanding of the underlying components.

1. Install the RabbitMQ Cluster Operator: Ref: https://www.rabbitmq.com/kubernetes/operator/install-operator

   kubectl apply -f "https://github.com/rabbitmq/cluster-operator/releases/latest/download/cluster-operator.yml"
   

2. Deploy a RabbitMQ cluster using the provided definition: Ref: https://www.rabbitmq.com/kubernetes/operator/cluster-configuration.html

Be mingdful if you are using a different namespace than rabbitmq-system, update the namespace field in definition.yaml accordingly.

kubectl apply -f definition.yaml

1. Access the RabbitMQ Management UI:

   kubectl port-forward rabbitmq-definition-0 15672:15672
   

   Then open your browser and navigate to http://localhost:15672.

To login, use the default credentials generated by the operator thats stored in a Kubernetes secret:

kubectl -n rabbitmq-system get secret definition-default-user -o jsonpath="{.data.username}" | base64 --decode
kubectl -n rabbitmq-system get secret definition-default-user -o jsonpath="{.data.password}" | base64 --decode

1. Import predefined exchanges, queues, and bindings using the definitions found here: https://github.com/rabbitmq/cluster-operator/tree/main/docs/examples/import-definitions

kubectl create configmap definitions --from-file='definitions.json=definitions.json'

This will create the exchanges, queues, and bindings needed for this project. Optionally, you can modify the definitions.json file to customize the setup.

1. Expose RabbitMQ outside the cluster (optional): In case you want to access RabbitMQ from outside the cluster, you can expose it using a LoadBalancer or NodePort service. For local clusters like Minikube or K3s, you can use NodePort:

   kubectl expose pod rabbitmq-definition-0 --type=NodePort --name=rabbitmq-service
   

   This is required if you want to run the test client outside the cluster (e.g., on your local machine or docker container on a different network).

2. Build and run the test client: docker build -t testclient:latest testclient

3. Run the test client container with appropriate environment variables: docker run --rm \ -e HOST=rabbitmq.example.com \ -e PORT=5672 \ -e USERNAME=myuser \ -e PASSWORD='mypassword' \ -e SCRIPT=log-consumer.py \ testclient:latest Replace rabbitmq.example.com, myuser, and mypassword with your RabbitMQ host, username, and password.

4. Monitor logs and messages: You can monitor the logs of the consumer service to see how it processes incoming log events. You can also use the RabbitMQ Management UI to monitor message flow, queue lengths, and other metrics.