How to Orchestrate microservices with KubeMQ Messaging Platform
In these days in which technologies have advanced at great speed, it is expected that they will be increasingly reliable and robust in information processing and with a large volume, an example would be tweets or publications on social networks, video streaming, payments, etc. We have established a very high level of requirement and which must be met otherwise the user will feel frustrated and abandon the product or service. The systems that must survive this test must be designed and implemented in such a way that they support thousands or millions of http requests/transactions/images/videos and store all this information for read it later. Have third-party software integrations where appropriate and know how to respond in case of not having communication with them. Every day the platforms provide more services that surround the applications to know not only what they process but how long they did it, they have some latency, they disconnected from a certain service or the execution failed due to lack of any resource, in short, there are several considerations what you have to care your an application living today.
I have recently collaborated on projects that process large amounts of information, in this case a web interface such as REST is exposed and it must be processed in seconds and ensure that they are persisted in a database, a very simple way.
To solve this problem most architects focus their architectures on Brokers or Event Bus that ensures that the information will be delivered to whoever requests it, will have replication in case of any failure and the information will be able to live temporarily. Components like these there are too many on the market, one of the best known is Apache Kafka, Apache Spark, Google PubSub, AWS Kinesis.
We live in the era of the cloud and where Kubernetes is the king of container platforms and have a native integration component should not be lacking within its ecosystem that lives and takes advantage of the benefits that Kubernetes provides.
In this post we are going to make a simple application that uses KubeMQ as its communication center. We will also have a publisher and subscriber made with Micronaut to deploy in Kubernetes, in my case I will use a Minikube.
Diagram of interact communication
In the diagram above, the flow begins with the web clients that send information to our exposed service, within kubernetes we have a load balancer that receives the requests and routes them to the publisher microservice so that a Message ID is assigned and saves the information in a MySQL database and publish the message in the KubeMQ queue that has a FIFO queue configured .The Consumer microservice has to obtain the message (pulling), process it and save it in the database.
Installing KubeMQ on Minikube
I ‘m going to use a local minikube installation and deploy the KubeMQ cluster. If you enter the page you will find the section where you can install it in the way they suggest. https://kubemq.io
In the documentation I found the steps to install on Minikube, I suggest you register to obtain the token that allows you to use it. https://docs.kubemq.io/getting-started/quick-start
Installing MySQL on Kubernetes
Follow the next tutorial for installation. https://kubernetes.io/docs/tasks/run-application/run-single-instance-stateful-application/
Publisher/Consumer Application
If you want to do the example from scratch you will need to create the project and configure it. I will use the Micronaut framework but you can choose the framework you like, most of them already have staters and configuring them is very easy.
Let’s go to the Micronaut Launch page. https://micronaut.io/launch/ .The page offers us the options to generate the project and download it. I leave you the configuration link that I use where the dependencies and the project preview already come. The Publisher microservice will use the following settings:
We need to add the KubeMQ dependency for java, in the gradle.build file (using gradle) we add the following line in dependencies:
dependencies { implementation group: 'io.kubemq.sdk', name: 'kubemq-sdk-Java', version: '1.0.3' ....}
The first java class that we must build for our application is the Publisher, the main thing it has to do is expose a REST API interface to consume the messages, then assign a Message ID to the request and save it in the Database and then publish it in the queue. To publish the message in the queue we need to generate a class that acts as a producer, I will put the controller class that is the one that will send the messages.
I ‘m using Micronaut Data Repository to persist the message information, it is very convenient for those who have already used some Repositories pattern that facilitates the data layer and improves readability and configuration. The sql scripts for tables I leave in the GitHub repository.
To configure the queue you can use a class of type factory to instantiate the Queue object that represents the queue in KubeMQ.
The configuration of the connection uri to the data base, user and password already comes with a template that is included by Micronaut when creating the project (application.yml) if you need to configure any more properties.
We are going to generate a new project to create our consumer, for this I use again the Micronaut stater page, I leave you the link. Don’t forget to add the kubemq-sdk-Java dependency.
Now we will create a consumer class to extract the messages, I’ve chosen to managed them with the Micronaut Server Events. So far I haven’t found an out-of-the-box solution like a listener annotation to listen to messages coming to the queue.
Using the next command on root directory location for packing a fat jar with all dependencies :
$ ./gradlew assembleOnce the producer and consumer have been made, we need to create the docker image that will be used to deploy the microservices on Minikube. We will need a DockerHub account to save the images publicly (only do this step if you don’t have a container registry).
The docker image is for each of the microservices:
Dockerfile Subscriber
Dockerfile Publisher
# Run the following commands in a terminal at each project level to publish the image on DockerHub.$ docker login
$ docker build -t username/image-name:tag .
$ docker push username/image-name:tag
Once we have the images in the registry, we need to deploy them in the Kubernetes cluster. Here I leave the manifests for the microservices. Don’t forget install kubectl tool for communicate to Kubernetes cluster.
Publisher Manifest
Consumer Manifest
If you are organized Kubernetes objects, you can use namespaces and deploy them with the following commands.
$ kubectl apply -f application-resources.yml -n <namespace>The MicroservicePublisher service is not exposed so that we can see it from our machine, so in the case of minikube we have to open a tunnel with the minikube tunnel command.
Finally we will generate a test data set to send requests to our MicroservicePublisher service and this queues the information in KubeMQ where the MicroserviceConsumer will treat the message.
$ for i in {1..100}; do curl — location — request POST ‘http://localhost:8080/messages' \ — header ‘Content-Type: application/json’ \ — data-raw ‘{ “message”:”Hello from KubeMQ” }’; doneConclusion
KubeMQ helps those of us who are already working with platforms like Kubernetes to integrate more easily and natively, allowing developers to focus their efforts on knowing, developing and exploring integrated projects. In addition, the frameworks or languages that we use must be the most decoupled from the platform, in my case Micronaut Java framework allows me to have portability and take the knowledge from one framework to another without having a long learning curve. Developing high demand and concurrency applications will always be a challenge for any software engineer, however we are already living a reality where solutions are increasingly easy to test and improve.
