Monitoring RabbitMQ With Prometheus and Grafana

 

Introduction 

Prometheus and Grafana are powerful monitoring solutions. Combined together, you can build insightful monitoring metrics and display analysis on a beautiful dashboard. Among many external systems and applications, you can also monitor RabbitMQ with both toolkits. To monitor your data efficiently, you can try MetricFire which provides hosted solutions for Graphite (our Prometheus alternative) and Grafana. If you would like to learn more about it, please book a demo with us, or sign on to the free trial today.

     

Key Takeaways

1. Prometheus and Grafana are powerful monitoring solutions that, when combined, can provide insightful monitoring metrics and beautifully displayed analysis on a dashboard.
2. The RabbitMQ management plugin provides an efficient way to manage and monitor RabbitMQ nodes and clusters through HTTP-based APIs.
3. External monitoring solutions like Prometheus and Grafana are recommended for long-term monitoring of RabbitMQ.
4. Prometheus and Grafana can be used to monitor RabbitMQ effectively, with pre-configured dashboards available for visualization.
5. Other monitoring tools like SolarWinds and Datadog are also options for RabbitMQ monitoring.

 

What is RabbitMQ?

RabbitMQ is an open-source message queueing broker that has gained popularity across the globe. RabbitMQ offers a way to control queueing resources by distributed and federated configurations. This makes RabbitMQ highly scalable and available. The queue broker supports multiple messaging protocols, messaging queueing, delivery acknowledgment, flexible routing to queues, and multiple exchange types. It supports almost all major programming languages including Java, .NET, PHP, Python, JavaScript, Ruby, and Go, to name but just a few.

    

How RabbitMQ works

RabbitMQ is fundamentally a messaging broker that delivers messages from a publisher to a receiver. If you are familiar with any other queueing system, you can quickly grasp how RabbitMQ works. To understand RabbitMQ, you first need to get familiar with the four components below.

• Producer: it writes a message and sends it to Exchange. Instead of sending it directly to Queue, in RabbitMQ, you send it to Exchange. You can define an Exchange type such as direct, topic, or fanout. 
• Exchange: depending on the Exchange type you set, it sends messages to a single or multiple queues as requested.
• Queue: a queue is where RabbitMQ stores messages to be consumed. In RabbitMQ, messages are ordered in FIFO (First In, First Out) format. However, ordering is not guaranteed for sharded and priority queues.
• Consumer: a message finally arrives at the Consumer. To receive RabbitMQ messages, you need to subscribe to a queue.

    

Among these four, it is worth looking at the three Exchange types in-depth since they will give you a more complete picture of how RabbitMQ works.

     

      

Exchange types

There are three exchange types - direct, topic, and fanout. Depending on the type you choose, you can send a message to all queues or specific queues.

Fanout

Fanout Exchange broadcasts messages to all the registered queues. When the Producer sends a message, a binding value can be defined to indicate to which queues messages are sent. But, since it simply sends all messages to all queues, a binding value becomes ignored.

Direct

You can use Direct Exchange to send a message to specific queues. You will define a binding key and a message is sent to the queues that match the binding key.

Topic

Topic Exchange works similarly to Direct Exchange but you can have more flexible controls over routing to queues. For example, let’s say Q1 is created to hear all about new members signing in and Q2 wants to receive new members whose gender is female. When a new message is created to handle a new male member, it will be sent to Q1 only. When it is a new female member, it is sent to both Q1 and Q2.

          

Key RabbitMQ monitoring metrics

Designing and implementing a queueing system with RabbitMQ is one thing and monitoring it is another. In terms of monitoring, there are specific metrics you want to focus on. To monitor those metrics with ease, you can use a monitoring tool such as SolarWinds or Graphite. Before you choose a tool, it is important to understand the monitoring metrics. There are three RabbitMQ monitoring metrics available.

• Node metrics
• Queue metrics
• Cluster-wide metrics

     

Node metrics

The runtime environment of RabbitMQ was built on a programming language called Erlang. A runtime system runs within each node. To monitor the usage of resources, you need to look at the node. The resource metrics you can monitor include memory, disk space, file descriptors, sockets, and garbage collector, among many others.

Queue metrics

A queueing stage is right before a message reaches a consumer. Monitoring queues, therefore, are important to ensure messages arrive successfully at the receiver. You can monitor various items such as the total number of messages, the number of messages ready for delivery, messages published recently, message publication rate, and more, by looking at queue metrics.

Cluster-wide metrics

A cluster consists of one or more nodes. This means that by monitoring a cluster, you can look at high-level performance across the entire RabbitMQ stack. Some examples of high-level metrics are interaction among nodes, cluster link traffic, and detected network partitions. Furthermore, you can use these cluster-wide metrics to check the total number of connections, channels, queues, consumers, and messages across all your nodes.

          

RabbitMQ Built-in CLI

In addition to the built-in UI, RabbitMQ comes with CLI for users to interact with various parts of RabbitMQ. You can use the web UI to check the status and control RabbitMQ in a user-friendly way. But, if you want to have more control over your RabbitMQ resources, you need to check out CLI.

When to use CLI

RabbitMQ comes with multiple command-line tools, and you can use them to automate tasks. Using some of the CLI commands, you can mimic and automate the tasks you can do with the RabbitMQ UI. You can gain various insights using the CLIs and build your own system. They can be used to integrate RabbitMQ into other more powerful monitoring tools such as Prometheus and Graphite.

Types of CLI tools and use cases

RabbitMQ ships with multiple CLI tools. Each of these tools is responsible for different tasks. Check out the list below to learn more about the role of each CLI.

• rabbitmqctl: to manage services and execute general operational tasks.
• rabbitmq-diagnostics: runs diagnostics and health checks across the RabbitMQ stack.
• rabbitmq-plugins: to interact with RabbitMQ plugins.
• rabbitmq-queues: to maintain tasks on your queues, especially quorum queues.
• rabbitmq-upgrade: to maintain upgrade tasks.

          

RabbitMQ Management Plugin

Among many plugins of RabbitMQ, the management plugin for monitoring is one of the most noteworthy. This section introduces the plugin for efficient monitoring.

   

What is the management plugin?

The RabbitMQ management plugin allows users to manage and monitor RabbitMQ nodes and clusters through HTTP-based APIs. You can use both web-based UI and rabbitmqadmin, the command-line tool for management. The plugin collects and aggregates data across the RabbitMQ system. These collected metrics are then shown to the UI and monitoring systems. The management plugin is the go-to source for RabbitMQ monitoring tasks and maintenance.

    

How to use the management plugin

Depending on your use case, you can use its web UI rabbitmqadmin CLI or both. The management plugin is part of the RabbitMQ distribution. Before it can be used, you need to enable it like any other built-in plugin. You can use the rabbitmq-plugins command below.

rabbitmq-plugins enable rabbitmq_management

    

Once it's enabled, you can use it via the web-based management UI or rabbitmqadmin.

          

Management UI

On your web browser, you can access the UI at:

http://{node-hostname}:15672/

For example, depending on your privileges, you can access a specific node either with http://warp10.local:15672/ or http://localhost:15672/. The management UI is a single-page application that allows you to control exchanges, queues, bindings, users, virtual hosts, and user permissions. You can also monitor nodes, queues, exchanges, resources, and more.

Although the built-in UI offers monitoring metrics and visualization, RabbitMQ recommends that, for a long-term solution, you will want to consider external solutions such as Prometheus and Grafana.

     

rabbitmqadmin

rabbitmqadmin is a set of CLI commands to let you control RabbitMQ systematically. Using rabbitmqadmin, you can perform some of the tasks you can do on the management UI. If you want to achieve automation, the rabbitmqadmin CLI can be more convenient and flexible. However, if you want to use it within your own program, rabbitmqadmin is not suitable. Try an HTTP API client library rather than an external program.

     

RabbitMQ monitoring tools

MetricFire provides hosted solutions for Graphite (a Prometheus alternative) and Grafana which are powerful monitoring tools. With both, you can extract in-depth insights and expose analysis into a beautiful dashboard. If you would like to learn more about it, please book a demo with us, or sign on to the free trial today. In this section, we will cover three common RabbitMQ monitoring tools.

    

Prometheus & Grafana

Prometheus is a monitoring toolkit and Grafana is a visualization system. They can both be used to monitor various applications, systems, platforms, CI/CD pipelines, Google Cloud Platform, infrastructure, Elasticsearch, and many more. It becomes more powerful when you combine the two toolkits and use them for long-term metric collection and monitoring for your RabbitMQ clusters. Note that although the RabbitMQ management UI comes with visualization, it is not suitable for a long-term monitoring solution. With that being said, RabbitMQ, as of 3.8.0, comes with built-in Prometheus and Grafana support. 

For Prometheus, you can use the rabbitmq_prometheus plugin that exposes all RabbitMQ metrics on a dedicated TCP port in Prometheus text format. These metrics give you complete insights into the status of RabbitMQ nodes and runtime. For Grafana, RabbitMQ provides a pre-configured set of Grafana dashboards that visualize the metrics from Prometheus.

    

SolarWinds

SolarWinds is a company that provides a wide range of monitoring software. Among its products, SolarWinds Server & Application Monitor (SAM) has built-in monitoring templates for RabbitMQ nodes. You can customize the templates and create your own performance metrics such as memory usage, free disk space, and uptime. SolarWinds RabbitMQ SAM has numerous monitoring templates available. RabbitMQ SolarWinds can automatically detect your environment and is operational within minutes of installation. The software makes monitoring simple and consolidates monitoring results from different devices, servers, and applications into a single view. SolarWinds can be compared to another similar monitoring tool called Graphite. Check the comparison article for more details.

      

Datadog

Datadog is a monitoring service provider for servers, databases, tools, and services via its SaaS analytics platform. Among their products, Datadog Agent helps you collect events and metrics from your applications and sends the results to the Datadog cloud. On the Datadog platform, you can analyze and visualize the metrics you sent. Using the Datadog Agent, you can track and monitor clusters, nodes, queues, messages, connections, and more.

    

Integration with Prometheus and Grafana

Prometheus and Grafana are recommended by RabbitMQ. Thus, you may want to consider starting with both tools before giving any of the others a try.    

 

Major benefits of integration with MetricFire

When you use MetricFire to integrate Graphite and Grafana, you can gain numerous benefits from both tools.

Grafana

• intuitive dashboard design and easy-to-create various dashboards
• simple to customize your dashboard by editing rows, panels, zooms, graphs
• Grafana-as-a-Service is included in all plans regardless of the size

MetricFire as a platform

• Easy installation and setup
• No need to maintain infrastructure and services
• Quick and comprehensive technical support
• Specialized in Graphite and Grafana
• Trusted by big organizations across the globe

   

Final thoughts

In this article, we explored RabbitMQ and why and how Prometheus and Grafana are perfect solutions for RabbitMQ monitoring. For your best integration experience, MetricFire provides a Hosted alternative to Prometheus which includes storing your data for two years and a complete tool. Plus, MetricFire offers Grafana-as-a-Service for intuitive data visualization. You can use the products with minimal configuration to gain in-depth insight into your environment. If you would like to learn more about it, please book a demo or sign on to the free trial now.