Table of Contents
Introduction
This article covers such a popular topic as using aggregation rules for metrics. We will learn why it is important to use aggregations and what tools exist for working with them. Also, we will explore all the benefits of using MetricFire to store, process, analyze and monitor your metrics.
If you want to know more about MetricFire's solution, sign up for a free trial today or book a demo with our team.
Why are aggregates useful?
Aggregation rules allow you to combine selected data from different sources of input metrics into a smaller set of output metrics automatically. They provide you with the ability to add several metrics together as they come in, reducing the need to sum a lot of metrics in every URL. Aggregation rules are very useful when you use a large number of input metrics in a query but expect a fast response. Using aggregations reduces the number of output metrics and speeds up the process of creating graphs.
Aggregation rules also allow you to efficiently precompute the most resource-intensive parts of some queries, which can speed up their execution time. You can create aggregation rules within the configuration file aggregation-rules.conf located in /opt/graphite/conf/. In this case, all changes you make to the file take effect automatically.
Of note, the carbon-aggregator service must be running for this. It listens to buffered metrics over time and after a defined time interval, it aggregates the values as per the defined function and returns a single value which is then saved to Whisper. Alternatively, you can use MetricFire’s Aggregation Rules API or Graphite Data Views features to create aggregation rules. We will discuss both of these methods in more detail below.
Aggregate your Graphite metrics using the Aggregation Rules API
Graphite is an open-source tool for monitoring systems, services, and application metrics. It allows you to store, retrieve, share and visualize time-series data. Graphite is a monitoring system that allows you to withstand heavy loads without degrading performance. It consists of several subsystems that allow you to collect metrics from different sources and transfer them into a database. Graphite also provides the ability to use mathematical functions to process and analyze stored data in real-time. Its math functions can be used to create aggregation rules.
MetricFire offers an Aggregation Rules API for aggregating your Graphite metrics. This allows you to create new aggregation rules, delete them and view the list of created rules. Let’s look at some examples of how to use the Aggregation Rules API.
Creating Aggregation Rules
To create an aggregation rule, you need to execute a request POST /v1/aggregates/. Two required parameters must be passed to the request body:
- pattern - This is a regular expression or node pattern to match received metrics.
- output - The output of the metric. When a metric is matched against a pattern, those metrics will be matched against that output and appear in your account.
Consider the example request:
curl -H "Content-Type: application/json" -X POST -d \
'{"pattern": "servers.*.requests.<response_code>", "output": "servers.all.requests.<response_code>"}' \
"https://YOUR-API-KEY@api.hostedgraphite.com/v1/aggregates/"
Metrics that match the pattern servers.*.requests.<response_code> (e.g. servers.server-0001.requests.200, servers.server-0002.requests.200) will be combined into a new group: aggregates.servers.all.requests.200.
Getting a list of aggregation rules
You can get a list of all rules by executing request GET /v1/aggregates/. If you want to get one specific aggregation, pass the optional parameter aggregation_id to this request: GET /v1/aggregates/<aggregation_id>/.
Removing aggregation rules
To delete the rule you need to use request DELETE /v1/aggregates/<aggregate_id>/ and specify the id of the rule.
Editing Aggregation Rules
To edit an aggregation rule, delete the old rule and create a new one. Rules are stateless, so you can safely delete them and create new ones.
What is Graphite Data Views?
MetricFire's Hosted Graphite automatically saves various statistical representations of your data by default. This allows you to ensure that you can quickly view dashboards with your metrics at the resolution that best suits you.
Let’s take a look at some of the views that are saved by default.
- :avg - average value.
- :sum - the sum of the data points acquired during the current chart resolution period.
- :sumrate - speed per second. This value is calculated by dividing the sum by the number of seconds between data points.
- :sum30 :sum60 :sumN - provides the amount received in the given number of seconds, calculated from :sumrate. For example, ”:sum30” provides the sum in 30 seconds.
- :min - minimum value.
- :max - maximum value.
- :obvsrate - per second rate, obtained by dividing the observation by the number of seconds in the collection period.
- :90pct :95pct :99pct etc - arbitrary percentage data. Available values are from 01 to 99.
Data Views and Graphite Functions
Graphite functions are used to process one or more data series and transform them into a new data series. Graphite provides an extensive library of statistical and transformative rendering functions that can manipulate serial data streams into critical indicators of system activity. One of the most important features of the Graphite rendering API is the ability to chain functions together, allowing engineers to create deep levels of detail.
The Graphite API provides the ability to add custom processing functions. Custom functions are packaged as Python modules and are loaded by Graphite when placed in the /opt/graphite/webapp/graphite/functions/custom folder. Graphite functions cannot change the data views that are used in a query. The functions allow you to change the resolution of the graph, but they cannot change the resolution of the data series. This means that changing the zoom level of the graph does not change the interval of data output, but may change the number of available data points.
The advantages of MetricFire's Hosted Solution
MetricFire is an open-source cloud-based solution that provides different tools for monitoring infrastructure, systems, and applications. MetricFire is scalable from hundreds to millions of unique metrics. Our Hosted Graphite solution allows you to collect, store, and work with aggregated metrics. MetricFire also provides advanced support for data backups, tagged metrics, and reliable engineering support. MetricFire does all the work of installing, configuring, and maintaining Graphite while you can use it as a web application. This allows you to save time and money on running your Graphite in-house.
Book a demo with the MetricFire team or create a free trial MetricFire account to learn how to integrate your system with MetricFire.
Let's list other important benefits of using MetricFire’s solutions.
- Full control over your data without depending on a provider.
- Ability to make changes and automation using built-in APIs.
- Ability to collaborate and share your work with your teams or restrict access.
- Affordable and transparent pricing with the flexibility to work within your budget.
- Transparency of work on all aspects of its activities to monitor the SaaS system.
- Ongoing reliable support from MetricFire engineers via live chat or video conference.
Conclusion
In this article, we have learned why it is important to use aggregation rules for metrics, how to use Aggregation Rules API to work with aggregations, and how Graphite Data Views simplify the process of analyzing and visualizing metrics. We also have explained why you should use MetricFire's Hosted Graphite solution and looked at all of its benefits.
If you have any additional questions, the MetricFire team is always happy to help you. Check up for MetricFire free trial or book a demo with our experts and get more detailed information on how to work with MetricFire.
