Table of Contents
Introduction
In the analytics domain, fast and reliable storage is an important aspect for businesses to handle a large amount of data. There are different types of data storage including RDBMS, NoSQL, data lake, data warehouse, and graph database. Among these, the most widely used is RDBMS that powers various systems and applications of companies of all sizes. RDBMS is easy to use and straightforward to understand thanks to its table-based (or column-based) data format. However, when it comes to generating analytical data reports, RDBMS may not be the best option.
If your monitoring tasks require processing large amounts of data quickly, you need a different database such as ClickHouse. In this article, we will discover how we can utilize Clickhouse and integrate it into MetricFire’s solutions. Before we begin, check MetricFire. MetricFire offers comprehensive monitoring solutions with hosted software such as Hosted Graphite. You can gain in-depth insight into your IT resources with minimal configuration.
If you want to learn more, Book a demo with us, or sign on to the free trial today.
What is ClickHouse?
ClickHouse is an open-source column-oriented DBMS specialized in OLAP - Online Analytical Processing. Using its main feature, users can use standard SQL queries to create analytical reports in real-time. The first is based in California, the U.S. They have two major product streams - ClickHouse cloud and open-source ClickHouse.
ClickHouse cloud
One of the advantages of using open-source software is cost-saving. When you want to use it, you do not have to pay any up-front cost to use it. However, everything from installing to maintaining the software is on your own. These are demanding tasks on top of actual analytical work that users perform every day. The same goes for the ClickHouse database software. To ease the burden, ClickHouse offers a cloud-based solution that takes care of the maintenance part so that companies can just focus on generating insights.
Open-source ClickHouse
ClickHouse offers a free open-source version to users. This option can be suitable for those who want to first try the solution before making a long-term commitment. ClickHouse can be both vertically and horizontally scaled. The software delivers fast query results and async replication that can be deployed in multiple data centers. To fully utilize these features, it requires a lot of effort.
Integrating ClickHouse with MetricFire
If you plan to use ClickHouse for monitoring, consider integrating ClickHouse into Graphite - also an open-source monitoring software.
What is Graphite?
Graphite is an open-source monitoring software that provides reliable performance whether your stack is on affordable hardware or cloud infrastructure. Businesses use Graphite to track the performance of their digital assets such as websites, applications, network servers, and more. It is known to provide a scalable solution for real-time graphing. You can write an application that digests numeric time-series data and sends it to Graphite’s processing backend - Carbon, which stores the data in its database. Then, you can visualize the data on Graphite’s web interfaces.
Major Features
Graphite is known for the following major features.
- Simple architecture and great performance.
- Time-series metrics in intuitive graphs and charts.
- Relatively easier to learn compared to other monitoring tools.
- Big community support and is widely used.
- Suits diverse use cases such as analytics, DevOps, prediction, and more.
Hosted Graphite by MeticFire
As pointed out, open-source software requires constant attention and effort from users. MetricFire offers a hosted Graphite to allow people to easy-start and maintain the monitoring software. MetricFire configures, installs, and maintains Graphite, while you can use it as a web application without having to worry about its performance.
Key benefits of using hosted Graphite from MetricFire:
- Data availability: MetricFire provides permanent access to your data, which you can export anytime.
- Affordable price: There are various tariff plans, among which you will choose the one that suits you.
- Saving time and money: There is no need to invest in deploying and running monitoring systems.
- Reliable support: We are always happy to answer all questions regarding working with MetricFire.
- Redundancy storage: Graphite’s default storage is file-based and antiquated. MetricFire offers 3 times the redundant storage for seamless scaling and better data protection.
- Control by APIs: The APIs that MetricFire provides let you control and automate the resources of Hosted Graphite.
- Tagged metrics: Hosted Graphite stores data using tags that enable viewing and organizing metrics with data views.
Hosted Graphite keeps all the benefits of open-source Graphite and further enhances the tool with the built-in agent, team accounts, granular dashboard permissions, and integrations to other major platforms and services such as AWS, Heroku, logging tools, and more.
How to integrate ClickHouse into Hosted Graphite
Combining ClickHouse and Graphite, you can build an efficient analytical and monitoring infrastructure. To integrate ClickHouse into Hosted Graphite by MetricFire, you need to update the configuration of ClickHouse.
To install ClickHouse, follow the steps described in the links below.
After installation of ClickHouse, you can try to install open-source Graphite. However, installing the software on your own can be challenging. Consider using hosted Graphite by MetricFire for an easy start.
Once you finish setting up Graphite, update the ClickHouse configuration for integration.
Find and open the config XML file of ClickHouse located at:
/etc/clickhouse-server/config.xml
Append the additional config parameters like below:
<graphite>
<host>192.168.74.150</host>
<port>2003</port>
<timeout>0.1</timeout>
<interval>60</interval>
<root_path>one_min_cr_plain</root_path>
<metrics>true</metrics>
<events>true</events>
<asynchronous_metrics>true</asynchronous_metrics>
</graphite>
<graphite>
<host>192.168.74.150</host>
<port>2003</port>
<timeout>0.1</timeout>
<interval>1</interval>
<root_path>one_sec_cr_plain</root_path>
<metrics>true</metrics>
<events>true</events>
<asynchronous_metrics>false</asynchronous_metrics>
</graphite>
Setting variables
- host – your Graphite host IP address.
- port – your Graphite port - 2003 is the default. Graphite has several open ports:
- 80 Nginx
- 2003 carbon receiver - plaintext this one should receive data from ClickHouse
- 2004 carbon receiver - pickle
- 2023 carbon aggregator - plaintext
- 2024 carbon aggregator - pickle
- 8080 Graphite internal gunicorn port (without Nginx proxying).
- 8125 statsd
- 8126 statsd admin
- interval – (in seconds) data sending interval from ClickHouse to Graphite
- timeout – (in seconds) data sending timeout
- root_path – Graphite prefix.
- metrics – should data from system_tables-system.metrics table be sent.
- events – should data from system_tables-system.events table be sent.
- asynchronous_metrics – should data from system_tables-system.asynchronous_metrics table is sent.
The <graphite> clauses can be added to define different data-sending rules. The two sets of <graphite> are only for a sample and it does not have to be two sets.
How you can use Clickhouse with MetricFire
When the configuration is completed, your monitoring data will flow into your Graphite cluster. Go to the Graphite dashboard by accessing http://host/dashboard. When things are normally set, you will have entries with prefixes that are defined in your ClickHouse configuration.
<root_path>one_sec_cr_plain</root_path>
...
<root_path>one_min_cr_plain</root_path>
This whole integration setup can be complicated if you try it by yourself. When you use MetricFire, you can get good support from experts.
Using ClickHouse with hosted Graphite is not the only integration you can easily do when you are with MetricFire. ClickHouse can be integrated with Grafana.
Grafana is also an open-source analytics and interactive visualization web application. You can customize charts, graphs, and alerts. On top of its visualization features, you can design its back end. With its extensive features, you can create customized dashboards and run queries to generate the metrics you want.
If you want to connect Grafana to ClickHouse, you can try to follow the steps on the ClickHouse website. Again, using open-source Grafana may not be straightforward for first users.
MetricFire, to remove the inconveniences, provides Grafana as a Service. Using this service, you don’t have to worry about maintenance and focus on what matters the most to your team. The hosted Grafana comes with all the good features from Grafana and, on top of them, you can benefit from:
- owning your own data
- saving cost with MetricFire’s flexible pricing plans
- getting expert engineering support
With these additional supports, you can start creating metrics to monitor important aspects of load balancers including latency, connections, 4xx and 5xx status codes, and target resource status.
Conclusion
In this article, we discovered the concept and use cases of ClickHouse and the advantages of integrating the database software with MetricFire’s hosted Graphite and even Grafana. You can access all these benefits and the MetricFire products with minimal configuration to gain in-depth and complete insight into your IT resources.
If you would like to learn more about these services, feel free to book a demo to speak with one of our experts or sign up for the free trial today.
