Table of Contents
Introduction
Graphite is a very popular enterprise monitoring tool. This article will address the common issues that occur while setting up a Graphite instance, and how they can be avoided. We will assume readers have already become acquainted with Graphite, but if you’re interested to learn about the basics of Graphite, check out our articles on the Architecture and Concepts of Graphite and the Installation and Setup before reading this article. For this article, the key terms you need to know are:
- Carbon - It is the storage server for Graphite. Graphite can run multiple instances of Carbon on a single node to handle the incoming load.
- Whisper - is the name of a database format that Graphite uses to store the data.
We will also evaluate the benefits and costs of setting up your own instance of Graphite vs MetricFire. As operations scale, it can become too time-consuming to run your own Graphite instance, making a hosted version much more feasible.
As you’re going through this article, try out MetricFire with our free trial account. You will be able to send metrics directly and test them to see how it works.
Key Takeaways
- Graphite is a popular enterprise monitoring tool, but setting up a Graphite instance can come with common issues that can be avoided with the right knowledge.
- Installing Graphite on a production machine running CentOS/Debian is recommended for optimal performance, and manual steps are required for the setup.
- SQLite3 is not recommended for production systems, so it's advised to install and configure PostgreSQL for Graphite to store its data.
- Graphite and Carbon won't start automatically on boot, so changes need to be made in the configuration file to enable automatic startup.
First steps in setting up Graphite
In our installation tutorial, we used docker images of Graphite to run and demo the Graphite application. However, to achieve the best performance, Graphite needs to be installed on a production machine running either CentOS / Debian. This requires following a lot of manual steps to get the Graphite up and running.
So, let’s get started. In this example, I will be using Ubuntu 18.04 to get the Graphite up and running.
- Update the local package by running the command:
sudo apt-get update
- Install the Graphite web application and Graphite Carbon:
sudo apt-get install graphite-web graphite-carbon
During the installation process, the prompt will appear asking if you want to keep the Carbon database if you ever decide to uninstall and reinstall. You can choose no for this prompt.
Although technically now you have installed Graphite and Carbon, there are various caveats that we are going to address now one by one.
Using SQLite3
The Graphite web application uses SQLite3 database files by default to store its data. This is not at all recommended for production systems since SQLite3 is not a full-fledged database system. Hence, we will install and configure PostgreSQL to let Graphite store the data there.
sudo apt-get install postgresql libpq-dev python-psycopg2
This will install PostGreSQL to which Graphite will connect to and store the data.
Now, we will create the user Graphite and database Graphite in the database.
madhur@madhur-Virtual-Machine:~$ sudo -u postgres psql
psql (10.12 (Ubuntu 10.12-0ubuntu0.18.04.1))
Type "help" for help.
postgres=# create user graphite with password 'password';
CREATE ROLE
postgres=# create database graphite with owner graphite;
CREATE DATABASE
postgres=# \q
Now, we need to tell Graphite to use our new database server settings instead of the default database which is SQLite. Change the values in /etc/graphite/local_settings.py from the following default values to the new database settings we just created.
Old Values:
DATABASES = {
'default': {
'NAME': '/var/lib/graphite/graphite.db',
'ENGINE': 'django.db.backends.sqlite3',
'USER': '',
'PASSWORD': '',
'HOST': '',
'PORT': ''
}
}
New Values:
DATABASES = {
'default': {
'NAME': 'graphite',
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'USER': 'graphite',
'PASSWORD': 'password',
'HOST': '127.0.0.1',
'PORT': ''
}
}
Run the command “sudo graphite-manage migrate” to let Django create tables in the new database. You should see the following output:
madhur@madhur-Virtual-Machine:~$ sudo graphite-manage migrate
Operations to perform:
Apply all migrations: admin, auth, contenttypes, sessions, tagging
Running migrations:
Applying contenttypes.0001_initial... OK
Applying auth.0001_initial... OK
Applying admin.0001_initial... OK
Applying admin.0002_logentry_remove_auto_add... OK
Applying contenttypes.0002_remove_content_type_name... OK
Applying auth.0002_alter_permission_name_max_length... OK
Applying auth.0003_alter_user_email_max_length... OK
Applying auth.0004_alter_user_username_opts... OK
Applying auth.0005_alter_user_last_login_null... OK
Applying auth.0006_require_contenttypes_0002... OK
Applying auth.0007_alter_validators_add_error_messages... OK
Applying auth.0008_alter_user_username_max_length... OK
Applying sessions.0001_initial... OK
Applying tagging.0001_initial... OK
Applying tagging.0002_on_delete... OK
madhur@madhur-Virtual-Machine:~$
Automatic start-up
If your production server gets restarted for some reason, the Graphite and Carbon won’t start automatically. For it to start automatically, we need to make some changes in /etc/default/graphite-carbon: Change the value of CARBON_CACHE_ENABLED to true.
This will ensure that Graphite and Carbon are started automatically on boot.
Log Rotation
By default, carbon doesn’t do any log rotation. That means the hard disk is at risk of filling up unless you clean your log files manually or enable log rotation.
1. Edit the file /etc/carbon/carbon.conf
2. Change the value of ENABLE_LOGROTATION to True
To get the graphite web interface up and running, we need to install Apache
sudo apt-get install apache2 libapache2-mod-wsgi
To enable the graphite on the Apache web server, run the following commands
madhur@madhur-Virtual-Machine:~$ sudo a2dissite 000-default
Site 000-default disabled.
To activate the new configuration, you need to run:
systemctl reload apache2
madhur@madhur-Virtual-Machine:~$ sudo cp /usr/share/graphite-web/apache2-graphite.conf /etc/apache2/sites-available/
madhur@madhur-Virtual-Machine:~$ sudo a2ensite apache2-graphite
Enabling site apache2-graphite.
To activate the new configuration, you need to run:
systemctl reload apache2
madhur@madhur-Virtual-Machine:~$ sudo service apache2 reload
Once this is done, if you go to http://localhost, you should finally see the Graphite Web application:
Troubleshooting Graphite Installation
If you face any issues while setting up Graphite and Carbon for production:
- Make sure you are using the supported operating systems such as RHEL, CentOS, Debian, and Ubuntu.
- Check to make sure the correct versions of Python, Django, Apache web server, and Database engine are being used. Most of the issues arise due to incompatibility between these running parts.
- Look for existing issues on the following repositories:
https://github.com/graphite-project/carbon/issues
https://github.com/graphite-project/graphite-web/issues
Most of the time, the problem you are facing has been faced by someone else previously as well.
MetricFire vs Self Hosted Graphite
If you have made it to the end of the article and are able to successfully run Graphite, congratulations. If you face any issue, there are numerous troubleshooting articles on the Internet to get it up and running, which we will not replicate here. You can also reach out to MetricFire and see if a hosted version of Graphite is better suited
In the rest of the article, we will analyze the benefits of using MetricFire vs using your own Graphite. MetricFire helps in the following ways:
Out-of-the-box scaling
In the article above, we haven't really touched on scaling Graphite, which will be required at any production-grade deployment. It requires running multiple instances of Carbon in the same host and also across multiple hosts in a cluster. Setting that up isn’t trivial and requires a lot of administrative manpower and monitoring.
Automatic backup of data
MetricFire offers automatic backup of data to make sure if something goes wrong, data can be recovered quickly and business isn’t affected. Deploying a custom backup solution in self-hosted Graphite is an expensive exercise and requires deploying additional manpower and storage to take care of backups.
24 / 7 on-call Support
MetricFire provides year-round support and automated monitoring anywhere in the world.
Security
MetricFire provides high security using protocols such as HTTPS and TLS to transmit data packets. Every Hosted Graphite account also comes with a unique API key that is used to authenticate all requests. Configuring self-hosted Graphite to provide such security requires a lot of custom configuration to get it up and running.
Automatic Updates
Graphite consists of a lot of moving parts such as Python, Django, Apache Web Server, and database engine. In order to run it at the optimum levels, all these components need to be updated at regular intervals. MetricFire solves these problems for you by applying the latest updates for these components in the background.
The actual technology of MetricFire is slightly different than regular Graphite. The developers have added to the Graphite project to make Hosted Graphite even easier for the users. We outline this topic in our docs here.
Conclusion
We have seen the process of setting up self-hosted Graphite. It doesn’t cover advanced use cases like scaling, security, and backups. Such features are only provided by Hosted Graphite. Self-hosted Graphite requires administration and specialized engineers who are experts in Linux Administration, Graphite, Python, Django, Apache Web Server, and PostgreSQL. For most setups, the cost of running self-hosted Graphite exceeds the cost of using MetricFire.
To try MetricFire, log in to MetricFire's free trial, and start sending metrics today. You can also reach out to us by booking a demo, we're always happy to jump on a call and talk about your company's monitoring needs.
