Monitor your GitHub Repos with Graphite and Grafana

Introduction

In this article, we will explore the main metrics of a GitHub repository and why it is important to monitor them. We will learn how to get GitHub data in a convenient format, process it, then visualize it for further analysis. Finally, we will analyze the main advantages of using data monitoring tools such as Hosted Graphite and Grafana by MetricFire.

If you have any questions about MetricFire and how to integrate GitHub with it, book a demo with our technical specialists or sign up for a free trial today.

Key Takeaways

1. Monitoring GitHub repository metrics is crucial for effective project management and development. It helps track project activity, user engagement, code changes, and more.
2. Various GitHub metrics can be monitored, including traffic (page views and clones), contributors (commits, issues, and pull requests), and maintenance (issue lifetimes and pull request merging times).
3. The GitHub API provides different endpoints to retrieve repository data, such as Metrics API, Pulls API, Deployment API, Pages API, Commits API, and Release API.
4. Grafana is an open-source visualization tool that allows you to create customizable dashboards for displaying and analyzing data from various sources, including GitHub. It supports different databases and plugins for extended functionality.
5. Grafana offers flexibility in data source selection, collaboration, advanced data filtering, and the ability to create informative graphs and charts.

Monitoring your GitHub data and releases purposes

GitHub is a web service for hosting and collaborating Git repositories. GitHub provides a user-friendly graphical web interface, as well as access control and collaboration features. If you are actively using GitHub to develop your own projects or keeping track of other open-source projects, monitoring your GitHub data and releases is a must. 

Monitoring a GitHub repository allows you to find out how active it is, track the progress of the project, add new features and fix bugs, find out the number of active users, track changes in the code, and much more.

Let’s take a look at some of the main benefits of monitoring your GitHub repository.

1. Effective management of the software development process. GitHub metrics allow you to view what issues each team member is working on, the time it takes to complete each issue, progress, and project statistics.
2. Thanks to the ability to track how users use your project, you can evaluate its popularity and how users react to releases of new features.
3. You can get a complete history of all files and release changes.
4. You can extract important data about your project from the GitHub repository that can help you decide on further project development.

Let’s list the main GitHub repository metrics you can track:

1. Traffic: You can see the total number of project page views, the number of unique visitors, the sites from which the visitors came, the most popular pages, and the number of times the repository was cloned.
2. Contributors: Metrics such as total number of contributors and number of commits per contributor, number of open issues and number of requests pulls, and types of contributions (e.g. commits, bug fixes, comments).
3. Maintenance: This includes metrics such as the average time an issue stays open, whether PR issues are closed, whether outdated issues are closed, and the average time to merge a pull request.

Getting your GitHub data in a time-series format

The GitHub API was created for getting data from a GitHub repository. It is divided into many parts, among which are:

• Metrics API: Allows you to get statistics and traffic for the repository.
• Pulls API: Allows you to view pull requests.
• Deployment API: Allows you to get data about deploy keys, deployments, and deployment environments.
• Pages API: Allows you to collect information about GitHub pages.
• Commits API: Allows you to list, view, and compare commits in a repository.
• Release API: Allows you to view the list and resources of releases.

Let’s look at a few examples of getting data from the GitHub API.

Getting the number of repository clones

To get the total number of clones and breakdown by day for the last 14 days, run the following request:

curl \
  -H "Accept: application/vnd.github.v3+json" \
  https://api.github.com/repos/{owner}/{repo}/traffic/clones

As a result, we get a JSON, which contains information about the creation date of clones, the total, and the unique number of clones.

{
  "count": 173,
  "uniques": 128,
  "clones": [
    {
      "timestamp": "2016-10-10T00:00:00Z",
      "count": 2,
      "uniques": 1
    },
    {
      "timestamp": "2016-10-11T00:00:00Z",
      "count": 17,
      "uniques": 16
    },
    ...
  ]
}

List of deployments

To view a list of all deployments for a selected project, run the following request:

curl \
  -H "Accept: application/vnd.github.v3+json" \
  https://api.github.com/repos/{owner}/{repo}/deployments

You can also filter this request by query parameters.

As a result of the request, we will receive a list of deployments with detailed information about each of them, as well as their creation and update dates.

[
  {
    "url": "https://api.github.com/repos/octocat/example/deployments/1",
    "id": 1,
    "node_id": "MDEwOkRlcGxveW1lbnQx",
    "sha": "a84d88e7554fc1fa21bcbc4efae3c782a70d2b9d",
    "ref": "topic-branch",
    "task": "deploy",
    "payload": {},
    "original_environment": "staging",
    "environment": "production",
    "description": "Deploy request from hubot",
    "creator": {
      "login": "octocat",
      "id": 1,
      "node_id": "MDQ6VXNlcjE=",
      "avatar_url": "https://github.com/images/error/octocat_happy.gif",
      "gravatar_id": "",
      "url": "https://api.github.com/users/octocat",
      "html_url": "https://github.com/octocat",
      "followers_url": "https://api.github.com/users/octocat/followers",
      "following_url": "https://api.github.com/users/octocat/following{/other_user}",
      "gists_url": "https://api.github.com/users/octocat/gists{/gist_id}",
      "starred_url": "https://api.github.com/users/octocat/starred{/owner}{/repo}",
      "subscriptions_url": "https://api.github.com/users/octocat/subscriptions",
      "organizations_url": "https://api.github.com/users/octocat/orgs",
      "repos_url": "https://api.github.com/users/octocat/repos",
      "events_url": "https://api.github.com/users/octocat/events{/privacy}",
      "received_events_url": "https://api.github.com/users/octocat/received_events",
      "type": "User",
      "site_admin": false
    },
    "created_at": "2012-07-20T01:19:13Z",
    "updated_at": "2012-07-20T01:19:13Z",
    "statuses_url": "https://api.github.com/repos/octocat/example/deployments/1/statuses",
    "repository_url": "https://api.github.com/repos/octocat/example",
    "transient_environment": false,
    "production_environment": true
  }
]

If you want to learn how to get other data from your repositories, read the GitHub API documentation.

Building a Grafana dashboard for deploys by repo

Grafana is an open-source visualization tool that allows you to store data and display it graphically. Grafana is used to visualize and analyze large amounts of data. Various metrics saved to a database can then be displayed using convenient, beautiful, and easily customizable panels. 

A Grafana Dashboard is a group of widgets with which you can create different types of graphs. It contains a lot of tools for visualizing metrics and customizing graphs. The basic building block for graphically displaying data in Grafana is the panel. You can create various types of panels, including chart panels, list panels, monitoring panels, and statistics panels. Within Grafana Dashboards, you can customize various dashboard properties such as styles and formats, metadata, strings, links, timing, and more.

Grafana allows you to use a lot of different databases, including MySQL, PostgreSQL, ElasticSearch, Influx DB, and Graphite. Graphite is one of the best data sources for Grafana. After creating a Grafana Dashboard, you can export it to a JSON file, generate an external link to it, or take a screenshot. If you need to expand Grafana Dashboard functionality, you can install additional plugins.

Key benefits of using Grafana

1. The ability to load data from any source, as well as from several different sources.
2. The ability to share dashboards with different users in the company. It allows collaborative data analysis.
3. The ability to create advanced requests and data filtering, allows you to create useful and informative graphs.

To get started with Grafana, you need to create an administrator user and log in as this user. Then create a new Grafana Dashboard. After that, you need to select a data source and you can proceed with setting up charts. There is a GitHub addon, that was developed to simplify the process of monitoring GitHub repositories. It allows you to send GitHub events to a special webhook URL and save this data in your account in annotations. After that, you can visualize the received data on the Grafana Dashboard. You can also save GitHub data to any database you like and connect it to Grafana to display the data.

For more information on integrating GitHub with Grafana, book a demo with our team or sign up for a free trial today.

The advantages of Hosted Graphite from MetricFire

Graphite is a tool that allows you to collect and store real-time time-series data. It also has basic data visualization tools, or it can be connected to the more powerful and flexible Grafana system. MetricFire is an open-source platform for monitoring infrastructure, systems, and applications using programming tools. MetricFire offers Hosted Graphite whereby can use Graphite as a web application without having to install and configure it locally. Using Hosted Graphite by MetricFire, you can track CPU metrics, memory, and other types of metrics. You can fully focus on the performance of your system, and we will take care of the installation, configuration, and maintenance of your monitoring system.

There are some other reasons why you should choose MetricFire.

1. MetricFire provides constant access to your data at any time.
2. You can track system metrics without having to deploy monitoring tools, saving you time and money.
3. MetricFire provides a wide range of pricing plans to suit your needs.
4. If you have any questions while working with MetricFire, our support team is ready to provide comprehensive answers either via our support channels or via video conference.      

Conclusion

In this article, we detailed the process of obtaining GitHub repository data in a time-series format. We also showed how to store and visualize this data using Hosted Graphite and Grafana, and analyzed the main advantages of using MetricFire.

To get more information about how to use MetricFire, book a demo with our team, or sign up for the MetricFire free trial today.