Receiving PagerDuty alerts from MetricFire

Introduction

One of the most critical aspects of monitoring your digital assets is getting a timely alert when something goes wrong. Even when you finish building a monitoring stack and expose metrics on a beautifully designed dashboard if you cannot notice abnormal behaviors and fail to take pre-emptive or follow-up actions swiftly, this means your monitoring system does not serve its purpose. After setting up all your monitoring infrastructure, the final task you should perform is designing and implementing a notification system so that you don’t get blindsided by incidents. In this article, we will discuss how we can receive PagerDuty alerts from MetricFire.

Before we start, check MetricFire. MetricFire provides comprehensive monitoring solutions with Hosted Graphite. You can set up various metrics to monitor Windows with minimal configuration. If you would like to learn more about it, please book a demo with us!

How PagerDuty works

PagerDuty is a cloud-computing firm that’s specialized in a SaaS incident response for IT infrastructures. The company’s SaaS-based platform helps IT professionals, prevent and resolve incidents that can impact business. 

When an incident occurs, it can damage a company’s brand image, revenue, and customer satisfaction. PagerDuty allows organizations to gain insights to pre-emptively control events. The SaaS products offer hundreds of native integrations, on-call scheduling, escalations, response orchestration, analytics, and more. 

Depending on your circumstances, there can be three major use cases of PagerDuty.

• You can customize notification rules for your existing monitoring tools. For example, set up the rules to text or call users for an urgent alert or send them an email if it’s not an urgent issue.
• You can arrange agile incident management workflows to your current setup for on-call scheduling, automated escalations, and incident tracking.
• You can build a single point where you can check the overall health status of your systems, applications, and operations.

How to set up PagerDuty with MetricFire

PagerDuty, when it is integrated into MetricFire, can boost the efficiency of incident monitoring. To set up alerts with MetricFire’s Hosted Graphite, follow the steps below.

1. Go to your PagerDuty dashboard, go to “Configuration” => “Services” => “+ New Service” on the button on the right side.

2. Name your new service and in the Integration Settings section, choose “Use our API directly”. This will generate an integration key for the new service we just made. Keep this key somewhere since we will want to add this key to your Hosted Graphite account.

3. Then, go to your Hosted Graphite dashboard.

4. Go to “Alerts” => “Notification Channels” => “+Add Channel” (the green button at the top-right corner).

5. In the “Add Notification Channel” panel, set “Pagerduty” for the Notification Type. Put the prepared integration key from PagerDuty into the PagerDuty Service Key input.

6. Here, we’ll create some alerts on one of the graphs. Go to “Alerts” => “Alerts” => “+ Add Alert”.

7. Put your alert name and your metric query. ex) collectd.kaenatsu.memory.memory-free To refer to the query you want

8. Let’s set the criteria for the alert. You can configure it to alert if the value goes above or below a certain value. You can also set it when the value becomes null. Once you decide the criteria, make sure you set “Notify me with…” to the new PagerDuty notification channel you created right above.

9. You can validate to see if the metric query is working properly by going back to “1. Choose Metric” and clicking “Check Metric Graph” at the bottom of the window. It will bring up something similar to the image below.

After you set up the integration as above, it is ready to receive alerts. Now, you decide how you want to receive notifications, either by SMS messaging or phone call. PagerDuty allows you to choose a different notification medium based on the conditions you define. If an incident is not critical, you can consider using email or SMS. If the issue is time-critical, receiving a phone call will be more effective to grab your attention.

Receiving Text Alerts

You can set up SMS notifications per PagerDuty action and define the SMS body. In your initial account setup, you can choose notification methods. To add a phone and SMS notification method, check the checkbox to the left of “Phone & SMS” and enter your phone number. If you want to run a notification test to your number, click “Send Test Notification”.

If you want to receive text alerts after you already joined PagerDuty, you will have to change the existing settings. To do so, follow the steps:

• Go to User Icon => “My Profile” => “Contact Information” tab.
• Click “Add SMS Number”. Here, you can add 10 different contact methods.
• Enter your contact details.
• Click “Save”

To test if you can receive notifications correctly,

• Go to User Icon => “My Profile” => “Contact Information” tab.
• Click “Test” to the right of the contact method you want to test.

To give you an idea of the types of PagerDuty actions that can trigger SMS notifications, check the action list below.

• A new incident triggers and the user is on call.
• The user has been added as a responder (new incident).
• The user has been added as a responder (existing incident).
• Status update to an incident that a user is assigned to or subscribed to.
• The user is going on call for an escalation policy.
• The user is going off call for an escalation policy.

Receiving Phonecalls

For urgent issues that require prompt actions, you can configure PagerDuty to receive phone calls. Adding a phone number follows the same methods for the initially created account. During your initial account setup, you can also choose a phone call notification. To do so, check the checkbox to the left of “Phone & SMS” and enter your phone number. If you want to run a notification test to your number, click “Send Test Notification”.

If you want to receive phone call alerts after you already joined PagerDuty, you will have to change the existing settings. To do so, follow the steps:

• Go to User Icon => “My Profile” => “Contact Information” tab.
• Click “Add Phone Number”. Here, you can add 10 different contact methods.
• Enter your contact details.
• Click “Save”

To test if you can receive notifications correctly,

• Go to User Icon => “My Profile” => “Contact Information” tab.
• Click “Test” to the right of the contact method you want to test.

The following points show you the types of PagerDuty events that can trigger phone calls.

• A new incident triggers and the user is on call.
• The user has been added as a responder (brand new incident).
• The user has been added as a responder (existing incident).

Advantages of using Hosted Graphite with MetricFire

Graphite is open-source software that requires users to maintain everything from installation and monitoring Graphite clusters. This can create a management overhead and prevent you from focusing on what matters. Hosted Graphite by MetricFire does heavy-lifting for users. In addition to the benefits that the open-source Graphite offers, it provides the following benefits.

• Redundancy storage: Graphite’s default storage is file-based and antiquated. MetricFire offers 3 times the redundant storage for seamless scaling and better data protection.
• Control by APIs: The APIs that MetricFire provides lets you control and automate the resources of Hosted Graphite.
• Tagged metrics: Hosted Graphite stores data using tags that enable viewing and organizing metrics with data views.

MetricFire keeps all the benefits of open-source Graphite and further enhances the tool with a built-in agent, team accounts, granular dashboard permissions, and integrations to other major platforms and services such as AWS, logging tools, and more.

In addition to the major features, there are more.

• MetricFirebacks up your user data and dashboard every hour.
• You are in good hands with MetricFire's technical experts.
• MetricFire offers an extensive range of options for users of all sizes with plans and customization to meet your needs.
• MetricFire’s on-call team is ready 24/7, 365. Our team keeps watching your hosted Graphite from around the world using an automated monitoring system.
• We have been ingesting billions of data points per day since 2012. Our mature metric processing and storage capabilities are trusted by thousands of engineers. 
• You can directly send metrics from your application without additional dependencies or aggregation services.
• You can receive alerts through many different channels, including email, Slack, and PagerDuty.

Wrapping up

In this article, we learned about PagerDuty and how to set up to receive PagerDuty alerts by linking to MetricFire’s Hosted Graphite. Choose hosted solutions from MetricFire and boost monitoring capacity by getting alerts at the right time. Book a demo with our experts to learn more about how MetricFire can help you with the alerting stack with PagerDuty.