Integrating Heroku Metrics with Amazon CloudWatch Metrics

Introduction

Application monitoring plays a critical role in the success of your digital products. As you monitor various performance metrics such as usage of CPU, memory, network traffic, and more, you can swiftly take pre-emptive actions before things develop into a larger problem. In spite of the importance of monitoring, the task can become challenging when your infrastructure exists across multiple cloud platforms including AWS and Heroku. That’s because you don’t have a single location to monitor all your applications and that makes you visit each of them to perform monitoring tasks. 

In this article, we will learn how to integrate Heroku metrics with AWS CloudWatch so that you can have one ground true location. The key to the integration is Hosted Graphite by MetricFire. MetricFire offers comprehensive monitoring solutions with Hosted Graphite.

You can gain in-depth insight into your resource with minimal configuration. If you would like to learn more about it, please book a demo with us, or sign on to the free trial today. 

Key Takeaways

1. Monitoring application performance metrics such as CPU usage, memory usage, and network traffic is critical to the success of digital products.

2. Integrating Heroku metrics with AWS CloudWatch through Hosted Graphite by MetricFire enables users to have a single location for monitoring all their applications.

3. AWS and Heroku are different cloud platforms with different strengths and weaknesses. AWS is Infrastructure-as-a-Service (IaaS) while Heroku is Platform-as-a-Service (PaaS).

4. Heroku provides essential monitoring features with add-ons, including logging, application performance monitoring, error monitoring, and platform monitoring. AWS CloudWatch, on the other hand, allows auto-scaling of resources, has anomaly detection capabilities, and supports more than 70 AWS services by default.

Differences between AWS and Heroku

Although there is an overlapping area, AWS and Heroku aim at different market segments and they have different strengths and weaknesses.

Key Concepts

AWS is Infrastructure-as-a-Service while Heroku is Platform-as-a-Service. When AWS was launched, there were only several services EC2, S3, and a few more. As of 2021, there are over 200 services and products in the AWS ecosystem. AWS provides dedicated environments for machine learning, IoT, and even Ground Station as a Service to communicate with your satellites in space. When you use AWS, you can access a variety of infrastructures.

On the other hand, Heroku is a managed platform that focuses on minimizing your time to manage your application infrastructure. Because of these key differences, people choose one over the other or use both for different use cases.

Suitable Users

Although AWS gives a vast ecosystem, because of the nature of IaaS, it requires deeper knowledge to start with and DevOps efforts to maintain your resources. Also, you will need dedicated infrastructure people to manage security, network, permissions, and more. This means that small-sized companies with limited IT resources may find it difficult to use it and maintain it. In contrast, Heroku, as a PaaS, makes things easier to build, deploy, and manage your applications. So, it can be suitable for startups that want to build their first working prototype quickly with limited developer resources.

Cost

What’s interesting is the fact that Heroku is built upon AWS infrastructure. Heroku, using AWS infrastructure is like adding an extra layer on top of AWS so that Heroku users can be less worried about infrastructure management. Because of the extra user-friendliness efforts, Heroku tends to be more expensive than AWS. 

Heroku Metrics

We learned the differences between the two cloud platforms in general. Let’s now focus on the monitoring side of them starting with Heroku. Heroku provides all essential monitoring features with add-ons. 

• Logging: you can collect and store your Heroku app and database logs. When an incident happens, you can use logs to identify the cause and quickly work on a fix. Logging addons will charge you depending on features and logging retention periods.
• Application performance monitoring: APM tools show insights about app performance, which helps you to detect the areas that may cause latency. APM is essential to ensure smooth service for your customers.
• Error monitoring: with error monitoring, you can identify exceptions or errors that occurred in your app. With error monitoring, you can quickly troubleshoot your apps and minimize service disruptions. You can set alerts, for example, when errors occur more than a threshold.
• Platform monitoring: platform monitoring can be used to collect the metrics from the dyno, Postgres, Redis, and routers in the Heroku platform and visualize them in dashboards. 

In addition to these major monitoring capabilities, Heroku native apps and add-ons allow you to perform threshold alerting, uptime monitoring, event-driven notifications for apps and databases, and many more.

CloudWatch Metrics

In AWS CloudWatch, you don’t install addons to extend monitoring capabilities. You can use native CloudWatch features. Although AWS boasts a vast ecosystem, CloudWatch is the central monitoring location that can cover the broad AWS area. 

• Logging: CloudWatch collects and stores logs from resources, applications, and services inside and outside the AWS environments. When you have on-premise servers, you can push logs to CloudWatch.
• Metrics: when you have to collect metrics from widely distributed applications, it can be challenging and time-consuming. But, in AWS, as of August 2021, more than 70 AWS services including EC2, DynamoDB, S3, ECS, Lambda, API Gateway, and more send metrics data to CloudWatch by default. With this functionality, you can monitor CPU usage, data traffic, and disk storage. You can customize metrics by using PutMetricData API too when you want more control over your data.
• Dashboard: CloudWatch dashboards let you create graphs and charts that can be viewed regularly. You can design a single dashboard that includes metrics and logs data to gain quick insights. This allows you to have broad visibility over all your resources. In dashboards, you can set alerts for performance and operational issues and you can take proactive actions before things deteriorate. 
• Anomaly detection: CloudWatch is equipped with machine learning technologies to analyze metrics data and capture anomalous behavior. You can set alerts that automatically adjust thresholds based on normal metric patterns. This helps you to detect unexpected changes and give wider visibility of your resource activities.
• Auto-scaling: one of the things that differentiate CloudWatch from the Heroku monitoring service is that CloudWatch allows auto-scaling of your resources. With CloudWatch, you can make, for example, your servers scale up or down depending on the threshold you set.

Besides these, CloudWatch offers an extensive range of features. Thanks to these features and the diversity of other services, users may want to consolidate metrics with AWS CloudWatch from Heroku.

How to Integrate Heroku Metrics with AWS CloudWatch

Then, how can we integrate Heroku metrics with CloudWatch? The best way is by utilizing Hosted Graphite and its CloudWatch integration addon. Try the Hosted Graphite to expand your monitoring capabilities and more insightful metrics. Then, you need to install the CloudWatch addon in Hosted Graphite to integrate metrics with CloudWatch.

First, you should set up Identity and Access Management (IAM) access keys within your AWS account and attach proper permissions.

Configure IAM in AWS 

We first need to create a policy that will be attached to the user. 

• Go to IAM in AWS and click Policies under Access management.
• At the top right area, find the Create Policy button and click it.
• Select the JSON tab.
• Copy and paste the policy JSON text into the editor.

{
    "Statement": [
        {
            "Sid": "PermissionsForMetrics",
            "Effect": "Allow",
            "Action": [
                "cloudwatch:ListMetrics",
                "cloudwatch:GetMetricStatistics",
                "ec2:DescribeInstances",
                "ec2:DescribeVolumes",
                "rds:DescribeDBInstances",
                "route53:ListHealthChecks",
                "sqs:ListQueues",
                "elasticache:DescribeCacheClusters",
                "elasticloadbalancing:DescribeLoadBalancers",
                "kinesis:ListStreams",
                "redshift:DescribeClusters",
                "elasticmapreduce:ListClusters",
                "elasticmapreduce:DescribeCluster",
                "cloudfront:ListDistributions"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Sid": "PermissionsForTags",
            "Effect": "Allow",
            "Action": [
                "elasticache:ListTagsForResource",
                "elasticloadbalancing:DescribeTags",
                "cloudfront:ListTagsForResource",
                "route53:ListTagsForResource",
                "kinesis:ListTagsForStream",
                "rds:ListTagsForResource",
                "lambda:ListFunctions",
                "lambda:ListTags",
                "iam:GetUser"
            ],
            "Resource": [
                "*"
            ]
        }
    ],
    "Version": "2012-10-17"
}

• Click “Next: Tag” and “Next: Review”. 
• In the Review policy, put in a name like “heroku_policy” and optionally the description.
• Finally, click “Create policy”.

After creating the policy, we need to create a user. Prepare your Access Key and Secret Key tokens.

• Go back to the IAM console again and click “Users”.
• Find the “Add users” button and click it.
• Put in a name like “heroku_addon_user” and select Programmatic access.
• Click “Next: Permissions” and “Attach existing policies directly”.
• Find the policy we created above, “heroku_policy”, and select it.
• Click “Next: Review”

Click “Create User” and copy your Access Key and Secret Key into the fields on the Hosted Graphite addon setup page.

Configure Hosted Graphite

Now, we need to enable CloudWatch addon in your Hosted Graphite. Go to the add-ons page and choose the option for Amazon AWS CloudWatch. In the Amazon CloudWatch Accounts section, you will be able to see an account with Access Key.

You can add AWS Tags by clocking “AWS Tagging”. The tags are to help you to easily identify your configurations. Each name should have a unique field name and you can add multiple tags.

If you click “Cancel” to go back to the previous screen and click “Account Names”, you can find a complete set of configuration fields. You can select the checkboxes if you want to enable more monitoring metrics.

After setting up, check our article that shares how to use Hosted Graphite to monitor your Heroku apps.

Wrapping up

In this article, we’ve learned the major features of Heroku monitoring and AWS CloudWatch. Although they both have powerful monitoring and metrics capabilities, when you have metrics to watch across multiple platforms, it reduces your productivity in monitoring and may lead to creating a change of blindspot. To centralize your monitoring efforts, try our Hosted Graphite and consolidate Heroku and Amazon CloudWatch into a single power source. Hosted Graphite provides an extensive range of metrics and features on top of intuitive dashboards.

Visit MetricFire today and check our Graphite as a Service.