Spark Performance Monitoring using Graphite and Grafana

Introduction

In this article, we will explore what Apache Spark is, what key metrics you need to track to keep it running, and how to set up a metrics-tracking process. We will also cover monitoring tools such as Graphite and Grafana, which make the process of monitoring metrics very easy, as well as how using MetricFire can make running your monitoring exponentially easier.

Check out MetricFire for free or book a demo with our team and learn more about all the benefits of using MetricFire solutions. 

Key Takeaways

1. Apache Spark metrics are crucial for evaluating and monitoring the performance of Spark components. These metrics cover resource usage, job status, worker processes, message processing, and more.
2. Users can view Spark metrics through the web user interface or the REST API. This information includes details about stages and tasks, RDD sizes, memory usage, and more.
3. The article explains how to customize Spark metrics using a configuration file. It distinguishes between internal and shared metric sources and provides example configurations for monitoring various Spark components.
4. Graphite is introduced as a monitoring tool for Spark, with instructions on how to set it up. It collects and visualizes time-series data, making it a useful addition to Spark monitoring.
5. The article mentions Grafana as an open-source web application for data visualization and analysis. It highlights its features, including flexible graph creation, dashboard customization, and the ability to create alerts.

What is Apache Spark?

Apache Spark is an open-source, multilingual framework for data analysis and machine learning on single-node machines or on clusters. To batch process your data and stream it in real-time, you can use one of these programming languages: Python, SQL, Scala, Java, or R. Apache Spark is a fast in-memory data processing engine, so it allows you to perform various tasks efficiently. It provides you with the ability to run distributed ANSI SQL queries for dashboards and custom reports faster than most data warehouses. Apache Spark also provides a lot of tools for machine learning, structured data processing, graphs, and streaming data.

What are Apache Spark Metrics?

Metrics are a set of tools that allow you to evaluate the behavior of key elements of the system, analyze the system’s performance in dynamics, and find and correct errors in time. Apache Spark metrics monitoring provides insight into resource usage, job status, worker processes, message processing, and performance of standalone Spark clusters.

Spark metrics are separated into different instances corresponding to Spark components. On each instance, you can configure the set of sinks to which metrics are reported. Spark system allows you to send metrics to various sinks, including HTTP, JMX, and CSV files. The metrics used by Spark come in several types: gauge, counter, histogram, and timer. The most common timing metrics used in the Spark toolkit are gauges and counters.

The most useful metrics for Spark performance analysis include:

• Average time spent on tasks and assignments.
• The amount of memory used.
• The amount of CPU used compared to the CPU used by garbage collection.
• The number of data records written and retrieved to/from disk in shuffle operations.
• Disk I/O statistics.
• The number of used and free workers.
• The number of employees in memory.
• the number of active, running, waiting, and failed jobs.

You can view Spark metrics using the UI or the REST API.

Each SparkContext launches a web user interface that displays useful information about the application, including:

• List of stages and tasks of the planner.
• Summary of RDD Sizes and Memory Usage.
• Environmental information.
• Information about current performers.

By default, this information is only available while the application is running, but it is possible to configure applications to view the web interface after the fact. The UI of an application can be created through the Spark History Server, provided application event logs exist. Using the REST API, it is possible to receive metrics in JSON format. This gives developers an easy way to create new visualization and monitoring tools for Spark. JSON is available to both running applications and the history server. 

How to Configure your Spark Metrics to MetricFire

You can customize the Spark metrics system using a configuration file that contains settings for Spark’s internal metrics system. The Spark component can be divided into instances: “master”, “worker”, “executor”, “driver”, and “application”. Each instance corresponds to an internal  Spark component and can report its metrics to one or more sinks. An instance contains a specific set of grouped metrics.

There are two types of metric sources:

1. Internal sources: MasterSource, WorkerSource, etc. They collect the internal state of a Spark component.
2. Shared sources such as JvmSource. These sources collect low-level states. Shared sources can be added via configuration options and then loaded using reflection.

Component settings:

1. The “receiver” field indicates where the metrics are delivered. Each instance can send metrics to one or more receivers.
2. The “destination|source” field specifies whether the property belongs to the destination or source.
3. The “name” field specifies the name of the source or destination.
4. The “options” field is a specific property of this source or destination. The source or destination is responsible for parsing this property.

In order to start monitoring a specific group of metrics, it must be configured in the configuration file. Use the following code to configure the monitoring master, worker, driver, and executor metrics.

master.source.jvm.class=org.apache.spark.metrics.source.JvmSource
worker.source.jvm.class=org.apache.spark.metrics.source.JvmSource
driver.source.jvm.class=org.apache.spark.metrics.source.JvmSource
executor.source.jvm.class=org.apache.spark.metrics.source.JvmSource

To use Graphite as a monitoring tool for Spark, use the following settings:

*.sink.graphite.class=org.apache.spark.metrics.sink.GraphiteSink
*.sink.graphite.host=carbon.hostedgraphite.com
*.sink.graphite.port=2003
*.sink.graphite.period=10
*.sink.graphite.unit=seconds
*.sink.graphite.prefix=<your-api-key>

You can find your API key in your account once you sign up for a free two-week MetricFire trial. For more detailed information on configuring Spark Metrics with Hosted Graphite check out this tutorial.

Visualizing your Spark Metrics with Graphite and Grafana

Grafana is an open-source web application for data visualization and analysis. It allows you to query, visualize, alert and understand your metrics. Grafana allows you to use various data sources, including MySQL, PostgreSQL, ElasticSearch, Influx DB, and Graphite.

The main features that Grafana provides:

1. Fast creation of flexible graphs on the client-side.
2. Create dynamic and reusable dashboards.
3. Ability to explore metrics with special queries and dynamic drill down. You can split view and compare different time ranges, queries, and data sources side by side.
4. Ability to explore journals with saved tag filters, quickly search through all journals, or broadcast them live.
5. Ability to create alerts and send notifications to systems such as Slack, PagerDuty, VictorOps, and OpsGenie.
6. Ability to use different data sources on the same chart. You can specify a data source for each request.

To get started with Grafana, you need to create a Grafana dashboard. It contains a lot of tools for visualizing metrics and allows you to create different types of graphs.

The basic block in the Grafana dashboard is the panel. Panels come in different types: chart panels, list panels, dashboards, and statistics panels. After creating a panel, you need to select a data source and set up a graph. The Grafana dashboard allows you to customize various dashboard properties such as styles and formats, metadata, strings, links, timing, and more.

For more information on how to visualize your Spark Metrics with Grafana, book a demo with our technical team or sign up for a free trial today.

The advantages of MetricFire

Graphite is an open-source tool that allows you to collect, store and visualize time-series data in real-time. Graphite can collect data from various sources such as infrastructure, servers, networks, and applications and provide it for analysis.

Graphite is made up of three components:

1. Carbon is a service that receives time-series data and feeds it to Whisper and to the Graphite web interface.
2. Whisper is a database for storing time-series data.
3. Graphite web interface is an interface that displays time series data and interacts with metrics in the system.

Graphite can be used as a data source for Grafana, which provides the ability to create beautiful, powerful, and customizable graphs.

MetricFire provides a solution that you can use as a web application to collect and store Spark metrics. Using MetricFire, you can fully focus on the process of working with your metrics, and we will take care of the installation, configuration, and maintenance of Graphite.

Let’s take a look at the main benefits of using MetricFire:

1. Ability to access your data at any time.
2. Affordable pricing and the ability to choose a plan that suits your needs and budget.
3. Using a ready-made monitoring system without having to invest time and money in its deployment and launch.
4. Reliable support. The MetricFire team of professionals is always ready to give a comprehensive answer to any of your questions regarding the operation of our system.

Conclusion

After reading this article, you learned what Apache Spark is and how to track its metrics. We also talked about the monitoring tools Graphite and Grafana, which allow you to collect, store and visualize your Spark metrics.

Use MetricFire and save time and money on setting up and maintaining your monitoring system. Sign up for the MetricFire free trial or book a demo with our experts and get detailed information about integrating your system with MetricFire tools.