Table of Contents
Nielsen is the leading global measurement and data analytics tool that has been innovating in their industry for over 90 years.
Headquartered in New York, Nielsen has operations in over 100 countries and cover more than 90% of the world’s population. The Nielsen Marketing Cloud combines Nielsen’s data analytics, media planning, marketing activation and data management platform capabilities in a single cloud platform. This allows clients to gain more accurate and validated data at scale. Nielsen Marketing Cloud use Hosted Graphite to understand what’s going on inside their systems.
The technical problem
Itai and the twenty member Big Data group collect, process, store and analyze all the data for Nielsen Marketing Cloud.
As they offer both data-as-as-service (DaaS) and software-as-a-service (SaaS) for the AdTech industry, for Itai that means data, and lots of it.
The team monitors both system (e.g. memory usage) and application metrics (e.g. the number of records per processing cycle). However, as the company expanded, the limitations of its tech stack became more obvious and they began the process of updating their tools. As Itai puts it, “We started a process of transforming our entire data tech stack to one that would be able to support our growing business needs. We were looking for better tools and frameworks to replace our old tools.” After deciding on Apache Spark and Druid, the team set their mind to finding a new monitoring tool. In addition to seamless scaling, they needed one that would:
- Ensure cost control and predictability: It was important that the monitoring tool had easy to understand, transparent pricing, to keep costs down and guarantee reliable billing.
- Take minimal effort to get up and running: The team didn’t have time to opt for a tool with a tricky set up. They needed to continue monitoring their key metrics and KPIs as quickly as possible so required a tool that, according to Itai, “took little effort to set-up and use...we wanted to create and view new dashboards with ease.”
- Integrate with other tools: The Big Data group uses a wide variety of services so the new monitoring solution needed to integrate easily with their current tech stack in addition to whatever new tools they might use down the line.
- Require little ongoing maintenance: The team are always transforming their stack to support growing business needs. As such, they needed a tool that could scale with them without using lots of precious Engineer and Developer hours in its upkeep and maintenance.
- Include reliable alerting: Itai’s group needed an alerting feature to get notifications when their systems go down or behave unexpectedly to minimize downtime, reduce customer impact as efficiently as possible and keep services operational 24/7.
We wanted a monitoring solution that could easily integrate with our new big data tools.
The solution
While Hosted Graphite met all the team's requirements, Itai and the Big Data group had one major concern: spiraling costs.
We work with various vendors, and your support is the best hands down. It’s Excellent!
They planned to integrate one of the new big data tools they were using, Apache Spark, to send application metrics. However the team were concerned that the metrics coming from Spark might drive costs up. Each time a Spark application starts, it generates an ID to be used as the application identifier, so a typical metric name would be:
application_<APPLICATION_ID>.driver.<APPLICATION_NAME>.<METRIC_NAME>
That way, if you have a batch application that runs every hour, it would send a metric: <METRIC_NAME> with the current <APPLICATION_ID>, so the team would end up with multiple metrics in Hosted Graphite, which are essentially the same metric. “Lucky for us, Hosted Graphite were able to solve this problem,” says Itai. The issue was fixed by removing the <APPLICATION_ID> part from the metric name coming into Graphite, such that,
application_<APPLICATION_ID>.driver.<APPLICATION_NAME>.<METRIC_NAME>
has become,
application.driver.<APPLICATION_NAME>.<METRIC_NAME>
This straightforward solution was put in place by Hosted Graphite’s technical support, without the need for Itai and the team to change anything in their code or configuration. This keeps their metric count under control, which put the team’s budgetary concerns to rest: “Hosted Graphite charges per metric, so it’s very easy to understand how to keep costs under control.” In addition, according to Itai, “when we saw how easy it was to use Hosted Graphite we were more than happy to choose it”.
The result
We now have an excellent understanding of what’s going on inside our systems, both from an infrastructure and application perspective.
Since migrating to Hosted Graphite for their monitoring, Itai and the Big Data group have a much deeper insight into what’s happening with their apps and infrastructure. They also heavily rely on Hosted Graphite’s advanced alerts feature and get notifications through Slack to keep on top of unusual activity. This allows the team to keep track of when a certain threshold is violated, and act quickly to rectify problems before Nielsen’s customers are affected.
The team also use the metrics dashboard feature to help them see how much traffic they’re sending, how it varies over time and over what protocols and the metric treemap features, to explore what part of their infrastructure is creating a lot of metrics and is a cost driver. Hosted Graphite’s continuous improvement has been the cherry on the cake for Itai: “It’s great to know that Hosted Graphite are constantly improving the product and adding new features.”
Hosted Graphite’s transparent pricing–no surprises and no hidden fees–has removed the team’s worry about spiralling costs. According to Itai, “there’s complete transparency with everything Hosted Graphite do which means we can accurately predict what we’ll be spending and comfortably keep within our budget.”
