Table of Contents
Introduction
Amazon CloudWatch Logs Insights is a vital component of Amazon CloudWatch, empowering users to efficiently search, analyze, and extract valuable insights from log data stored in Amazon CloudWatch Logs. In this article, we'll delve into CloudWatch Logs Insights and its capabilities, offering a clear understanding of how it enhances AWS users' log data management and analysis efforts.
We'll also provide an overview of Amazon CloudWatch, its metrics repository, visualization features, and alerting capabilities. Additionally, we'll discuss its global infrastructure and how CloudWatch metrics are stored and summarized across regions.
Key Takeaways
- CloudWatch Logs Insights is a powerful feature within Amazon CloudWatch that enables users to search, analyze, and gain insights from log data stored in Amazon CloudWatch Logs.
- Amazon CloudWatch is a metrics repository that collects data from all AWS services, with the ability to create custom metrics and set up alarms for automated responses.
- CloudWatch offers visualization capabilities, including the CloudWatch Console for metrics and statistics dashboards.
- Real-time log data processing can be achieved through subscriptions, which feed data to other AWS services for further analysis.
- Synthetic tests in CloudWatch enable monitoring of web application interactions and can trigger alarms based on API responses.
What is CloudWatch Logs Insights?
Amazon CloudWatch provides Log Insights, a feature that can help you:
- Search and analyze log data located in Amazon CloudWatch Logs
- Perform queries required for operational issue response
- Identify root causes and validate fixes
CloudWatch Log Insights uses a proprietary query language with several basic commands. It provides sample queries for common AWS service log types, as well as query auto-completion. Learn more about CloudWatch Log Insights capabilities and how to use them.
Users can also leverage log field discovery, which automatically locates fields in JSON-based AWS service logs, including Amazon Route 53, AWS Lambda, AWS CloudTrail, and Amazon VPC.
Connect AWS CloudWatch to MetricFire as a data source in the MetricFire free trial, and start reaping its benefits! You should book a demo and get in touch with one of our MetricFire engineers!
How Amazon CloudWatch Works
Amazon CloudWatch is a metrics repository that collects data from all Amazon services. Infrastructure as a Service (IaaS) offerings like Amazon EC2 or S3 place metrics in the repository, and retrieve statistics based on these metrics. You can also place custom metrics in the repository and retrieve statistics.
In addition, CloudWatch provides:
- CloudWatch Console—lets you visualize metrics and statistics in dashboards
- Alarms—you can set up alarm actions to stop, start, or terminate Amazon EC2 instances when certain conditions are met
- Integration with Auto Scaling and SNS—CloudWatch alarms can trigger Amazon EC2 Auto Scaling events and notifications by Amazon Simple Notification Service (SNS)
The AWS global infrastructure is divided into several regions. CloudWatch metrics are stored in each Amazon region separately, but you can use CloudWatch's cross-region feature to summarize statistics from multiple regions.
Recommended reading: 5 Cool things you can do with metrics on AWS
CloudWatch Log Insights Best Practices
Visualizing Log Data in Graphs
CloudWatch Log Insights generates bar charts, line charts, and stacked area charts using the stat function and aggregation functions.
Visualizing time series data
Time series visualizations are possible for queries that contain one or more aggregation functions and use the bin() function to group data by a single field.
Visualizing log data grouped by fields
Bar charts can be generated by running a query using the Visualization tab. Clicking on the arrow next to Line reveals the Bar option. Charts are limited to 100 bars.
Filter and Pattern Syntax
You can use metric filters to find and organize terms, phrases, or values in log events. For example, you can create a metric filter to detect and count occurrences of the word WARNING in log events for a specific Amazon service.
Metric filters can be used in several ways:
- Counter—when the "metric" filter detects a matching term, phrase, or value in a log event, it can increment the metric.
- Numeric value—metric filters can extract numbers from log events, such as the latency of a web request. In this example, the indicator value can reflect the actual number extracted from the log.
- Conditional operators—you can also use condition operators (such as AND, OR) and wildcards for more precise matches.
Before creating a metric filter, you can test your search pattern in the CloudWatch console.
Saving and Re-running CloudWatch Logs Insights Queries
CloudWatch Logs Insights provides a query language, allowing you to perform structured queries on log data. After creating a query, you can save it to run it again later. This can save time and help you build a library of routine analysis patterns. Saved queries are stored in a folder structure to keep them organized. Each account can store up to 1000 CloudWatch Logs Insights queries per region.
Note that to save queries or view saved queries, you must have the appropriate permissions in your Identity and Access Management (IAM) role.
View Currently Running and Recent Queries
You can view the query currently running, and the history of recent queries. It can be useful to see which analyses your colleagues recently performed, and reuse them instead of running new queries. Also, each account can run up to 10 CloudWatch Logs Insights queries simultaneously, including queries added to the dashboard.
So it is important to see how many queries are available before running new ones. To view current and historical queries, open the CloudWatch Console, select Insights from the navigation pane, then select History.
Real-time Processing of Log Data with Subscriptions
Subscriptions provide access to a real-time feed of CloudWatch Log events. This feed can be sent to other Amazon streams, such as AWS Lambda, Amazon Kinesis, or Amazon Kinesis Data Firehose for custom processing and analysis. It can also be delivered to other systems, encoded in Base64, and compressed as a gzip file or stream.
Before subscribing to log events, you must create a receiving resource to which events will be delivered—for example, a Kinesis stream. Events and their destination can be filtered using up to two subscription filters. Cloudwatch Logs also generates metrics describing the forwarding of events to subscriptions.
Analyze Patterns with Contributor Insights
Contributor Insights identifies common system behavior patterns by analyzing log events. It also identifies top influencers, such as specific host IPs with higher-than-normal calls to an application. By specifying particular values for specific fields, users can filter log entries for further analysis.
They can then compare them to other CloudWatch metrics and add the results to a CloudWatch dashboard, for further troubleshooting and optimization. To do this, select one or more CloudWatch Log groups, then specify the fields to be examined, and select filtering conditions. You can also set conditions for taking specific actions, such as triggering an alarm.
Check End-User Performance with Synthetic Tests
CloudWatch Synthetics enables the configuration of canaries for triggering and monitoring common interactions on a web application by evaluating data and status codes returned by APIs. If needed, these canaries can also trigger a CloudWatch alarm based on response.
Synthetic logs let you evaluate a real end-user experience. They can replicate real-life interactions and evaluate responses, using flexible Node.js scripts.
Recommended reading: Getting started with AWS CloudWatch
Conclusion
In this article, we've covered the basics of Amazon CloudWatch Log Insights, a tool that can help you search and analyze logs from across your AWS deployment. In addition, we’ve also covered seven best practices that can help you do more with CloudWatch Log Insights:
- Visualizing log data in graphs to identify patterns and trends in system logs
- Use filters and pattern syntax to search through large volumes of log data
- Save and re-run common queries to save time and create a library of analysis patterns
- View currently running and recent queries to manage concurrent queries
- Use subscriptions to process log data in real-time
- Use Contributor Insights to identify common system behavior
- Use synthetic tests to measure performance from an end user's performance
We hope this will be helpful as you deepen your use of Amazon CloudWatch and learn to extract more valuable insights from your cloud log data.
Get started with a quick demo of MetricFire today and take charge of your Amazon CloudWatch! Go ahead and avail a free trial as well. We integrate with your existing platforms to produce top-of-the-line performance metrics.
