How to Use AWS CloudWatch for Performance Testing

AWS CloudWatch is an observability tool by Amazon that enables monitoring the performance of applications, services & infrastructure resources deployed on AWS, as well as on hybrid and on-premises environments. When performance testing, observability tools like AWS CloudWatch can help extend our visibility and show us the server-side of application performance. This will help us complement our performance tests. 

In this blog post, we will explore the insights and metrics provided when performance testing and how server-side metrics, with AWS CloudWatch, can help us improve our application’s performance.

Table of Contents:

• Why Use AWS CloudWatch as Part of Performance Testing? 
• How to Use AWS CloudWatch During Your Performance Test
• Using the AWS CloudWatch Console

Why Use AWS CloudWatch as Part of Performance Testing? 

When performance testing, testers, developers, or QA simulate a load of virtual users on the tested application. Our goal when performance testing is to observe the application’s performance, assess whether we achieved the predefined SLAs, and identify performance bottlenecks that may be limiting the application’s scalability and functionality.

Monitoring performance test results with commonly available performance testing tools provide us with client-side metrics of the application performance. These include KPIs like transaction response times, throughput, hits per second, and more.

While these metrics are valuable, the complete performance picture is more extensive. There are additional questions we might need answers to. For example:

1. How are the server’s resources being utilized during the test? Do we have enough CPU capacity?

2. Is there a memory leak?

3. Is there an external service that is performing poorly?

4. Are there any errors thrown by the backend services?

5. Are there any slow DB queries that are impacting performance?

Getting the answers to all such questions is critical to determining whether the application’s performance is acceptable by our standards.

Tools like AWS Cloudwatch can provide us with a comprehensive view of the server-side aspect of application performance, which can help us answer such questions. Let’s dive into three main areas where AWS Cloudwatch can especially help during performance testing:

• Monitoring application and infrastructure resources.
• Monitoring logs.
• Identifying the root cause analysis of issues.

Monitoring Application and Infrastructure Resources

When performance testing, it is a best practice to monitor how the application is utilizing infrastructure resources such as CPU, Memory, Disk, and IO. With AWS CloudWatch, we can create dashboards that will plot the graphs of CPU, Memory, Disk & IO utilization for servers/containers over time. If there were response time spikes during the tests, we can monitor these dashboards and spot any anomalies in the server utilization that might lead to the root cause of the problem.

In addition, we can use AWS CloudWatch to monitor performance in real-time or by looking at historical data and comparing performance across various releases. This can help compare the performance across different releases of application code and determine performance improvement/degradation over time.

Monitoring Logs

Applications log crucial information regarding their performance alongside overall application health. When performance testing, visibility into these application logs can provide us with performance insights. We can monitor these logs from AWS Cloudwatch to see if there are any errors being logged by any of the backend services. This can help us troubleshoot any performance issues seen during the test. We can also investigate these application logs side by side with other system metrics to get a better understanding of performance issues that occurred during the test.

Root Cause Analysis

The data collected by AWS CloudWatch is from all application layers. If we see issues, like transaction response times that are breaching SLA, we need to understand what the root cause of this slowness is. For example, it might be a slow DB query, a slow external service call, an issue related to a resource level bottleneck, or other reasons.

By monitoring through AWS CloudWatch and looking at features like service maps, we can drill down into each request, each application tier, and the respective calls happening in the backend. This granular detail can help us identify the root cause of performance issues.

How to Use AWS CloudWatch During Your Performance Test

Let’s see how to run a performance test for your application and then monitor it with AWS CloudWatch. For performance testing, we will use BlazeMeter, an industry-leading performance testing tool for running performance tests and monitoring client-side performance. As discussed above, adding server-side metrics provides a wider view and a better understanding of the application’s performance.

To enable this, BlazeMeter provides a seamless integration with AWS CloudWatch, with which we can configure BlazeMeter tests to pull the server-side metrics from our AWS Cloudwatch account. By doing so, you can monitor relevant AWS CloudWatch metrics in the BlazeMeter console along with your load test metrics in real-time while running the performance test.

Configuring CloudWatch Metrics in BlazeMeter

1. Create a test in BlazeMeter.

2. Select “CloudWatch” from the APM Integration section.

3. Select “existing IAM key” and enter your IAM key details. (Create a new one if necessary).

4. Select your Region and Namespace. This will fetch a list of items available for your IAM account.

5. Select the Items you want to monitor -> name your profile -> Save -> Apply created profile. You can create as many profiles as you want and subscribe to them from different tests.

6. Run the test and you will see an option to include CloudWatch data in the timeline report.

7. While your test is running, you can correlate various metrics from CloudWatch with client-side metrics from the BlazeMeter test to understand the performance of your application and spot any performance bottlenecks.

This makes identifying performance issues during our test easier and more efficient. We are able to see all the application’s metrics in one single view instead of going to different observability tool consoles to get this information.

Using the AWS CloudWatch Console

For more advanced monitoring using CloudWatch, you can log in directly to the AWS CloudWatch console while running your performance test and explore all your application’s available metrics regarding. This may be required in cases when the metrics pulled into BlazeMeter are not sufficient or you need to have a more enhanced dashboard already set up in AWS for exploring these metrics.

To view these metrics, access the CloudWatch console.

Most of the AWS services will automatically start reporting metrics in CloudWatch as soon as you start using them, so no additional configuration is required in most cases to enable metric collection.

Let’s take an in-depth look at how we can use AWS Cloudwatch features during performance testing.

Using AWS CloudWatch Dashboards 

AWS CloudWatch dashboards provide a single pane of glass where you can configure different graphs to monitor different application resources, such as CPU, memory, etc.

Creating a Custom Dashboard

Although the default AWS CloudWatch dashboards are impressive, we may want to combine a few key metrics that are relevant to our performance test into a single dashboard so we can easily monitor them during the performance test.

To create a custom dashboard:

1. In the console, go to Dashboards -> Custom dashboards -> Click on “Create Dashboard”.

2. Choose a name for your dashboard

3. Choose the type of widget you want to monitor (Line graph, Stacked Area graph) and then configure the metrics such as CPU Usage (%), Memory Usage (%), Disk usage (%), IO, etc.

4. You can add multiple widgets as per your needs.

5. Finally, save the dashboard.

Once you have your relevant metrics plotted in the dashboard, you can use this dashboard during your load test to monitor the resource utilizations. Moreover, after the test ends, you can correlate the Response Time graph from your load testing tool with the Resource Utilization graphs in this dashboard to identify if some resource constraints are causing response times to spike. 

Using AWS CloudWatch Metrics

As we plotted only the key resource metrics in the dashboards, it may happen that we do not see any performance bottlenecks in these metrics. However, our load test will still show response times breaching SLAs or transaction failures. In such cases, we need to look at some additional metrics, in addition to what is already plotted in AWS dashboards.

In this section, you can browse, query, and plot graphs with all the available metrics that are being reported by your application and infrastructure. For example, we may want to monitor how our cache is performing, how were the read/write latencies on the DB, or how the containers were scaling based on load. To obtain such information, we can utilize the Metrics feature of AWS Cloudwatch.

Browsing for Relevant Metrics

You can browse through a list of services that are reporting metrics and plot a specific metric that you feel is relevant to your application.

Then, you can then correlate this metric graph with the client-side observations gathered from your performance testing tool and try to diagnose the cause of the problem.

Querying Specific Metrics

You can also query specific metrics by using search expressions. Use the AWS query builder or directly add your query from the query editor. Search expressions enable us to create dynamic graphs with multiple related metrics.

Using AWS Service Map for Root Cause Analysis

When we see a performance issue during the test, it is critical to dig deeper and understand the origin or root cause of this issue so that we can implement the fix and tune the application performance. However, dashboards or metrics are not always sufficient for understanding the root cause. In such cases, we may need a deeper view of how the application is performing at each application and DB layer.

AWS X-Ray is a distributed tracing service that enables you to trace each individual request served by your application. It provides the ability to drill down into each of the downstream services, API calls, and database calls that your application makes. This granular visibility helps in identifying the root cause of the performance problems and identifying the opportunities for optimization.

While running performance tests, if we see that some transactions are performing slowly and breaching the SLAs, yet there is no clear clue into what the cause could be, we can refer to X-Ray traces to analyze further. 

The Traces View

Let’s say we see a particular HTTP request that is slow in our performance test. We can use the Query refiner in the trace view to filter query expressions by HTTP requests. Based on our filter criteria, a list of available traces will appear. Click on a single trace id to open the trace details page for that specific HTTP request, as seen here:

In the Trace map, we can see all the participating application components including external services and the database tier. From this map, we can visually spot if any particular tier, service, or Database is performing poorly.

The Timeline view at the bottom will show the hierarchy of calls made to the downstream systems in that HTTP request. With this view, we can easily spot which specific downstream call or DB query is contributing to the overall slowness of the application request and thus identify the root cause of the performance problems.

Bottom Line

As modern applications are getting more complex, it is getting tougher to understand the application’s performance. Client-side metrics provide valuable insights, like Transaction response times, Throughput, Hits per second, etc. During performance testing. Having observability into the server side and how the backend infrastructure is performing helps expand that observability and ensure optimal performance.

By combining your performance test with tools like AWS CloudWatch, we can efficiently monitor the server-side performance and get to the bottom of the performance issues seen during the performance test.

Get started with performance testing today with a free trial of BlazeMeter. 

Start Testing Now