Load testing in the cloud: Raising the bar
Load testing of websites has changed dramatically in the past few years. While cloud based services such as Blazemeter have made it a lot easier to conduct load tests, the changing nature of hosting applications and services on the internet have introduced some new complexities.
We are no longer talking about your daddy’s web server anymore. A typical website is no longer hosted on a single server in a datacenter. It integrates a bunch of services coming from various providers. These services include shopping carts, analytics systems such as Google analytics, and CDNs (content distribution networks) for static content. You may or may not want to include these component services in your load test. Including these external components in your test is more realistic. But you will have to take care to make sure your tests do not hit the main production side of those services. You don’t want to pollute your Google analytics statistics with your load test.
Let me share with you a few stories about complexities I have run into when doing real life load testing.
For a realistic load test on a website you need a realistic load. One obvious way of doing that is recording and scripting the major transactions that you want the website to perform. Each transaction (or click paths) then results in a series of hits on the server (i.e. GET requests). However, in a number of cases I found out that the distribution of GET requests on the server is very different from the distribution of these requests in the scripts. Contributing to this are caching effects by the browser: a well designed website allows the browser to reuse a lot of static content on a repeat visit. I have even noticed that these caching effects can be dependent on the provider of the client, which can only be explained by the provider caching web content. Obviously if the traffic mix in the load test is different from the traffic mix in real life, the predictive value of the load test is reduced. The real life traffic mix is likely to have a higher fraction of dynamically generated content, which would generate a higher demand for resources on the servers.
A serious website has serious load balancing that distributes the traffic over multiple servers. This distribution could be based on the origin of the requests, so that the load can be handled in a geographically distributed way. I once worked with a client where the load balancing was based on client IP address. The complication then is that this implies that a load test conducted from a single location will probably only exercise a part of the full infrastructure, and then will underestimate the total capacity of the service under test.
So maybe we will have to reconsider the way we look at load testing. It is no longer realistic to view your test situation as having a single load generator fire away to the server under test. A better way of looking at it would be as one big load generating cloud that drives test traffic to another cloud interacting with it in multiple touch points around the world.
Then there are situations where we will not be able to load test the service as a whole. For example, a high end website may have auto scaling on the server side. More traffic will lead to more servers being deployed on the website, automatically. You will see some pretty weird performance results as you ramp up the test traffic. The website will seem to be able to handle more and more traffic, and after a while it will handle load that it could not handle in the beginning.
We have just begun to explore the world of cloud computing, and load testing in the cloud. I am sure we will see some very interesting developments in the near future.
For more developments in cloud computing follow my blog at http://blog.clubcloudcomputing.com
Peter HJ van Eijk is a trainer, writer, consultant and speaker on Cloud Computing and other digital infrastructures, based in the Netherlands. He is master trainer for the Cloud Essentials course www.cloudessentials.net and a Cloud Credential Council Certified Trainer.