How To Test 50k+ Concurrent Users in Performance Tests

Learn how to easily test concurrent users using JMeter. This article will describe the steps you need to take to easily run a load test with 50K concurrent user tests (as well as bigger tests, up to 2 million users).

How to Test Concurrent Users Using JMeter: Quick Overview 

1. Write your script

2. Test it locally using JMeter

3. BlazeMeter SandBox testing

4. Setup the number of concurrent users - the amount of Users-per-Engine using 1 Console & 1 Engine

5. Setup and test your Cluster (1 Console & 10-14 Engines)

6. Use the Master / Slave feature to reach your max CC goal

Step 1 : Write Your Script

Before you begin, make sure to get the latest JMeter version from the JMeter Apache community jmeter.apache.org .

Before you get started, you need to download the JMeter Plugins Manager. Once you've downloaded the JAR file put it into JMeter's lib/ext directory. Then start JMeter and go to "Options" menu to access the Plugins Manager.

 There are many ways to get your script:

1. Use the BlazeMeter Chrome Extension to record your scenario 
2. Using the JMeter HTTP(S) Test Script Recorder which basically sets up a proxy so you could run your test through and record everything
3. Going fully manual and constructing everything from scratch (probably for functionality/QA tests)

If your script is the result of a recording (like steps 1 &2), keep in mind that:

1. You will need to change certain parameters such as Username & Password or you might want to set a CSV file with those values so each user can be unique.
2. You might need to extract elements such as Token-String, Form-Build-Id and others using Regular Expressions, JSON Path Extractor, XPath Extractor - in order to complete requests as "AddToCart", "Login" and more...
3. Keep your script parameterized, and use config elements such as HTTP Requests Defaults to make your life easier when switching between environments.

Step 2 : Testing Locally Using JMeter

Start debugging your script with 1 thread, 1 iteration, using the View Results Tree element, Debug Sampler, Dummy Sampler and the Log Viewer opened (in case some JMeter errors are reported).

Go over all the scenarios (True and False responses) to make sure the script behaves as it should...

After the script has run successfully using 1 thread - raise it to 10-20 threads for 10 minutes and check:

1. If you intended that each user be unique - is that so?
2. Are you getting any errors?
3. If you are doing a registration process, look at your backend - are the accounts created according to your template? are they unique?
4. From the summary report you can see statistics about your JMeter testing - does it make sense? (average response time, errors, hits/s)

Once your script is ready:

1. Clean it up by removing any Debug/Dummy Samplers and deleting your script listeners
2. If you use the Listeners (such as "Save Responses to a file") make sure you don't use any Paths! , rather if it's a Listener or a CSV Data Set Config - make sure you don't use the path you have used locally - instead, use only the filename (as if it was on the same folder as your script)
3. If you are using your own proprietary JAR file(s) be sure to upload it too.
4. If you are using more than 1 Thread Group (or not the default one) - make sure to set the values before uploading it to BlazeMeter.

Step 3 : BlazeMeter SandBox Testing

If that's your first test - you should review this article about how to create tests in BlazeMeter. This is an important step to help you learn how to test concurrent users using JMeter. 

SandBox it actually any test which has up to 300 users, uses 1 console only and up to 50 minutes.

The configuration for SandBox allows you to test your script and backend and ensure everything works well from BlazeMeter.

To do that, first press the grey button: JMeter Engines I want complete control! - to get full control over your test parameters..

Common issues you may come across:

1. Firewall - make sure your environment is open to the BlazeMeter CIDR list (which is being updated from time to time) and whitelist them
2. Make sure all of your test files e.g: CSVs, JAR, JSON, User.properties etc.. are present
3. Make sure you did not use any paths

If you still having trouble look at the logs for errors (you should be able download the entire log).

A SandBox configuration can be:

• Engines: Console only (1 console , 0 engines)
• Threads: 50-300
• Ramp-up: 20 minutes
• Iteration: Test continues forever
• Duration: 30-50 minutes

This will allow you to get enough data during your ramp-up period (in case you will get some issues there) and you will be able to analyze the results to insure the script executed as expected.

You should look at the Waterfall / WebDriver tab to see the requests are ok, you shouldn't get any error at this point (unless that was your intention).

You should watch the Monitoring tab to see how much memory & CPU was used - this should help you with step 4 while you will try to set up the amount of users per engine.

Step 4 : Setup The Number of Concurrent Users or Amount Of Users-Per-Engine Using 1 Console & 1 Engine

Now, that we are sure the script runs flawlessly in BlazeMeter - we need to figure out how many users we can apply to one engine.

If you can use the SandBox data to determine that, great!

Here, I will give a way to figure this out without looking back on the SandBox test data.

Set your test configuration to:

• Number of threads: 500
• Ramp-up 40 minutes
• Iteration: forever
• Duration: 50 minutes

Use 1 console and 1 engine.

Run the test and monitor your test's engine (through the Monitoring tab).

If your engine did not reach either a 75% CPU utilization or 85% Memory usage (one time peaks can be ignored) :

• Change the amount of threads to 700 and run the test again
• Raise the amount of threads until you get either to 1000 threads or 60% CPU or memory usage

If your engine passed the 75% CPU utilization or 85% Memory usage (one time peaks can be ignored :

• Look at the point of time you first got to 75% and then see how many users you had at that point.
• Run the test again, instead of a ramp-up of 500 put the amount of users you got from the previous test
• This time put the ramp-up you want to have in the real test (5-15 minutes is a great start) and set the duration to 50 minutes.
• Make sure you don't go over 75% CPU or 85% memory usage throughout the test...

You can go safer and decrease 10% of the threads per engine just to be on the safe side.

Step 5 : Setup and Test Your Cluster

We now know how many threads we can get from 1 engine, at the end of this step we will know the amount of users 1 Cluster (test) can get us. This will help us to determine how to test concurrent users using JMeter

A Cluster is logical container which has 1 console (only 1) and 0-14 engines.

Even though you can create a test with more than 14 engines - it actually creates 2 clusters (you can see that the amount of consoles will grow) and clone your test...

The maximum number of 14 engines per console is based on BlazeMeter’s own testing to guarantee that the console can handle the pressure of 14 engines which creates a lot of data to process.

So at this step, we will take the test from step 4, and change only the amount of engines and raise it to 14.

Run the test for the full length of your final test (1,2,3..) hours. While the test is running, go to the monitoring tab and verify:

1. None of the engines is passing the 75% CPU, 85% Memory limit
2. Locate your console label (you can find its name if you will go to the Logs Tab -> Network Information and look for the private IP of your console) - it should not reach the 75%CPU or 85% Memory limit.

If you console reached that limit - decrease the amount of engines and run again till the console is within these limits

At the end of this step you know:

1. The Users per Cluster you will have
2. The Hits/s per Cluster you will reach

Look for other statistics in the Aggregate Table found under your Load results graph for more information about your Cluster’s throughput.

Step 6 : Use the Master / Slave feature to Reach Your Maximum CC Goal

We've got to the final stage of learning how to test concurrent users using JMeter. 

We know the script is working, we know how many users 1 engine can sustain and we know how many users we can get from 1 Cluster.

Let’s assume these values:

• 1 engine can have 500 users
• The cluster will have 12 engines
• Our goal is 50K test

So to do that, we will need to create 50,000 \ (500*12) = 8.3 clusters..

We could go with 8 clusters of 12 engines (48K) and 1 cluster with 4 engines (the other 2K) - but it would be better to spread the load like this:

Instead of 12 engines per cluster we will use 10, so we will get 10*500 = 5K from each cluster and we will need 10 clusters to reach 50K.

That will help us by:

1. Not maintaining 2 different test types
2. We can grow by 5K by simply duplicating an existing cluster (5K is much more common than 6K)
3. We can always add more if we need to...

We are now ready to create our final Master / Slave test with 50K users:

1. Changing the name of the test from "My prod test" to "My prod test - slave 1".
2. So we go back to our test in step 5 and under the Advanced Test Properties we change it from Standalone to Slave.
3. Pressing save - we now have the first out of 9 Slaves and 1 Master.
4. Go back to your "My prod test -slave 1".
5. Press Duplicate.
6. Now repeat steps 1-5 until you create all 9 slaves.
7. Go back to your "My prod test -salve 9" and press Duplicate.
8. Change the test name to "My prod test -Master".
9. Go to Advanced Test Properties and change it from Slave to Master.
10. Check all the slaves we've just created (My prod test -salve 1..9) and press save.

Your Master-Slave test for 50K users is ready to go. By pressing start on the master you will launch 10 tests (1 master and 9 slaves) with 5K users from each one.

You can change each test (slave or master) to come from a different region, have a different script/csv/other file, use different Network Emulation, different parameters.

The aggregate report of your master and slave will be found in a new tab at the master report called "Master load results” and you could still see each individual test result by simply opening its report.

Want to easily run performance tests for up to 50,000 virtual users, and even up to 2 million virtual users? You can try out BlazeMeter for free.

START TESTING NOW

Related Resources; 

• What’s the Max Number of Users You Can Test on JMeter?