JMeter Performance and Tuning Tips

Recently, we came across a great, informative article on JMeter and wanted to share it with our readers. So, many thanks to Philippe Mouawad for graciously allowing us to repost.  

JMeter Performance and Tuning Tips


Context

You want to make a Load Test that will use hundreds to thousands of Threads and you've read on so many blogs that :
  • JMeter consumes a lot of memory...
  • JMeter generates OutOfMemory...
  • JMeter consumes a lot of CPU...
  • JMeter can only cope with 100 threads and not much more...
  • JMeter is fine for playing but once it becomes serious use another tool :-)
  • etc.,etc,....
This has become a kind of "Urban Legend" partly due:
  • to issues that have been fixed for a while now
  • and partly in my opinion to some default configuration parameters that lead to these issues
  • misuse of elements that are to be used during scripting and never during Load Test
Hopefully, You're in the right place, Everything you read is WRONG. :-)
 
In this short article, we will see how to get the most out of JMeter.
 

JMeter Performance Basic concepts

Understand important concepts about JMeter:
  • The more it computes to create or process your samples, the less time it spends sampling, so when doing custom coding, ensure it is efficient
  • Listeners receive Sample Results and do some processing with it, this takes resources (memory, CPU) so during Load Testing, a very simple rule is to REMOVE ALL LISTENERS!, you will ask How can I use my results then?, the answer is, you will do it after the run.

    So NEVER EVER have one of these listeners in your Test Plan during a Load Test:
    • View Results in Table => OutOfMemory guarantee
    • View Results in Tree => OutOfMemory guarantee
    • Graph Results => Performance issues
    • Assertion Results => OutOfMemory guarantee
    • Comparison Assertion Visualizer
    • Distribution Graph (alpha) => Performance issues
    • Graph Results => Performance issues

Performance Checklist

Here are our tips to get the most out of JMeter:
 

Always use the last version of JMeter

JMeter performances and memory usage have been highly improved in last 3 versions and upcoming 2.9 version will bring new set of improvements regarding memory and CPU:

Use NON-GUI for Load Test

JMeter has 2 modes:
  • GUI mode is for creating the test plan, checking it, debugging it BUT NOT FOR MASSIVE LOAD TEST. AWT Event Thread will disrupt your Load Test.
  • NON-GUI mode is for massive load testing, it is as simple as:

    <JMETER_HOME>/bin/jmeter -t <Path to Test Plan>  -n -l <path to results>/results.csv

Configure JMeter Java options to meet your requirements

  • Default JMeter java configuration comes with 512 Mo and very little GC tuning.
  • First ensure you set -Xmx option value to a reasonable value regarding your test requirements.
  • Then change MaxNewSize option in jmeter file to respect the original ratio between MaxNewSize and -Xmx.
  • Finally try tuning GC options only if you master this domain.

Use CSV as output for SaveService

XML is verbose, it takes resources to be written (CPU and memory) and for analysis, CSV is great so forget about XML. Furthermore, for massive load tests there are many result data you don't need.

So, in user.properties, add:
 
jmeter.save.saveservice.output_format=csv
jmeter.save.saveservice.data_type=false
jmeter.save.saveservice.label=true
jmeter.save.saveservice.response_code=true
jmeter.save.saveservice.response_data.on_error=false
jmeter.save.saveservice.response_message=false
jmeter.save.saveservice.successful=true
jmeter.save.saveservice.thread_name=true
jmeter.save.saveservice.time=true
jmeter.save.saveservice.subresults=false
jmeter.save.saveservice.assertions=false
jmeter.save.saveservice.latency=true
jmeter.save.saveservice.bytes=true
jmeter.save.saveservice.hostname=true
jmeter.save.saveservice.thread_counts=true
jmeter.save.saveservice.sample_count=true
jmeter.save.saveservice.response_message=false
jmeter.save.saveservice.assertion_results_failure_message=false
jmeter.save.saveservice.timestamp_format=HH:mm:ss
jmeter.save.saveservice.default_delimiter=;
jmeter.save.saveservice.print_field_names=true
 

Use Post-Processor and Assertion efficiently

Post-Processor and Assertions have a cost.
Ensure you use them when required and use the ones that consume the less memory and CPU:
 

Use Regular Expression Extractor

Use Regular Expression Extractor for extracting data BUT never ever check Body (unescaped), choose among:
  • Body
  • Headers
  • URL
  • Response Code
  • Response Message
Use efficient Regular expressions and extract as less data as possible
 

Avoid XPath Extractor when possible

XPath builds a DOM tree so it consumes CPU and memory, prefer Regular Expression extractor
 

Use Response Assertion or Size assertion

These 2 implementations fit 99% of requirements.
Avoid costly ones as:
  • XML Assertion
  • XML Schema Assertion
  • XPath Assertion

Use JSR 223 + Groovy for Scripting

You have a lot of options to do scripting with JMeter:
  • Beanshell
  • BSF and all supported languages Javascript, Scala , Groovy, Java ...
  • JSR223 and all supported languages Javascript, Scala , Groovy, Java ...
Although you can be lazy and choose the language you know, FORGET ABOUT IT.
Use the most efficient option, which is JSR223 + Groovy + Caching (supported since JMeter 2.8 in external script and in next upcoming JMeter 2.9 also supported with embedded scripts).

Using Groovy is as simple as adding groovy-VERSION-all.jar in <JMETER_HOME>/lib folder.
 
But of course ensure your script is necessary and efficiently written, DON'T OVERSCRIPT
 

Generate Reports AFTER the Run

Always generate Graphs and Reports AFTER the Load Test and NEVER EVER DURING IT
You can use JMeter reports:
  • Aggregate Graph
  • Response Time Graph
  • JMeter Plugins graphs

Distributed (Remote) Testing

Once you reach the limits of one machine, you can switch to distributed or remote testing.
But JMeter defaults are not fine for efficient remote testing, so in user.properties, add: mode=StrippedBatch
This will:
  • remove some data from the SampleResults as the response body, but do you need response body during a High Load Test, NO, DEFINITELY NO !
  • will send Sample Results as Batches and not for every sample reducing CPU, IO and network roundtrips

 

About the Author

Philippe Mouawad works as an Architect and technical expert for Ubik-Ingenierie where he leads the development of Commercial Apache JMeter based Plugins for RIA and Ajax technologies (GWT, Flex). Philippe is also a committer on the Apache JMeter project and member of the PMC since October 2011.

*This article is reprinted with permission from Philippe Mouawad and Ubik-Ingenierie. (Original posting December 24, 2012 http://www.ubik-ingenierie.com)