Alexis Alvarez is a certified programmer analyst who works at Abstracta, with over 9 years of experience as an automation engineer. He has been involved in automation framework design and development, estimation effort, automation development, review and continuous integration setup for many clients, such as Google, LAN Airlines, Carnival Cruises, and McKesson. He is skilled in development, software testing and continuous process improvement.

Learn JMeter in 5 Hours

Start Learning
Slack

Run massively scalable performance tests on web, mobile, and APIs

Apr 20 2017

How I Used Taurus and Jenkins for Smoke Testing with JMeter

Smoke testing is a testing technique that is inspired from hardware testing, which checks for the “smoke” from the hardware components once the power is switched on. Smoke testing refers to testing the basic functionality of the build. If the test fails, the build is declared as unstable and it is NOT tested anymore until the smoke test of the build passes.

 

To detect performance issues as soon as possible and more, you can add smoke tests to your continuous integration (CI) workflow. CI allows us to run automated performance tests after each or every desired commit, and to send quick feedback with results to developers and all stakeholders. Good automated tests should cover all functionality, or at least most of it, so we can become familiar with the performance and behavior of our system from the earliest stages. As we all know, the earlier the better, as it’s easier to put out small fires rather than large ones.

 

As an automation engineer of almost a decade, I integrated Apache JMeter™ scripts as part of a smoke test, by using Taurus in my Jenkins pipeline. This blog post will explain how.

 

First off, these were the tools I used:

  • JMeter for load and performance testing
  • Taurus to run JMeter from a Jenkins job
  • Jenkins as a continuous integration agent
  • BlazeMeter for test reporting

 

JMeter for Load and Performance Testing

 

First of all, create your smoke test and save it as a jmx file. This blog post will not go over how to create such a test, but there are also multiple resources you can learn JMeter from. I saved my file under the name SmokeTest.jmx.

 

Taurus to Run JMeter

 

When we talk about performance testing, JMeter is a great tool but, it can be very complex. I found Taurus was much more simple to use. It’s a free and open-source automation framework, which is basically an abstraction layer over JMeter. Taurus tests can be easily integrated with Jenkins since it is in plain text, uses YAML, and you can separate the scenario and the acceptance criteria from the script test flows. All of those factors helped me to run the tests even more smoothly.

 

In Taurus, every execution has a pass/fail status and there is a way to set this status based on runtime criteria. The special pass/fail module offers this functionality. Another useful Taurus feature of pass/fail criteria is auto-stop functionality, allowing to interrupt failed tests automatically, sparing the time and resources. Pass/fail criteria are specified as an array of criteria, set through the reporting item in configuration:

 

- module: fail-criteria
  criterias:
  - avg-rt>250ms for 30s, continue as failed
  - failures>10% for 10s, continue as failed
  - failures>70% for 15s, stop as failed

 

Reasons Why I Liked Using Taurus

  • Extremely simple setup
  • Ability to execute existing JMeter scripts
  • The acceptance criterion can be separated, so each test could be executed with different criteria, for example when executing them in different environments
  • Ability to create new tests from scratch using user-defined files
  • Real-time reporting (in JMeter you don’t use listeners during your tests because they are too heavy, so this is a lighter alternative)
  • Ability to integrate with Jenkins for CI
  • Multiple test output formats
  • Console statistics and pseudo-graphic charts during the test execution
  • Easy way to define flexible pass/fail criteria.
  • Automatically mark test(s) as failed
  • Platform-independent- runs where Python and Java can run

 

So first, install Taurus, here.

 

Second, create a configuration YAML file for each of the environments and test types. This is the example I used:

 

YML example (SmokeTest.yml):

 

---
 execution:
   concurrency: 20
   ramp-up: 1m
   hold-for: 10m
   scenario:
     script: SmokeTest.jmx
     properties:
         hostname: EXAMPLE_HOST.COM
  
 reporting:
   - module: blazemeter
     test: Smoke Test
   - module: junit-xml
     filename: report.xml
     data-source: pass-fail
   - module: fail-criteria
     criterias:
     - avg-rt>250ms for 30s, continue as failed
     - failures>10% for 10s, continue as failed
     - failures>70% for 15s, stop as failed

 

Third, run the test with the command: bzt SmokeTest.yml

 

Taurus will open up and display the test results.

 

Configuring Performance Tests Under CI

 

After running the tests in Taurus, I added performance tests to my CI, since I wanted to run a smoke load test with several virtual users for every release build. To do that, I configured the Jenkins job to run the tests on each build. 

 

On Jenkins, I created a “New Item” (Job), entered an item name, selected “Freestyle Project” then saved it. On Job “General” configuration, I enabled the “This project is parameterized” option and added the following “String Parameters” and default values:

  • Concurrency
  • Hold For
  • Iterations
  • Ramp Up

 

integrating jmeter smoke testing to jenkins

 

integrating smke tesing with jmeter into jenkins

 

On the “Build Triggers” section, I marked the “Build after other projects are built” option and typed the project that I was going to execute before it. Finally, in the “Build” section, I selected “Execute shell” and typed the following command:

 

bzt SmokeTest.yml -o execution.0.concurrency=${Concurrency} -o execution.0.hold-for=${HoldFor} -o execution.0.ramp-up=${RampUp} -o execution.0.iterations=${Iterations}

 

This does three things:

  • Overrides the settings in the JMeter script and creates a new one in the test directory
  • Starts JMeter in a non-UI mode using the new script and executes the test(s)
  • Sends report data to BlazeMeter in real time, so I could analyze and share it on the fly

 

The best part is that now I could use one JMeter script.jmx for all environments and test types: all I needed to do was change the hostname and runtime settings like number of concurrent users in the YAML configuration.

 

Reporting with BlazeMeter

 

If you want the results to be stored in your existing BlazeMeter account, you'll need to specify the reporting settings in your configuration file (Never put API key into your main config files!). It is recommended to place the token setting in your personal per-user config ~/.bzt-rc to prevent it from being logged and collected in artifacts.

 

You might just want to use anonymous reporting, but you will lose the feature that I enjoy the most: comparison trends for response times, errors and hits/sec for different builds:

 

reporting with blazemeter after smoke testing

 

Overall, this was a very effective way to run smoke tests in my CI and I highly recommend the use of Taurus! Below are just a few of the business benefits of using this scheme.

 

Business Benefits of Using Taurus and Jenkins in Your CI

 

Besides BlazeMeter, the implementation requires only open-source tools and since you can use BlazeMeter reports for free, that means no additional investment is needed.

 

The Taurus files are easy to understand, so no expertise is required in JMeter for performance test development and maintenance. They can easily be handled by any team.

 

Daily reporting of the metrics helps to make concise decisions about the overall quality of the product under test.

 

This scheme is highly suitable for Agile based environments.

 

Congratulations! You now know how to run your smoke test through Taurus and Jenkins. To start testing with BlazeMeter, just put your JMX file or URL in the box right here, or request a BlazeMeter demo.

 

 

Interested in writing for our Blog?Send us a pitch!

Your email is required to complete the test. If you proceed, your test will be aborted.