How I Used Taurus and Jenkins for Smoke Testing with JMeter
Smoke testing is a testing technique that is inspired from hardware testing, which checks for the “smoke” from the hardware components once the power is switched on. Smoke testing refers to testing the basic functionality of the build. If the test fails, the build is declared as unstable and it is NOT tested anymore until the smoke test of the build passes.
To detect performance issues as soon as possible and more, you can add smoke tests to your continuous integration (CI) workflow. CI allows us to run automated performance tests after each or every desired commit, and to send quick feedback with results to developers and all stakeholders. Good automated tests should cover all functionality, or at least most of it, so we can become familiar with the performance and behavior of our system from the earliest stages. As we all know, the earlier the better, as it’s easier to put out small fires rather than large ones.
First off, these were the tools I used:
- JMeter for load and performance testing
- Taurus to run JMeter from a Jenkins job
- Jenkins as a continuous integration agent
- BlazeMeter for test reporting
JMeter for Load and Performance Testing
First of all, create your smoke test and save it as a jmx file. This blog post will not go over how to create such a test, but there are also multiple resources you can learn JMeter from. I saved my file under the name SmokeTest.jmx.
Taurus to Run JMeter
When we talk about performance testing, JMeter is a great tool but, it can be very complex. I found Taurus was much more simple to use. It’s a free and open-source automation framework, which is basically an abstraction layer over JMeter. Taurus tests can be easily integrated with Jenkins since it is in plain text, uses YAML, and you can separate the scenario and the acceptance criteria from the script test flows. All of those factors helped me to run the tests even more smoothly.
In Taurus, every execution has a pass/fail status and there is a way to set this status based on runtime criteria. The special pass/fail module offers this functionality. Another useful Taurus feature of pass/fail criteria is auto-stop functionality, allowing to interrupt failed tests automatically, sparing the time and resources. Pass/fail criteria are specified as an array of criteria, set through the reporting item in configuration:
Reasons Why I Liked Using Taurus
- Extremely simple setup
- Ability to execute existing JMeter scripts
- The acceptance criterion can be separated, so each test could be executed with different criteria, for example when executing them in different environments
- Ability to create new tests from scratch using user-defined files
- Real-time reporting (in JMeter you don’t use listeners during your tests because they are too heavy, so this is a lighter alternative)
- Ability to integrate with Jenkins for CI
- Multiple test output formats
- Console statistics and pseudo-graphic charts during the test execution
- Easy way to define flexible pass/fail criteria.
- Automatically mark test(s) as failed
- Platform-independent- runs where Python and Java can run
So first, install Taurus, here.
Second, create a configuration YAML file for each of the environments and test types. This is the example I used:
YML example (SmokeTest.yml):
Third, run the test with the command: bzt SmokeTest.yml
Taurus will open up and display the test results.
Configuring Performance Tests Under CI
After running the tests in Taurus, I added performance tests to my CI, since I wanted to run a smoke load test with several virtual users for every release build. To do that, I configured the Jenkins job to run the tests on each build.
On Jenkins, I created a “New Item” (Job), entered an item name, selected “Freestyle Project” then saved it. On Job “General” configuration, I enabled the “This project is parameterized” option and added the following “String Parameters” and default values:
- Hold For
- Ramp Up
On the “Build Triggers” section, I marked the “Build after other projects are built” option and typed the project that I was going to execute before it. Finally, in the “Build” section, I selected “Execute shell” and typed the following command:
This does three things:
- Overrides the settings in the JMeter script and creates a new one in the test directory
- Starts JMeter in a non-UI mode using the new script and executes the test(s)
- Sends report data to BlazeMeter in real time, so I could analyze and share it on the fly
The best part is that now I could use one JMeter script.jmx for all environments and test types: all I needed to do was change the hostname and runtime settings like number of concurrent users in the YAML configuration.
Reporting with BlazeMeter
If you want the results to be stored in your existing BlazeMeter account, you'll need to specify the reporting settings in your configuration file (Never put API key into your main config files!). It is recommended to place the token setting in your personal per-user config ~/.bzt-rc to prevent it from being logged and collected in artifacts.
You might just want to use anonymous reporting, but you will lose the feature that I enjoy the most: comparison trends for response times, errors and hits/sec for different builds:
Overall, this was a very effective way to run smoke tests in my CI and I highly recommend the use of Taurus! Below are just a few of the business benefits of using this scheme.
Business Benefits of Using Taurus and Jenkins in Your CI
Besides BlazeMeter, the implementation requires only open-source tools and since you can use BlazeMeter reports for free, that means no additional investment is needed.
The Taurus files are easy to understand, so no expertise is required in JMeter for performance test development and maintenance. They can easily be handled by any team.
Daily reporting of the metrics helps to make concise decisions about the overall quality of the product under test.
This scheme is highly suitable for Agile based environments.
Congratulations! You now know how to run your smoke test through Taurus and Jenkins. To start testing with BlazeMeter, just put your JMX file or URL in the box right here, or request a BlazeMeter demo.