Blog
May 4, 2022
API Performance Testing Tools: JMeter, Taurus, and BlazeMeter
Performance Testing,
API Testing
Let's talk about APIs from a performance testing perspective. Imagine you have a web app that consumes an API. Once your web application is publicly available, multiple clients will be hitting the same service at the same time. That being the case, you would definitely want to execute performance tests to be sure your servers are accessible to all users, even under a heavy load, and that there are no API delays.
But how can you do it? Many engineers are not sure how to run software performance testing at the API level. In this article, I am going to walk you through different ways for running API performance testing tools using three of the most useful API load testing tools: JMeter, Taurus, and BlazeMeter.
We will be using a public API called Deck of Cards in all of the examples. You can find the documentation here.
What are API Performance Testing Tools?
Back to topAPIs are usually consumed by multiple systems, which makes them an important testing target. API performance testing tools will not only detect failures on the API itself, but also on other, third-party components.
API Performance Testing Tool: JMeter
JMeter is an open-source tool used for performance testing. It can be used on any platform that supports Java, which makes it a good option for automating and executing API performance tests.
Let’s see how to create an API test:
1. Open JMeter
2. Right-click on Test Plan - Add - Threads (Users) - Thread Group
11. As we saw in step 7, the response to the request is:
{ "success": true, "deck_id": "3p40paa87x90", "shuffled": true, "remaining": 52 }
So if we want to extract the value of deck_id, we may add a regular expression like this one: "deck_id":\s"(.*)",
12. Going back to JMeter, we will complete the fields as follows:
Remember to remove or disable the View Results Tree when running high load tests, in order to improve the performance of the load generator.
📕 Related Resource: Learn more about JMeter Performance Testing: Upload and Download Scenarios
Back to topAPI Performance Testing Tool: Taurus
Taurus is an open-source API testing tool that automates and runs performance tests. Unlike JMeter, Taurus doesn’t have a GUI, but it enables simplified running of other tests from tools like Gatling, Locust, Selenium, and others.
In Taurus, you can write your script on a YAML File and run it on a terminal. YAML is an easy-to-understand language (which leads to a lower learning curve). This makes it easy to maintain using version control tools such as GitHub. It is also an advantage when adding your tests to continuous integration tools like Jenkins.
Here you can read the complete guide to API testing with Taurus. Now let’s see how to test our Deck of Cards API.
Every YAML file starts with an “execution” section. This is where you define the parameters corresponding to the load and the execution engine to be used (which is JMeter by default).
execution:
- executor: jmeter
concurrency: 5
ramp-up: 10s
hold-for: 60s
scenario: deck of cards
- scenario: it is the name of the scenario that is going to be executed. It must be defined in the “scenarios” section (which we will see next). It is important to keep in mind that this field does not have a default value, therefore we must specify it. If not, Taurus will not know which scenario to run.
- concurrency: the number of target concurrent virtual users
- ramp-up: the ramp-up time for reaching target concurrency
- hold-for: the time to hold the target concurrency
After defining the test, it’s time to add the “scenarios” section to the YAML file. This section tells Taurus which steps to execute during the test.
1. First, we are going to name the scenario. This name matches the scenario in the “execution” section.
execution:
- executor: jmeter
concurrency: 5
ramp-up: 10s
hold-for: 60s
scenario: deck of cards
scenarios:
deck of cards:
2. Then we will add requests and transactions. The transactions are the same as the Transaction Controllers on JMeter. They are useful to identify the different steps of our tests.
execution:
- executor: jmeter
concurrency: 5
ramp-up: 10s
hold-for: 60s
scenario: deck of cards
scenarios:
deck of cards:
requests:
- transaction: Shuffle the cards
do:
3. Inside those, we will list the individual requests to be executed. We have a GET HTTP request to Shuffle the cards.
execution:
- executor: jmeter
concurrency: 5
ramp-up: 10s
hold-for: 60s
scenario: deck of cards
scenarios:
deck of cards:
requests:
- transaction: Shuffle the cards
do:
- url: 'http://deckofcardsapi.com/api/deck/new/shuffle/?deck_count=1'
method: GET
- Now we are going to add an assertion to check if we are getting the right response.
execution:
- executor: jmeter
concurrency: 5
ramp-up: 10s
hold-for: 60s
scenario: deck of cards
scenarios:
deck of cards:
requests:
- transaction: Shuffle the cards
do:
- url: 'http://deckofcardsapi.com/api/deck/new/shuffle/?deck_count=1'
method: GET
assert:
- contains:
- deck_id
5. As we did on JMeter, on the “Shuffle the cards” request we extracted the value of deck_id and used it later on the “Draw a card” request.
execution:
- executor: jmeter
concurrency: 5
ramp-up: 10s
hold-for: 60s
scenario: deck of cards
scenarios:
deck of cards:
requests:
- transaction: Shuffle the cards
do:
- url: 'http://deckofcardsapi.com/api/deck/new/shuffle/?deck_count=1'
method: GET
assert:
- contains:
- deck_id
extract-regexp:
deck_id:
regexp: '\"deck_id\":\s\"(.*)\",'
default: NOT_FOUND
match-no: 1
template: 1
subject: body
scope: all
- regexp: the regular expression
- default: the default value to use when the regular expression doesn’t match.
- match-no: If multiple values have matched, you specify which match to use. In this example, we are matching the first result.
- template: Which capture group to take
- subject: the subject for search (body, headers,http-code, url)
- scope: If we set the scope to “all”, it will check main and sub-samples
- Let’s add a transaction called “Draw a card”, which is the second step of our test.
execution:
- executor: jmeter
concurrency: 5
ramp-up: 10s
hold-for: 60s
scenario: deck of cards
scenarios:
deck of cards:
requests:
- transaction: Shuffle the cards
do:
- url: 'http://deckofcardsapi.com/api/deck/new/shuffle/?deck_count=1'
method: GET
assert:
- contains:
- deck_id
extract-regexp:
deck_id:
regexp: '\"deck_id\":\s\"(.*)\",'
default: NOT_FOUND
match-no: 1
template: 1
subject: body
scope: all
- transaction: Draw a card
do:
7. Inside this transaction, we’ll add a GET HTTP request to draw a card.
execution:
- executor: jmeter
concurrency: 5
ramp-up: 10s
hold-for: 60s
scenario: deck of cards
scenarios:
deck of cards:
requests:
- transaction: Shuffle the cards
do:
- url: 'http://deckofcardsapi.com/api/deck/new/shuffle/?deck_count=1'
method: GET
assert:
- contains:
- deck_id
extract-regexp:
deck_id:
regexp: '\"deck_id\":\s\"(.*)\",'
default: NOT_FOUND
match-no: 1
template: 1
subject: body
scope: all
- transaction: Draw a card
do:
- url: 'http://deckofcardsapi.com/api/deck/${deck_id}/draw/?count=2'
method: GET
- Finally, we are going to add an assertion to the “Draw a card” request.
execution:
- executor: jmeter
concurrency: 5
ramp-up: 10s
hold-for: 60s
scenario: deck of cards
scenarios:
deck of cards:
requests:
- transaction: Shuffle the cards
do:
- url: 'http://deckofcardsapi.com/api/deck/new/shuffle/?deck_count=1'
method: GET
assert:
- contains:
- deck_id
extract-regexp:
deck_id:
regexp: '\"deck_id\":\s\"(.*)\",'
default: NOT_FOUND
match-no: 1
template: 1
subject: body
scope: all
- transaction: Draw a card
do:
- url: 'http://deckofcardsapi.com/api/deck/${deck_id}/draw/?count=2'
method: GET
assert:
- contains:
- deck_id
The final script will look like this:
execution:
- executor: jmeter
concurrency: 5
ramp-up: 10s
hold-for: 60s
scenario: deck of cards
scenarios:
deck of cards:
requests:
- transaction: Shuffle the cards
do:
- url: 'http://deckofcardsapi.com/api/deck/new/shuffle/?deck_count=1'
method: GET
assert:
- contains:
- deck_id
extract-regexp:
deck_id:
regexp: '\"deck_id\":\s\"(.*)\",'
default: NOT_FOUND
match-no: 1
template: 1
subject: body
scope: all
- transaction: Draw a card
do:
- url: 'http://deckofcardsapi.com/api/deck/${deck_id}/draw/?count=2'
method: GET
assert:
- contains:
- deck_id
To execute the script, navigate to the location where you saved the script in your command line window and execute bzt.yml. Where “.yml” is the name of the script.
As the script executes, a live report will be displayed showing various metrics.
All of these tools are great options for API performance testing. Each offers different benefits:
- JMeter allows you to create tests with a GUI, making it more user-friendly and letting them run from your machine.
- Taurus enables simplified coding with quick results.
- BlazeMeter lets you scale your JMeter and Taurus scripts to thousands of users and run your test from different locations using a user-friendly dashboard for looking at the live results.