Yuri Bushnev is a Head of System Quality at AlphaSense, with more than 5 years of experience in Quality Assurance, DevOps and Development.

Learn JMeter in 5 Hours

Start Learning
Slack

Run massively scalable performance tests on web, mobile, and APIs

Jul 11 2017

Regression Testing With JMeter - Learn How

Apache JMeter™ is one of the best performance testing tools. It is especially powerful for non-functional testing, where we already know the application works correctly, but still need to ensure it works reliably under different kinds of loads.

 

But did you know that JMeter can be successfully used for functional tests, too? Well, it can! With each version, JMeter provides more and more out-of-the-box functional test elements available. Due to fact that JMeter supports lots of different features, it is very easy to find everything that you need for functional tests in its arsenal.

 

By functional tests we mean tests that verify the existing functionality and main application user workflows. One of the most important roles of functional testing is to maintain every piece of your application in good quality, and test it thoroughly from release to release. You can write and use automation scripts to test the same functionality each time you need, just by pushing a button. This type of verification is called regression testing. 

 

Once you know how to make performance and functional tests with JMeter, you can easily use this one app as standalone solution for a whole regression suite. Let’s learn how!

 

How is it that we can think about JMeter as a tool for regression testing?  Nowadays, JMeter contains everything you might want for creating a reliable regression test suite:

  • A variety of samplers that allow you to simulate any kind of request to your application (HTTP(S), FTP, JDBC and many others)
  • Powerful assertions for a detailed verification of each step in your test workflow (on response code, body, headers and so forth)
  • A diversity of listeners to verify results and make the debugging process easy

 

In this article we are going to create an end-to-end regression scenario, which contains both functional and non-functional checks. Our tested application will be the well known http://blazedemo.com/ web application. These are the list of checks we are going to perform in this regression scenario.

 

I decided to include functional checks (step 1-2) and one non-functional (step 3) just to show how we can combine all kinds of checks together in one suite:

 

1. Open the http://blazedemo.com/ application home page

 

Verify the application opened successfully and we got the welcome page

 

2. Find flights from Paris to Rome

 

Verify we could find at least some flights from Paris to Rome

 

3. Perform flight reservation

 

Verify the flight reservation request was completed in less than 1 second

 

Functional Regression Tests

 

Now that the test case is ready, we are good to go! Let’s create a template for our test. As we are going to verify the functional aspect instead of testing a load, it is enough to use only one user and to perform only one loop of checks.

 

To do so, create a ‘Thread Group’ with only one user and ‘1’ as the ‘Loop’ value. Until you have a huge regression suite,  I would recommend keeping your regression tests in one separate ‘Thread Group’, which contains multiple ‘Simple Controller’ elements to keep it simple. These simple controllers are used for better grouping and separation of your tests.

 

1. So, first of all, this is our ‘Thread Group’:

 

regression testing with jmeter

 

2. Add a ‘Simple Controller’, which will be used as a container for the end-to-end regression scenario:

 

Right click on Thread Group -> Add -> Logic Controller -> Simple Controller

 

apache jmeter regression testing

 

Regression Test Step 1 - Verify the application opened successfully and we got the welcome page

 

The first thing we need to test is opening the application (request the home page) and confirming it was opened successfully. The easiest way is to create response body assertions.

 

We know the http://blazedemo.com/ home page contains a “Welcome to the Simple Travel Agency!” text string. So to ensure the homepage was opened, all we need to do is to perform an assertion on that.

 

The best practice for unit testing: “One test = one assertion”. But since we are not performing unit tests, but rather an end-to-end integration scenario, the more assertions the better. This time we will limit the number of assertions, to stay focused on the main topic, but keep this in mind.

 

3. Our first sampler should reach the main page:

 

jmeter functional testing

 

4. At this point we can add our first assertion:

 

jmeter regression functional testing

 

We also recommend you add response code assertions on each step of your test. The reason for this is simple. If your response body assertion fails, all you know is that the response body doesn’t contain the expected text pattern. But it might be absent for many reasons: maybe the page was not loaded properly, maybe the request was incorrect or maybe you don’t have permissions to see that page.

 

Each of these reasons has its own response code. For an incorrect request you will get 400 while for a server error you might get 500. The response code will tell you much more than just a failed assertion. That’s why it is important to keep adding response code assertions to each step of your test script.

 

5. Add a response assertion to the first request. There are some exceptions, but you should usually expect the 200 response code if the page successfully loaded.

 

regression testing with jmeter

 

6. The most convenient way to verify the regression test result is to add the ‘View Results Tree’ listener:

 

Right click on Thread Group -> Add -> Listener -> View Results Tree

 

We can see the assertions passed and everything is green.

 

7. You can confirm the assertions actually work by trying to edit the expected response code for a different one, like 201:

 

jmeter regression testing

 

 

 

Confirmed. The assertions work as expected and we can continue to our test implementation.

 

Regression Test Step 2 - Verify we could find at least some flights from Paris to Rome

 

8. Based on our test case, the next step is to find the flight from Paris to Rome. For that we need to send a POST request with appropriate arguments:

 

jmeter regression testing

 

On this step, our test case states we need to ‘Verify we could find at least some flights from Paris to Rome’. This time you cannot say which text we should expect in the response, which makes our verification more tricky.

 

But due to powerful JMeter assertion capabilities, we always have lots of different ways to achieve our test goals. This time we can use ‘XPath Assertion’ element to verify that we have at least one flight.

 

9.  We just need to find appropriate XPath that represents one flight. We can use my favorite Firefox plugin to do that - FirePath.

 

jmeter regression tesring with firepath

 

As you can see, the ‘//table[contains(@class,'table')]/tbody/tr’ XPath represents flights on this page. If at least one element with the mentioned XPath is presented on the page, we can say that the search returned at least some flights.

 

10. Let’s add the assertion to our test. The ‘XPath Assertion’ can be added this way:

 

Right click on Thread Group -> Add -> Assertion -> XPath Assertion

 

Keep in mind that the ‘XPath Assertion’ element also validates HTML page structure. So if your page is not following all HTML rules, you have to check the “Use Tidy” and “Quiet” checkboxes, to avoid errors:

 

jmeter regression testing with assertions

 

Our test passed. This shows how useful the XPath assertion is for regression tests.

 

jmeter continuous testing regression testing

 

Regression Test Step 3 - Verify the flight reservation request was completed in less than 1 second

 

For our next step we need to proceed with theflight reservation. Reservation endpoints require flight details, like the flight number and airline. In order to do that, we need to use one more powerful JMeter instrument - JMeter PostProcessors.

 

JMeter PostProcessors allow you to parse the response and to fetch data into variables. These variables can be used in the test afterwards. In the same way we did with the XPath assertion, we need to find out which XPath which represents the flight number and airline.

 

Based on FirePath we can find that ‘//table[contains(@class,'table')]/tbody/tr[1]/td[3]’ XPath represents airline while ‘//table[contains(@class,'table')]/tbody/tr[1]/td[2]’ XPath represents flights number.

 

11. Add the ‘XPath Extractor’ element for airline:

 

Right click on Thread Group -> Add -> Post Processors -> XPath Extractor

 

jmeter regression testing functional testing load testing

 

12. We have done the same for flight number but this time with ‘//table[contains(@class,'table')]/tbody/tr[1]/td[2]’ XPath, and now these values are available with the ${Airline} and ${FlightNumber} variables. Now all we need is to use these variables in consequent requests, to perform flight reservation.

 

regression testing with open-source

 

We know how to create response assertions, page text validation and other tricks to confirm the application’s functionality, by setting different checks based on expectations. But if you remember, in the introduction we mentioned that JMeter is a great tool for regression testing because we can do both functional and non-functional verifications.

 

Non-Functional Regression Tests

 

So now is a good time to talk about non-functional regression checks. The first that comes to mind is request performance checks. These are important for tracking and catching basic performance regressions even before running huge load tests.

 

To reach this goal we can use the ‘Duration Assertion’, which asserts the request was completed in a reasonable amount of time. Let’s say we have specifications which state the flight reservation shouldn’t take more than one second. In this case, we can just add an assertion on that, and use as a regression check for each test cycle.

 

13. To add the ‘Duration Assertion’: 

 

Right click on Thread Group -> Add -> Assertion -> Duration Assertion

 

open-source regression testing

 

Usually, you need to test the same scripts across different environments (test, pre-production and so on). That’s why it is very important to make your test configuration flexible and avoid manual editing scripts each time you need to run the suite across other environments.

 

Let’s assume we have different test environments with different URLs. As currently we specify that URL in each HTTP sampler, then if we need to change it, we have to go over each sampler. If we have few of them it is fine. But imagine if you have huge regression suite with hundred of samplers. Then you definitely want to parametrize all samplers and specify that environment URL only in one place for editing.

 

14. To do so, we can add the ‘User Defined Variables’ element:

 

Right click on Thread Group -> Add -> Config Element -> User Defined Variables

 

regression tesring with open-source jmeter

 

15.  Remove the blazedemo.com URL from each sampler and specify ${EnvironmentUrl} as the server name to ‘HTTP Request Defaults’:

 

Right click on Thread Group -> Add -> Config Element -> HTTP Request Defaults

 

open-source regression testing

 

From this moment on, if we need to run our regression suite against another environment, we can change the URL in just one place.

 

Congratulations! Your first regression suite in JMeter is ready. You can save the regression suite as a JMX file and run it even without the JMeter UI application itself. Check out this article to get a better idea how.

 

To sum up, you should always use the appropriate tools for your different needs. The purpose of this article was not to force you to burn your current regression tests and use JMeter to create new ones. Rather, to show that in addition to it’s main functionality, JMeter provides outstanding capabilities for full stack testing of your application. Therefore, if you do not have enough resources to use different tools for different needs, or you just feel very comfortable with the JMeter UI, then just go ahead with best practices and check out all the available JMeter options to achieve your regression goals.

 

Learn more about JMeter from our JMeter academy.

 

To start testing BlazeMeter, which enhances JMeter abilities, just put your URL or JMX file in the box below and your test will start in minutes.

 

Click here to subscribe to our newsletter.

You might also find these useful:

Interested in writing for our Blog?Send us a pitch!

Your email is required to complete the test. If you proceed, your test will be aborted.