Taking Control of Performance Tests With Test Automation Failure Criteria

It seems simplistic to say it: the more control a tester can have over the performance tests they are executing, the better and more efficient that performance test will ultimately be. 

That control can come in the form of mock services that help remove testing bottlenecks. It can come in the form of streamlining test data usage with artificial intelligence. Or — as with the latest BlazeMeter release — it can come in the form stopping a performance test before it is completed when you know ahead of time that it will fail. 

In this blog, we will discuss the newest BlazeMeter feature — Test Automation Failure Criteria — and how it benefits testing teams and helps them shift their testing left. 

What is Test Automation Failure Criteria?

Traditionally, users would have to wait for a performance to be complete to optimize it, despite knowing it would fail.  

Now, BlazeMeter users can issue a STOP order when their Taurus tests are reaching the failure threshold. Doing so saves two crucial things — time and resources.  

BlazeMeter users can now convert Taurus’ Stop/Continue and Timeframe orders to BlazeMeter’s Failure Criteria with the newly available Stop and Sliding Windows, respectively.   

Run Taurus tests in the cloud with a Pass/Fail module that includes a Stop order and have the test run in BlazeMeter — stopping when the test is reaching a threshold as defined in the Taurus yaml script.  

How It Works

Like other Taurus pass/fail criteria, baseline-based criteria in YAML are translated to BlazeMeter failure criteria and appear in the test configuration UI. 

• KPI: the Key Performance Indicator defined in the pass/fail criteria 

• Condition: the comparison operator in the pass/fail criteria will be translated to the parallel phrase, e.g. “>=” will be translated to "Greater than or equal to". 

• Threshold: the value is calculated from the baseline and offset. For example, if average response time in baseline is 100 ms and the criteria had a “+5%” offset, the threshold would be baseline+5% which is 105 ms. 

• Use from baseline: true 

• Percentage offset = the percentage defined in the pass/fail without the +/- sign. The +/- sign will be used to add or reduce the offset percent from the baseline when calculating the threshold.