Introducing BlazeMeter Test Data: Build Test Data On-the-Fly

Every test requires data

Believe it or not, test data affects every test. Yes – every single test has to have test data. And what’s more – test data affects every test in a number of various aspects. Learning how to build test data is critical for every company today.

How to build test data

Test data is required to run any test. There are many ways to build test data – here are a few examples; manual test generation; test data is mannually entered by testers per requirements; automated test data generation; done with the help data generation tools and back-end data injection; done through SQL queries. 

So why has the talk around test data gained so much importance and popularity? On one hand – with GDPR and other privacy-related regulations and policies put into place, there is growing focus on the importance of sensitive data protection and data privacy in general. This is something that has to be respected also for data used during the testing. Test data cannot simply be taken from production and used for testing you must build test data. 

Every test has to be driven by some data. If we will stay even on a very simple level of complexity, in such case test data may be simply hard-coded directly into the test script, or you may have prepared some datasets (like csv files) that you use for your test. Test data is a natural and fundamental part of the tests – even in case of very simple tests.

For example, let’s take an application for ordering goods and the case of a test scenario that tests the ability to log in to your application, submit, and then display user details for the logged-in user and their submitted orders. When speaking about users, we can imagine data like username, e-mail address, phone number, credit card number, date of birth or selected date, and time of order delivery. The test has to use all this data to proceed through the test scenario; compose API requests using the data, fill them to the UI fields, and assert them in responses. 

Another important aspect is test data availability in the test environment as a critical prerequisite to be able to successfully execute a test. These are the cases where the database of your application under test has to be seeded with an initial data that your test expects to be returned by API or loaded to the UI in order to let the test scenario to progress – such as user information of the specific user account used for the test. Following our previous example, we can also mention some initial set of submitted orders to be returned by an API call, history of orders and perhaps some favorite items to be seeded in the application under test so the test that relies on them can use them during execution.

The last aspect to talk about is the variety of your test scenarios in order to cover more than just the trivial “happy path” scenario – i.e. scenario based on a tailored test-case that runs successfully for some specific sample test data set. Let’s assume a “submit order” user flow to test. It may be easy to put together data to just proceed somehow to success and get green “pass” result – but there are many ways how to divert from this simple and expected flow that your application has to cover as well. For example – a user submits an invalid delivery day (like yesterday’s date)... or makes a typo in credit card number… or has a discount because he or she is a premium member… or lives in a country where the selected delivery method is not supported… Simply speaking, even a test scenario focused on a simple use-case could explode into a lot of combinations when considering a potential variety of test data driving the test scenario that's why it's so critical to be able to build test data. 

When thinking more about the above example of the “submit order” use-case combinations, we realize that what actually drives the test are in fact primarily the test data – whether its valid or invalid delivery dates; correct or incorrect credit card numbers; basic or premium membership of the user; supported or unsupported destination for delivery, and so on. From the above it is clearly visible that what makes up the variety of test scenarios is the test data.

In other words, we can say that “the test is only as comprehensive as its test data set”.

📕 Related Resource: Learn more about What Is Test Data? How to Prepare For Scenarios With BlazeMeter

Build Test Data: Synthetic or from Database – WANTED!

Once it is clear that test data is a critical and integral part of the test design and a prerequisite to proper testing, let’s see what are the most typical challenges to build test data.

Let’s start simple again – when dealing with test data, the most obvious challenge is how to get test data effectively. Creating a table of various data combinations may be an option, but the drawback of this approach is that such list is static. It may be tedious and time-consuming to prepare it – especially when we consider a need for structured data like IDs or credit card numbers. It is not possible to simply use something simple like 12345 for a credit card number that expects specific and complex rules to be followed for a valid credit card number. And using real personal credit card numbers in a test does not sound like the right solution.

The other issue arises when I need to have test data specific to some relative context – such as today’s date. A test is expected to run repeatedly in the future when the statically provided date may become invalid or expired. Consider a scenario for selecting order delivery for “tomorrow”. With static data, the “tomorrow” will quickly become “yesterday”.

The other, and really big challenge, is setting up or seeding the desired test data into the test environment. There is an implicit complexity when dealing with the “physical” seeding of data into the system – there may be complex relationships between the data, multiple schemas,  databases or data stores (of different platforms) involved. This is not a trivial task at all. Also the related problem there is the data consistency. Suppose I have the right test data, I still have to make sure they are correctly distributed and available for all components that my test interacts with.

Importance of test data consistency

Let’s take a look at the diagram above. It describes our simple, but still illustrative enough, test case (and as we will see at the end, is not simple at all). 

To recap:

1. Enter login credentials into the application’s login screen
2. Perform login
3. Display a list of orders that were shipped for the logged-in user (suppose there is such a list displayed on the home page after user logs in)

Sounds easy, right? But the reality is pretty complex...

As you see in the diagram – there is user Joe with a password (let’s put aside how weak or strong this password is). These are the data that drives the test, and will be used to interact with the login step.

For the test scenario we need to see a list of orders that were shipped. This effectively means that for the logged in user we have to have some entries already present in the database in order to let the application return data that my test relies on. In this case, there has to be some entries relevant for Joe in the database.

Even in this simple setup I have to make sure that Joe appears in the data set that drives the test as well as it appears in the database. It does not matter whether the system first makes sure that there is some Joe as data that drives the test and subsequently seeds some entries for Joe to the database, or whether the entries for Joe are already in the database and the system will propagate Joe back to the test data set that drives the test – both approaches are valid.

Now, imagine the application is dealing with more databases (SQL, no-SQL, there are various types of data stores you can imagine…) and also with 3rd party services that you want to mock for the needs of your test. These mock services should be also aware of what data drives the test because it’s very likely they would need to return data which are consistent with the data that flows throughout the test scenario. And all of this is getting far from being easy…

Build Test Data in BlazeMeter 

Take any challenge or problem described above and you will get a reason why we are introducing Test Data Management for BlazeMeter – BlazeData. BlazeData provides a set of capabilities that helps to deal with all above related test data needs and resolve related challenges.

BlazeData opens new options how to:

• Easily define, maintain and get desired test data for your tests.
• Keep test data under the control across your test environment and its components.
• Make your tests and test scenarios more robust and comprehensive than before.

All of the above points are essential to enable agile teams to practice continuous testing properly and at an unprecedented level that BlazeMeter Continuous Testing platform supports. Yesterday, developers and testers had to face these challenges without proper and coherent support; today, BlazeMeter CT platform is here to help with these challenges.

UI Functional Scriptless testing is the first BlazeMeter area where we are bringing the power of BlazeData. As part of your Scriptless test you can now easily build up your test data model based on the synthetic data generation approach – just describe what data you need, link them to your test and your test will get them prior to its execution using the available synthetic data generator functions.

For example, let’s take a look on some of these functions:

• guid() – to get back random, but properly formatted uuid, e.g. “53297f41-617e-7eae-8523-ba1464d76c3f”
• randFromSeedlist(“firstnamegerman”) – to get random value from built-in seedlists (there are more than 50 seedlist available out of the box…) – in this example the function returns random German first name, e.g. “Erich”
• addDays(now(), -1) – to subtract one day from the test execution’s current date to effectively get the date of yesterday.
• randFromSeedlist("usaddress-multicol") to get random value from built-in seedlist of US street addresses, e.g.”533 3rd Ave W”
• randCreditCard(“VISA”) - to get vendor specific credit card number that follows rules of valid credit card number, e.g. 4400515803076376.
• ...plus additional 50+ functions for synthetic data generation you can use, nest inside each other, and combine to define the data needed for your testing. Check BlazeMeter documentation for all of the functions and seed lists available.

You can also specify the quantity of these test data sets based on your test scenario needs and required level of variety. Feel free to combine these new test data capabilities with your existing test data sets you may have already defined in CSV files to augment your existing static data with new data variants and dynamic nature where needed.

All of these new topics are covered in our free, BlazeMeter Scriptless Testing: Advanced Features course at BlazeMeter University. We encourage you to go through this training to get enabled on how to utilize this new dimension of capabilities for your testing.

This is just the beginning of the Test Data Management capabilities journey in BlazeMeter supporting your ability to build test data. We are working on enabling these options for the other test types across BlazeMeter Continuous Testing platform as well opening ways to link your tests with test data retrieved from the database using Broadcom’s Test Data Manager solution that will provide a way to publish data back to the database to seed environment with the exact data the test requires.

BlazeMeter Continuous Testing platform also focuses on the test environment dependencies which could be mocked using BlazeMeter Mock Services – to make them data-driven and keep the mock data consistent with data flowing through tests will be another focus point of BlazeData going forward.

Ready to try it yourself?

START TESTING NOW

Related Resources

• Test Data Setup: How to Use BlazeMeter to Get Reliable Data
• Learn How to Create Salesforce Lightning Automation Testing with BlazeMeter