Konstantin Tonkov has been working in the software testing sphere for 7 years and is now at ISS Art. In addition to coding, Konstnatin likes playing board and video games.

Learn JMeter in 5 Hours

Start Learning
Slack

Run massively scalable performance tests on web, mobile, and APIs

Aug 21 2018

JMeter’s Raw Data Source PreProcessor: The Ultimate Guide

There are lots of ways to extract data from a file to Apache JMeter™ variables. For binary data, one of the most handy tools is the Raw Data Source preprocessor plug-in. It especially shines when you have to process a large amount of raw data, like when you need to collect data from various samplers, such as HTTP, TCP and UDP, or when have a large data file with a lot of records, and you need to distribute the values of these records across your Test Plan.

 

However, the Raw Data PreProcessor does not support HTTPS, and it limits your test plan to one endpoint only. We recommend using it when you want to be lean with the resources used to make requests. The Raw Data Source Preprocessor was written by BlazeMeter’s Andrey Pokhilko.

 

How does it work? The Raw Data Source PreProcessor reads raw and  binary data, prefixed by a byte-length string, from a file. The prefixes divide the data into chunks, and the preprocessor parses each chunk to a defined JMeter variable. This variable can be used in the script samplers. So now, the same variable has various values, according to the different chunks. These values can be used in your script.

 

File format for the Raw Data Source PreProcessor:

 

50
GET /page/95561
Host: localhost
Connection: close
 
48
GET /page/559
Host: localhost
Connection: close
 
49
GET /page/4032
Host: localhost
Connection: close

 

I will show how to create this type of file in JMeter in the next section of the blog. As you can see, a data file consists of a data chunks, and each chunk follows a number that shows the PreProcessor how many bytes of data from a followed chunk to assign to a specified variable.

 

Each time a sampler is invoked in the preprocessor scope, the preprocessor does the following:

 

1) Reads the data until the end-of-line marker (CR or LF) is located, and tries to parse it as a length-prefix value. If it fails, it repeats this step.

 

2) Reads the data for the calculated byte-length and assigns it to a variable in a sampler.

 

3) Returns to the first step and keeps going in the file.

 

So, for the data above, it will assign:

 

GET /page/95561
Host: localhost
Connection: close

 

to the specified variable for the first sampler.

 

GET /page/559
Host: localhost
Connection: close

 

will be assigned to the variable in the second sampler run and so on.

 

Raw Data Source PreProcessor Component Description

 

To add the Raw Data PreProcessor, go to: Thread Group -> Add -> Pre Processors -> jp@gc Raw Data Source PreProcessor

 

Make sure you have JMeter plugins in advance.

 

how to run the raw data preprocessor in jmeter

 

  • Rewind on the end of file: Allows the component to process a file from the beginning after the end of a file is reached.
  • Data file path: The location of the data file.
  • Variable name: The variable that stores the chunk of processed data. For each sampler in the thread the preprocessor reads and parses the next chunk.
  • Encode read data as HEX: When selected, it converts processed data to the HEX-encoded format.
  • Checks file consistency: Checks the consistency of the file and shows you its status as well as the amount of chunks of data processed. Any errors will be shown here too.

 

Gathering Raw Data and Creating a Raw Data File

 

The Flexible File Writer listener JMeter plug-in allows you to form a file that will be a great source of data for the Raw Data preprocessor. In such a file, the response message for a sample follows a received bytes value, or a sent bytes is followed by a sample request.

 

Let’s collect data from samplers and write it to a file in the length-prefixed format.

 

1. Add a Thread Group to your Test Plan. Right click -> Add -> Threads -> Thread Group

 

creating a raw data file for a jmeter test

 

2. Set the Number of Threads (users). For our example we will generate 10 users to collect a fair amount of data.

 

3. Add an HTTP Request sampler component. Right click -> Add -> Samplers -> HTTP Request

 

jmeter load testing with the raw data preprocessor

 

4. Configure your sampler

 

5. Add the Flexible File Writer listener component to your Thread Group. Right click -> Add -> Listener -> jp@gc - Flexible File Writer

 

jmeter, raw data preprocessor guide

 

6. Clear the Write File Header field, set your desired Filename and modify “Record each sample as” field:

 

For request data:

 

raw data preprocessor plugin tutorial

 

For response data:

 

how to use the flexible file writer in jmeter

 

7. Now, run your script to get a file suitable for the Raw Data preprocessor!

 

Data Preprocessing

 

Data gathered in the previous example might be used by the preprocessor in a number of ways. You may treat response messages as a reference variable for regular expression extractors in future scripts, or, alternatively, you may use saved requests as a source for the HTTP Raw Request sampler plug-in.

 

As the name suggests, it’s as if the HTTP Raw Request sampler was created for the Raw Data preprocessor. It would be a mistake to miss out on such an iconic example, so we won’t. In this example I will use a data file that contains requestData from the previous example, to construct a HTTP request for the HTTP Raw Request sampler by using the Raw Data preprocessor.

 

1. First, we need to set the variables for the hostname and path. We will need these variables later. This has to be done in a User Defined Variable Config Element or right in the Test Plan rather than in an HTTP Request Defaults, because an HTTP Raw Request sampler does not refer to an HTTP Request Defaults config element.

 

2. Add a Thread Group to your Test Plan. Right click -> Add -> Threads -> Thread Group

 

3. Set the Number of Threads (users). As long as we check the “Rewind on end of file” property for our Raw Data Source PreProcessor, we can use any number of users, even exceeding the number of requests in our data file.

 

4. Add a View Results Tree listener. Right click -> Add -> Listener -> jp@gc - View Results Tree

 

5. Add a Raw Data Source PreProcessor to the thread. Right click -> Add -> Pre Processors -> jp@gc - Raw Data Source PreProcessor

 

jmeter, data preprocessing

 

6. Specify a source file and a variable for the processed data.

 

open source, raw data source preprocessor

 

7. Add an HTTP Raw Request sampler component. Right click -> Add -> Samplers -> jp@gc - HTTP Raw Request

 

jmeter tutorial, raw data

 

8. Set Host Name, Port, and Timeout fields. Set your Raw Data variable as the Request Data.

 

jmeter, raw data variables

 

Are we ready to run our script? Not yet.

 

Remember: “requestData” of the sample is not a proper HTTP request! Do not paste requestData from a regular HTTP sampler to a HTTP raw request.

 

The Flexible File Writer also has a pitfall: when it writes a requestData to a file, it omits the carriage return special symbol “\r” leaving only the newline symbol “\n”, making the resulting string a literally “Bad Request”. Moreover, HTTP Raw Request does not respect the Header Manager, so we have to manually add the required headers. This can be achieved via a JSR223 PreProcessor script. Here is an example for POST requests (keep in mind that the contentType variable is set as a user defined variable in this case):

 

files for jmeter

 

if (vars.get("rawData")[0..3] == "POST"){
	String postDataCookies = vars.get("rawData").substring(vars.get("rawData").indexOf("POST data:") + 11)
	String postData = postDataCookies.substring(0, postDataCookies.indexOf("\n"))
	String methodPath = "POST " + vars.get("path") + " HTTP/1.1\r\n"
	String contentTypeHeader = "Content-Type: " + vars.get("contentType") + "\r\n"
	String contentLengthHeader = "Content-Length: " + postData.getBytes().length + "\r\n"
	String hostHeader = "Host: " + vars.get("host") + "\r\n"
	vars.put("rawData", methodPath + contentTypeHeader + contentLengthHeader + hostHeader + "\r\n" + postData)
}

 

Now it’s ready! Run the script.

 

Check the View Results Tree listener for the results. Each sampler got its own request data from the PreProcessor and sent it exactly as it was recorded in the previous example.

 

jmeter testing with raw data

 

The Raw Data preprocessor enables you to use any kind of external scripts for the data file generation, and you can use the processed variables in as many ways as you like. So there are many applications for this element.

 

After creating your JMeter file, you can upload it to BlazeMeter to run it. In BlazeMeter you can massively scale the number of users and locations, collaborate on tests and reports, share results with managers and get drilled down insightful analytics or compare results over time.

 

Put your URL in the box below to try out BlazeMeter, or request a live demo from one of our performance engineers.

Interested in writing for our Blog?Send us a pitch!

Your email is required to complete the test. If you proceed, your test will be aborted.