Node-RED Flows and IoT Device Testing with BlazeMeter

One important aspect of developing code for devices is performance testing — specifically IoT device testing. The reason is quite simple: you don’t want to ship out your devices if the supporting infrastructure can’t carry the traffic generated by them. This could cause your device not to perform properly or even malfunction. 

But how do you go about testing IoT devices? Can you use IBM Node-RED flows? Find out in this blog! 

What Is IoT Device Testing?

IoT device testing refers to process of testing IoT devices against real-world conditions. 

I’ve seen many discussions on IoT forums about testing devices, and the general consensus is that everyone does it differently if at all. There doesn’t seem to be a formulated methodology for IoT device testing. The purpose of this article is to help fill that void for users of Node-RED.  

In previous articles, I showed how JMeter and Locust could be used for IoT testing. Here, we’re going to take a look at the popular Node-RED programming tool, which is used to code devices using "flows". I’ll show how you can use Mocha to run Unit tests written in JavaScript to test the flows. Finally, we’ll discuss how to load test them with Taurus and BlazeMeter.

Why and When to Load Test IoT Devices

IoT device testing programmers can use many different programming languages for coding the functionality of an embedded device. Common languages are C, C++, Python, and Node-RED - which will be discussed below. No matter which language is used, the end result is software code. 

Ideally, developers will want to perform load and performance testing after the deployment of the code and before going into production. Performance testing is one aspect of IoT testing in general, as you can see in the following chart:

Although I’ve been using the terms Load and Performance testing almost interchangeably, there is a significant difference between the two.  The difference between Load and Performance testing is described well in this article by Machnation.com, and quoted here:

“Load testing is conducted at the extremes of measurement, while performance testing is conducted at a normal operating range to simulate a real-world IoT use case. The goal of load testing is to determine the ultimate scalability of an IoT solution. During load testing, the IoT solution is pushed to and beyond its capacity in order to understand its ability to operate at maximum scale. The goal of performance testing is to simulate a real-world IoT solution at real-world operating levels. This type of testing ensures that an IoT solution will deliver a customer’s key performance indicators (KPIs) while meeting service-level agreements (SLA).”

—  Steve Hilton author of IoT load testing versus IoT performance testing – 3 differences 

Because this article talks about the MQTT protocol, Brokers, and Publishing/Subscribing to topics, consider reviewing these concepts covered in previous articles:

• A Primer to IoT testing using the MQTT Protocol
• IoT testing using the MQTT Protocol and BlazeMeter
• IoT testing using Locust, Paho, and BlazeMeter

Challenges in IoT Device Testing

Hopefully now you are convinced that testing of IoT devices is crucial! The major challenge faced by developers is: 

1. How to integrate testing in the CI/CD pipeline.
2. How to stress test a system based on hundreds, thousands, or even millions of devices.
3. What tools to use for testing and getting metrics.
4. How to test different network connectivity and speed.
5. How to test m2m devices that are interdependent.

This article will offer a solution to some of these issues. Primarily, we will take a look at Node-RED and how unit tests can be used to simulate the behavior of the device. Then we will propose a methodology for running these tests using BlazeMeter for load/performance testing. This will become clearer shortly.

The next section will explain the fundamentals of Node-RED.

What is Node-RED?

Node-RED was developed by IBM as a visual tool for programming a device. It is based on Node.js, which is where the first part of the name comes from.

As far as what RED stands for: your guess is as good as anybody else's. Funny to say, but it’s true. Some people say it stands for Rapid Event Development and some say the name is a play on the term "Code-RED". In any case, you don’t need to be a programmer to code Node-RED. It comes with a GUI interface in which you drag-and-drop components into the screen and wire them up.

In the image below, you can see how the Node-RED IDE looks like. On the left-hand side you’ll find the components palette (this is only a partial list in the illustration). To create a flow, drag the component into the drawing board and connect them with "wires". On the right-hand side of the IDE, you’ll see a section which gives information on each component.

Let us say, for example, that you are a device developer and you need to program a DHT22 sensor on a Raspberry Pi device. If the sensor is not defined as a component, then just go to this site and install the sensor node definition using "npm install". There are definitions available for most of the electronic components, which is one reason Node-RED is a popular tool.

Following is an example of a very simple device programmed using Node-RED.

In this example of a simple flow, the device injects a temperature reading of 22 to a MQTT publisher. Below it is a MQTT subscriber that listens to the topic and displays the payload in the debugger. Here we are using the MQTT component that has an in and out version. Both are configured with the name of the broker (in this case broker.mqttdashboard.com:1883) and the topic name.

Flows are saved in JSON format and the definitions can be exported and imported. You can find more information on this flow here. 

This is the JSON source for 'Flow 2' described above:

Now that we have a simple flow, which is basically a piece of code, we need to test it. Although this example is a very simplistic one, you can imagine that a device taking measurements from a sensor at short intervals and sending that data to a server could generate a lot of traffic with the server. Multiply that device by many folds and that server may be overloaded or have poor performance. This needs to be checked.

The next section explains how to write a unit test for Node-RED flows. The unit test will be the basis of simulating the functioning of a device and running load/performance tests for it.

How to Test Node-RED Flows and IoT Devices

Unit testing of Node-RED flows is done using JavaScript and Mocha: JavaScript serves as the testing tool and Mocha is an open source framework that runs the JavaScript tests in the Node.js environment. This testing methodology is also proposed in an ACM article called: Towards an approach for developing and testing Node-RED IoT systems. 

To better understand how both these tools work together, we’ll create a test based on the simple example above. I’ll give an overview on how to design a Node-RED unit test, and you can also get more details here.

1. Create a test folder in your local file system. For this demo we’ll call it examples_flow2.
2. Initialize the project: In the command line of the examples_flow2 folder, type:

• npm init

You can accept the default settings when prompted.

3. Install the necessary modules:

• npm install node-red-node-test-helper node-red --save-dev

4. Update the “scripts” section of the package.json configuration file

"directories":{"test":"test"},"scripts":{"test":"mocha \"test/**/*_spec.js\" --timeout 10000"},"node-red":{"nodes":{"flow-2":"flow-2.js"}},

Specify the timeout parameter, otherwise your test will fail (the default value is 2000 ms).

Note: Here I named the node in the nodes section "flow-2", but this is just specific to this example.

5. Export the Flow from the Node-RED IDE to a JSON file in the project root directory. 
6. Create the test folder called 'test'.

Under the test folder, create a new JavaScript file called flow-2_spec.js with the following test program:

Explanation of the Test Code

There is a lack of information regarding how to do testing of Node-RED flows. For programmers/testers just starting out with writing Unit Tests for Node-RED, the following explanation will be extremely valuable.

The first part of the code is to reference the three types of nodes being used in the test. The test is made up of a Function Node, MQTT Node, and an Inject Node. Therefore you must reference/import the definitions of these node types in the Require function. I installed Node-RED locally in my project to ready it for running in the cloud environment.

Next, we define the Helper module that we installed alongside Node-RED. This helper takes care of many of the behind the scenes tasks like loading the server and then disconnecting. You must install the node-red-node-test-helper module as instructed above.

consthelper=require("node-red-node-test-helper");

Read the JSON flow created in step five above

constfs=require("fs");varnode_red_flows_string=fs.readFileSync("myflowsflow2.json");

Next, the "describe" section is used for defining the test. Here it is important to note that we are adding the following:

describe("flow-2 Node",function(){this.timeout(10000);

The default timeout for mocha tests is 2000 ms, so you need to change this setting in order for the test to work. One way to do this is by setting it in the package.json file, as mentioned above. However, if you are running Mocha from Taurus (see below), the setting must be done in code.

Skipping down to the following code, which takes the JSON flow (Node-RED maintains the flows as JSON files) and converts it to an object Array. So, we need to use this Regular Expression to convert the double-quoted key to a constant.

node_red_flows_string=node_red_flows_string.toString().replace(/"(\w+)"\s*:/g, '$1:');
var node_red_flows_Array = eval(node_red_flows_string);

Next, it is necessary to just get rid of the first node in the flow of type "tab", which is not necessary.

for (let i = 0; i < node_red_flows_Array.length; i++) {
    if (node_red_flows_Array[i].type === "tab" ) {
      node_red_flows_Array.splice(i, 1);
    }
  }

A final fixing up of the flow task is to convert the ‘debug node’ to a ‘helper node’. That’s because debug nodes aren’t recognized by the helper module.

  node_red_flows_Array[
    node_red_flows_Array.findIndex(function(node) {
      return node.id === "8640b8ff.f82ff8";
    })
  ].type = "helper";

The next part involves actually performing the test. For this, we need to tell the Helper what kind of node types to expect. So we set up an array of all the Node objects we defined in the beginning. "it" specifies a unit test and what is being tested. You could change the name of the test to whatever is logical. The Helper function loads the flow being passed as a parameter.

  var nodesTypes = [ injectNode, functionNode, mqttNode];
  it('should be loaded', function(done){
    helper.load(nodesTypes,node_red_flows_Array, settings, function(){

You reference the nodes in the flow using the ID value (see the flow’s JSON file).

    var start1 = helper.getNode("29d760365102e9ce");
    var mqout = helper.getNode("2c6873d2.992abc");
    var debug3 = helper.getNode("8640b8ff.f82ff8");

This next step isn’t necessary. I just added it to make sure the MQTT Out node (a publisher) is getting the value I’m pushing it.

    mqout.on("input", function(msg) {
      console.log("mqout.payload " + msg.payload);
    });

This call-back function will be initiated when a value is returned from the broker to the MQTT In node (a subscriber). I’m expecting the value of the payload to be what I injected in the first place. This is the first Unit Test and "done" means the test is finished.

    debug3.on("input", function(msg) {
      console.log("Msg.payload " + msg.payload);
      msg.payload.should.equal("22");
      done();
    });

This is a really important point when you are doing unit testing with the MQTT protocol: When you load the flow the MQTT broker is asynchronously connected. It takes a split second, but in that time, if you don’t give it a little wait, the broker may not be ready to receive a value and the test will fail. I gave it a wait of 2 seconds to stall the test. A better approach would be to verify if the broker is loaded and then send the receive event, instead of waiting an arbitrary number of seconds. 

Finally, another important thing to mention is that instead of calling inject1.receive to begin the flow, I am by-passing the inject node and passing a value to an intermediary function node. This is a work-around for some of the limitations of the "inject" node.

    setTimeout(() => {  console.log("sending payload");
                        start1.receive({payload: "22"});
                     }, 2000);

Running Your IoT Device Testing in Taurus

Taurus is the name of an open source configuration tool used to run tests from different tools. The idea behind Taurus is that you can define your test in a YAML structured file, by specifying all the parameters of the run. Taurus can execute any test written in JMeter, Gatling, Locust.io,PyTests, and many more, including Mocha. With Taurus you can run simulations of high loads, stress tests, and load tests. The tool measures many types of metrics to assess the performance of a system. 

So what we’ll do is create a YAML file that will tell BlazeMeter to run our unit test using Mocha. We’ll also specify that the test should run for 100 iterations and with 50 concurrent users. Then we can get statistics to see how much time it took to publish and subscribe to a topic. This is a small test, so we can run it locally.

This is the definition of the test in a YAML file:

---
execution:
- executor: mocha
  iterations: 100
  concurrency: 50
  ramp-up: 20s
  hold-for: 2m
  scenario: mytest

scenarios:
  mytest:
    script: test/flow-2_spec.js

You can monitor how the test is running using the Taurus GUI interface.

And here are the aggregate results printed to the console.

Running the Test in the Cloud Environment with BlazeMeter

Previously, we ran the test locally which is the recommended methodology: first make sure your test runs locally before running it on the cloud. But running tests locally is not going to work for simulating a large amount of devices. You're going to hit 100% CPU before you know it. 

BlazeMeter has the capacity to run big tests.  The BlazeMeter cloud environment is actually both a Platform as a service (PAAS) as well as Software as a service (SAAS) because BlazeMeter runs Taurus for you on the cloud using servers provided by Google, Azure, and Amazon. The general rule is if you can run a test locally then it will also run on BlazeMeter. 

While this is true, you still need to set up the test properly, and for running Mocha there is one caveat: installing some modules using NPM is problematic in the cloud environment. 

Unfortunately, there is a problem with installing using NPM on BlazeMeter because of security reasons. No worry, there is an easy work-around for this. What you need to do is create a ZIP file of the local node_modules folder. So here are the steps for creating a Performance Test in BlazeMeter:

1. First initiate a performance test by clicking on “create test”
2. Upload you YAML file, package.json and the JSON flow definition
3. Upload the zip file of the node_modules.
4. The next step is to create a Shared Folder.

To do that, click on the Shared Folder image here:

then create a shared folder called 'test'.

5. Upload the JavaScript test file and the mocha.opts file to this folder.
6. Now you're all set to run the test! Click 'Run Test' and after the test starts running you will see the test results in a screen such as this one:

To check the outcome of your Mocha tests, go to the Logs tab and download the artifacts.zip file. Then extract the mocha.out file which will have the results of your testing assertions, like this:

Bottom Line

In this article we took a look at Node-RED and discussed how BlazeMeter could be leveraged to run Load and Performance tests for IoT device testing based on "flows".

One of the reasons for writing this article is comments made by people on IoT forums on the lack of a formulated methodology for IoT device testing. Hopefully, this article will fill that gap and offer good testing techniques. Another reason is to show-case how BlazeMeter could be integrated into the testing cycle, no matter what testing technique is chosen.

BlazeMeter supports many of the scripting languages and provides a platform for performing heavy Load and Performance testing, whether as part of IoT device testing or on its own. Be it standard HTTP protocols or MQTT, as is used by IoT, BlazeMeter can handle these testing needs. To try out BlazeMeter for yourself, sign up for free.

START TESTING NOW

Related Resources

• How to Test the MQTT Messaging Broker