The Internet of Things (IoT) market is booming and with it comes the need for adequate testing. Continuous Testing, also known as Shift-Left, is a methodology of supplementing the CI/CD pipeline with early testing. The capability of identifying weaknesses in a system while still in the development process will result in reduced risks. This article introduces a methodology for doing Continuous Testing (CT) with Jenkins with a focus on IoT.

I’ll be demonstrating the testing cycle using Locust with a pyTest. However, BlazeMeter supports many types of scripting languages such as JMeter, NUnit, Mocha, and more. For a complete list see the Taurus documentation here.

Can You Build IoT Apps with Jenkins?

Yes. Jenkins is a popular CI/CD open-source tool for automating the building, testing, and deploying of software development projects—including IoT applications. BlazeMeter provides a plugin for Jenkins to integrate testing with the CI/CD process. With the plugin, the testing phase is augmented to the CI/CD pipeline thereby facilitating Continuous Testing.

Continuous Testing for Jenkins IoT

In a research paper published by the IEEE on “Continuous Testing in the Development of IoT Applications”, the authors stated:

“IoT applications need to adapt frequently and rapidly to new requests. Increasing competition and rapidly changing market needs require from nowadays companies' flexibility and fast time to market of their products. To achieve these goals more and more organizations are relying on new IT technologies and software development processes.”

This statement emphasizes the need for using Software Development technologies, specifically Continuous Testing,  for the fast-moving IoT market. 

BlazeMeter Jenkins Plugin

The BlazeMeter Jenkins plugin can be easily downloaded, installed, and configured into Jenkins. The plugin allows you to automatically run a BlazeMeter test based on the most current testing artifacts. Jenkins is synced with GitHub and knows to update the BlazeMeter test with the latest up-to-date scripts. That way you can be sure that your test will run using the correct script. I’ll give more details on this process below, but for more details on how to use the Jenkins plugin, see this article.

Overview of the Methodology

The following illustration depicts the methodology of doing Continuous Testing in general, and for IoT in particular. The process starts with a change in the testing script. Say that a tester has a set of pyTests in their local environment and made changes to them. They would then commit the changes to GitHub using Git. Since Jenkins is integrated with GitHub, it would be aware of any changes in the script. Jenkins is used to run a build manually or scheduled for periodic builds. The installed BlazeMeter plugin is activated during the build process. The plugin updates the BlazeMeter test with the updated script and then runs the test. Results of the test are transferred back to Jenkins and reported.

How to Set Up the Jenkins BlazeMeter Plugin

For Jenkins to be able to run BlazeMeter tests, it must have accurate credentials. To add BlazeMeter credentials (the BlazeMeter API Keys), go to “Manage Jenkins” in the Jenkins menu.

Next, click on “Manage Credentials” followed by choosing the Global Store (see Stores scoped to Jenkins):

Next, click on Global credentials (unrestricted):

Finally, click on “Add Credentials”:

And in the credential screen give a description (such as “BlazeMeter Credentials”), API Key Id, and the secret ID.

BlazeMeter Plugin Configuration

Once the credentials are set up, you can configure the BlazeMeter plugin. In the project screen click on “Configure”. In the following illustration, I set up a new build project called “Project IoT CT Example 1”. 

In the “Build” section, find “BlazeMeter”. You may get a message that there is a new version of the plugin available (see below). Click on the link to update the plugin. Then, don’t forget to go to the console where you ran the Jenkins startup command, and restart Jenkins. 

There are a couple of settings that you need to enter. The BlazeMeter credentials that were setup will appear in the dropdown box according to the description. Choose the credentials to use. (Remember that BlazeMeter credentials are reset every month, so you’ll need to update periodically). Next, choose the Workspace and Test from the dropdown boxes (this assumes you already set up a test in the BlazeMeter environment). This is the test that will be updated with the recent code and run with the Jenkins build.

There are “advanced” settings in this section accessed by the “Advanced” button. The Main Test file is the YAML used to run the pyTest. The additional test files are the pyTests and other support files. These are the files that will automatically be updated in BlazeMeter when the build is executed.

Keep in mind that in the “Branches to build” section it will say */master. This needs to be changed to “main”.

Now the BlazeMeter plugin is all set up to implement the continuous testing mechanism. Next, we’ll show how to get the process going.

Testing IoT devices with Locust

As mentioned earlier, BlazeMeter supports running tests of different types. For this example, I chose to use Locust. Locust is a great open-source platform for running pyTests concurrently. The goal of Locust is to create a “swarm”, which basically means many concurrent users. Locust is one of the supported “executors” of Taurus, which is what BlazeMeter runs under the hood.

A Taurus configuration file is in YAML format. Following is an example of how to run a scenario based on a pyTest called testEntries.py. Notice that the parameters used to declare the test, such as iterations and concurrency, are also common to the BlazeMeter UI used to configure a test. However, the “hold-for” parameter is similar to the “duration” configuration in BlazeMeter and is required at this time to set the termination of the Locust test.

In this example, the script being run is a pyTest called testEntries.py which is shown below. However, for all intents and purposes, you could have used any other supported language. In the IoT world, the most popular languages for testing applications are Java, Python, C, and Javascript which are all supported in BlazeMeter via Taurus. The script, testEntries.py, is a pyTest that simply performs a GET to the BlazeMeter demo site for demonstration purposes.

Congratulations! Our test is now set up and we can incorporate it in the Continuous Testing pipeline.

Committing Changes

In the following screenshot, you can see the steps for using Git to update the repository. With Git, the developer can make changes to code locally and upload the changes to a repository when done. The git commands used here are basic:

• > git add .
• > git commit -m “label”
• > git push

Running the Build in Jenkins

You can run a Build in Jenkins manually or with a scheduled job. For Continuous Testing you could create a scheduled job at preset intervals, trigger a build using a remote script, or use any other of the options Jenkins offers . To set the trigger, go to the configuration page for your test and scroll down to “Build Triggers”. If you choose to run the build at a specific schedule, click on “Build periodically”. Jenkins uses Cron expressions, so H 9 * * * would specify that the build should run every day at 9:00 A.M. 

To see exactly what is happening behind the scenes in Jenkins, click on the link of the build and go to the “Console Output”.

Here you can see a log of all the steps the build is executing. This log is important in case there are errors in the build process. One of the items being reported in the log is the URL of the test’s summary report which can be shared publicly.

Viewing the Test Results in BlazeMeter

After running the test, you can view the results using BlazeMeter’s reporting tool. This tool displays the results of the test in a graphical manner. There are several views such as the Summary page. This summarizes the outcome of the test: how many VUs were run, the throughput, errors, response time, bandwidth and more. The other tabs give a drill down of the results. The Engine Health view is particularly important for monitoring the server CPU, memory, network I/O, and number of connections. This essentially is the goal of Continuous Testing: to monitor the performance on the back-end servers as changes are made to the system. Catching performance problems early on in the development cycle can save a lot of rework time in the future. See here for more detailed information on BlazeMeter’s reporting capabilities.

Conclusion

This article introduces an important concept to IoT testing: Continuous Testing using Jenkins and BlazeMeter. Although CT has been around for a quite some time in Software Development, it is not that common in the IoT world. We discussed how the CT methodology could be implemented using Jenkins, Git, GitHub, and BlazeMeter. This article is one of several hands-on and practical demonstrations on how to adapt standard methodologies to the IoT world. Following are some additional resources on this topic. 

START TESTING NOW

Related Resources

• How to Test the MQTT Messaging Broker
• JMeter IoT Testing Using the MQTT Protocol: A Primer 
• IoT Testing Using the MQTT Protocol, JMeter, and BlazeMeter
• IoT Testing Using Locust, Paho, and BlazeMeter
• Node-RED Flows and IoT Device Testing with BlazeMeter 
•  IoT Predictions for 2022
• 4 Benefits of Continuous Testing and How to Get Started
• How to Achieve Continuous API Testing With BlazeMeter