Performance Testing With Open Source Tools: Busting The Myths

In times of crisis and when in search of possible ways to reduce costs, the idea of adopting or migrating to open source performance testing tools suddenly becomes more appealing. In this blog post, I want to share some things you should take into account in the process of selecting or migrating to an open-source tool for performance testing.

There are some myths I’ve come across over the years which I’d like to dispel based on my 15 years of experience working with these tools.

Are Performance Testing Tools Open Source?

There are many open source performance testings tools, including:

• JMeter
• Gatling
• Selenium
• Taurus

Myths of Using Open Source Performance Testing Tools

Myth 1: Open Source Performance Testing Tools Aren’t Mature Enough

If you’re going to invest time and effort in doing a performance test, you’ll most likely care about the results you get from it. You’ll rely on these results, so you will have to rely on the tool. Hence, it’s imperative that the tool is one that you can trust. It’s been said that open source tools are not very trustworthy. My colleagues and I have spent a great amount of time analyzing and contrasting the results we obtain from different tools. Nowadays, after doing some digging, the two we trust the most for load testing are JMeter and Gatling. We’ve even run the same test with both in order to compare the results in different conditions. In those tests, we also compared different metrics to see if they correlate properly; for instance, the number of bytes received by the server and the ones shown by the tool’s report or the number of connections, etc.

Another way to think about this is that open source solutions see more code review than any commercial software does. Great software engineers review the open-source code in order to start contributing, understand how it works, and make their own extensions. Whilst doing so, they also think critically about it and contribute to the project with different improvements in quality and security. They do that because they know how to do it and because they care (as they will be using the software as well). The process involved and the products and the results I’ve seen make me consider the open-source performance testing tools I’ve been using to be very mature, even compared to legacy testing solutions.

Myth 2: Open Source Tools Have Limitations and a Lack of Functionality

When I just got started with software performance testing, 15 years ago, I remember that my internal rule was: if the protocol is open, we can use JMeter, if it’s a proprietary protocol, then we should use Rational Performance Tester or HP Loadrunner. Nowadays, JMeter has lots of plugins supporting different protocols, not only those that are open source, but also some commercial ones.

Apart from that, it’s great how many features have been added, like pretty HTML reports, better recording (with more automations) and traceability that makes debugging the scripts much easier.

In case you are missing a key feature, JMeter allows you to include Groovy code in your scripts, which is very useful when you cannot solve something with its native functionality. With this possibility, you can actually do whatever you need in your scripts. This also applies to Gatling where scripts are part of a Scala solution, so you can include your own libraries for any particular thing you need to do within your scripts.

Myth 3: It’s Harder to Integrate Open Source Solutions with Other Tools

Again, it depends on the tools you’re considering, but typically there are other tools that offer integrations or extensions for the most popular open source testing tools. For example, in test automation, you can see that Selenium is the standard and can be integrated with anything. It also has bindings to most popular programming languages and connections to any CI/CD engine.

In the case of performance testing tools, I dare say that JMeter is the standard. Platforms like BlazeMeter allow you to scale your performance scripts in the cloud and run from multiple locations and with many concurrent users, always based on JMeter files. Also, thanks to Taurus, it’s very easy to integrate JMeter scripts with any CI/CD platform like Jenkins, and get testing results as soon as you push your code to your repository.

It’s also worth noting that open source tools can be tailored to your own specific needs. This could require some effort, especially if you are new to the solution. In my case, my team has been developing extensions for JMeter for a long time. There was a learning curve, but now it’s easier to, for example, develop support for a new protocol. For this, you should check the available resources along with the code repository of the tool. See how extensive the documentation is, verifying that there is a section clarifying how to customize and extend it.

Myth 4: Open Source Tools are Ugly and Hard to Use

I’ve heard that open source tools are always hard to use; made by programmers for programmers. In this case, yes, in the majority of cases, performance tests are going to be driven by programmers and performance engineers. In my experience, the usability and user experience of these tools is great. It’s important to mention that there are even multiple varieties; if you like code, you can use Gatling or Taurus, and if you prefer a graphic user interface, you can use JMeter, where the scripts are represented with a tree with the different elements.

I think it’s important to distinguish what is good usability from what is “pretty.” What’s important is how easy it is to use, how user-friendly, and how efficient you are using it. Imagine that you receive an error and you have no clue how to fix it. In that case, you will need a lot of time to achieve your goal of preparing a performance test. This would demonstrate bad usability. This is not the case with these open source tools like JMeter and Gatling because there are different elements that help you to have complete visibility into what you’re sending to the server, what you’re receiving and it’s easy to try different alternatives.

Myth 5: Open Source Testing Tools Are Not Well Supported

It’s important to choose your open source performance testing tool wisely, paying attention to the level of adoption and community  support behind the tool. Do some homework before deciding which tool to use, review its Github repository, documentation, courses and tutorials available, companies offering professional services with them, etc. Also, check Linkedin to see if there are people with experience in the tool, if you need to hire. In this way, you can plan for the support you will need.

JMeter, for example, has the biggest community as far as performance testing tools. Also, many of the open source tools have a company behind them providing support like Gatling, which has an open source tool as the base of its commercial version.

It takes just one search on the web to realize how much support you could expect to get from an open source tool. For JMeter, you’ll find tons of forums, groups and channels to ask questions if you need help or support. You can also check how much time it takes for a question to be answered.

Myth 6: Migrating to An Open Source Testing Tool Takes A Huge Effort

If you are already using a commercial testing tool, it could seem very expensive to migrate to open source alternatives. You would need to make a great investment in order to move from one solution to the other without losing anything in the migration and after that process, you start saving costs (which is probably one of the initial motivations you had in the beginning).

My suggestion here is to start looking at similar experiences of other people trying to do the same migration process. In some cases, there are some tools or scripts that can help in the migration process. For example, Blazemeter offers a converter from Loadrunner to JMeter automatically, even with different protocols, not only HTTP. 

The converter allows you to drag and drop your Loadrunner scripts obtaining an initial version of your JMeter scripts. Depending on how specific the things you were doing in your script were, you may have to spend some time to finish it up after using the converter to make any necessary adjustments. In any case, the converter is a great free tool that drastically reduces the cost of migration.

There is also another way to reduce the cost which is outsourcing the gap, something that my colleagues and I help our clients do at Abstracta, a software testing outsourcing firm.

Myth 7: It's Hard to Get Professional Training For Open Source Testing Tools

From what I’ve seen in the testing world, there are endless trainings you can find online for open source tools in multiple languages. Just make a youtube search for “performance testing with” and add the name of your favorite open source performance testing tool and you will get many results. Also, you will find many other options on platforms like Udemy, Coursera, and the like which are indeed, high quality.

For JMeter, Taurus, BlazeMeter training and more, you can check out Blazemeter University where you will find lots of free, fully accredited courses for all skill levels.

Do Not Dismiss Open Source Tools for Performance Testing

After going through all these myths, I’d also like to point out that the main myth maybe is that open source is just the cheap way to go to reduce costs. The truth is that there are excellent options to choose from, and that open source in no way means less quality than commercial. Paying attention to the community behind open source tools and their adoption by other companies, you can decide which are competitive and can best solve the challenges you’re facing.

📕 Related Resource: Learn more about API Performance Testing Tools: JMeter, Taurus, and BlazeMeter

In my experience, I can suggest you take a look at JMeter, Gatling and Taurus. At Abstracta, we have been working with them for more than 10 years and recommend them time after time, project after project!

START TESTING NOW