The State of AI in Testing

Forward-thinking organizations are rushing to introduce AI into their operations, and rightfully so. But when it comes to using AI in testing, it is important not to use AI for AI’s sake. Instead, organizations need to make sure it provides value, makes testing more accurate, and more efficient. 

In this blog post, we dive into the exciting future of AI in testing with new capabilities to watch out for and how it will become the new paradigm.

Don’t Do the Obvious GenAI Thing

Testing companies have rushed to add AI into their testing capabilities. This mostly plays out as using GenAI as test generation copilots. But with recorders and scriptless capabilities, generating scripts is no longer as challenging as it used to be. For example, autocorrelation with a record replay already allows recording mobile tests and functional tests at the browser level, making the process straightforward and effective.

Despite the hype, GenAI today adds limited value for test generation. GenAI’s value for testing today lies in its use for creating data and virtual services and for identifying relevant code coverage. But the real value of AI in testing lies in the (not so far) future.

The Value of AI-Powered Analysis

An example of added value that testers and enterprises can expect to enjoy soon, is root cause analysis. Today, test analysis is a meticulous and dreary process. For example, if a tester runs thousands of tests and they generate a few hundreds of errors, combing through them and understanding where the error lies - if it’s in the test or the system - and what’s required to fix it, is time-consuming and dull. Then, AI can help prioritize which areas to fix.

AI can help not only with analyzing the errors and results, but with identifying the problem. For example, if the script was written incorrectly, if there was a network issue, if there was an issue in the backend or the UI, etc.

But the most exciting capability will be when AI helps with the fixes. AI will be able to suggest required changes, and even carry them out themselves. This is groundbreaking in terms of code quality, time-to-market, and the tester and developer experience.

Next Level AI-Driven Creation

In the upcoming future - which means a few months or years down the line, not decades - AI will provide even more incredible value for testers. This will be carried out by assisting with tasks humans cannot perform. A few examples include:

• Identifying issues in production. For example, queueing, database configurations, generating tests from errors, recreating the issue in pre-production.
• Advanced report analysis - Currently, reports provide information like whether the test passed or failed, the web logs, performance, executed code, etc. AI will use a wide range of data, like historical data, network traffic, responses from API calls and more, to provide insights into the real problem and to make suggestions. This will happen instantaneously.
• And above all - AI-driven test execution.

What is AI-Driven Test Execution?

AI-driven test execution is a paradigm shift in testing. AI will take away manual script creation, including maintenance and updating. Instead, AI will create the tests itself, based on tester inputs.

Here is how it will work:

Step 1 - AI-Driven Test Creation:  The test cases and requirements will be generated in natural language by the tester. The tester will also include information like application documentation, previous tests, production plans, flows derived from production, telemetry, etc.

Step 2 - AI-Driven Navigation: The model will visually go through the application to instantaneously create the test.

Step 3 - AI-Driven Validation: The tester will add validation steps and acceptance criteria to ensure test quality.

Validation will go beyond today’s existing, limited validation capabilities. For example, let us say you have a weather app. AI can validate that the temperature graph is trending upwards in the summer. Such validation is not possible today.

With AI-driven test execution, testers no longer the need to:

• See how backend changes impact automated tests
• Update regression suites
• Learn how to use testing tools like JMeter, Selenium, Cypress, etc.
• Worry about objects

Use Case Spotlight: Tests Created From Production

An interesting use case of AI-driven test execution is using information from production for test creation. This information is valuable because it covers real use cases, which are impossible to completely recreate and anticipate beforehand. 

AI will be able to use telemetry data, HAR files, APM data, and other types of information, to generate tests based on the patterns of actual use. To ensure data leaves production, AI will profile the data and use synthetic data.

The New Role of Testers With AI in Testing

What do these changes mean for testers? AI-driven testing does not mean testers will no longer be needed. Rather, testers’ roles are expected to change. They will be able to focus on strategic tasks, like creating test cases, test requirements, and acceptance criteria. AI will make their jobs faster and easier.

AI in Testing: The Future of Testing

Currently, AI for testing is still in an experimental state. But soon enough, it will bring groundbreaking advancements to testing. In the upcoming years, testers and organizations will no longer have to worry about how to create tests: functional, performance, security, or their accessibility and user experience.

Instead, AI will generate the tests, allowing the tester to focus on feeding the models with requirements, test cases and anything else AI needs to understand what to do. Once tests are generated, AI will analyze the results, provide recommendations and even make fixes itself.

Currently, these advancements are being productized. Perforce has added various AI capabilities over the years to our testing product strategy. These include:

• Using ML to detect when a test is stuck due to a pop-up
• Self-healing object identification and location with ML
• Generating reliable and accurate test data with AI

Coming soon, Perforce users will be able to enjoy:

• AI-powered root cause analysis
• AI-driven validation
• AI-driven test execution - test execution based on visual and contextual understanding of the flow and objects on the screen
• Auto-scaling - Using AI to auto-scale load engines in a resource-efficient manner
• Visual test data generation - Using GenAI to generate images and photos for testing

Bottom Line

The future of software testing is not so distant anymore. With AI in testing, new capabilities and opportunities are being discovered that is making (or will soon make) testing more impactful and efficient than ever before.

BlazeMeter is at the forefront of this innovation with implementing AI in testing through. And while many of the capabilities are still to come, you can start using our cutting-edge, AI-driven Test Data Pro to transform the way you use test data.

Start Testing Now