What is Automated Regression Testing and How to Do It

Regression means going backward. Regression testing determines whether the state of a software system under test has gotten worse or 'regressed' to a worse state than it was before. Regression is ordinary in software development. Developers break things that already work all the time. A new feature has been introduced, and it looks great, but some existing functionality has been broken in the process. Regression doesn't happen because developers are bad at their jobs. Developers have to deal with highly complex systems where the interactions between all components within are complicated, if not impossible, to keep track of or even identify without deep analysis.

Testing for regression is critical every time there is a new release. We know regression is common, so testing is vital. It's better to know about an issue before release rather than afterward when you have the pressure of resolving it urgently because you discover it or a customer discovers it in production. Having a list of test cases recorded and ready to run each release is essential. Ensuring that the existing functionality is intact requires comprehensive regression testing. Manually testing all functionality can take days. That's where automated regression testing becomes useful. Even so, only some tests can or should be automated without considerable investment, so it's essential to prioritize test cases and set severity levels. Signing in to the application must be a top priority, whereas the dark and light view toggle not working is not as big of a deal.

Difference between retesting and regression testing

Retesting is:

• A new feature may contain bugs when it is developed and tested. A fix is made, and the new build is tested again.
• Load testing a system, viewing the results, and then stepping up the number of requests to test again.
• Make repeated changes to an asset, such as a referenced image, and check that it looks right in the user interface.

Regression testing:

• Testing that existing, previously released functionality still works correctly.

Regression testing can often be overlooked because the thinking goes, well, that's old; it's already working; it's been tested before. Software systems are complex, and a change somewhere else can be linked to another feature in a complicated way for anyone to understand without testing. Another thing to keep in mind is that doing regression testing for a release has to wait until the new feature being introduced has completed testing. Regression testing any earlier is testing the system when in the wrong state. While changes or fixes are being made, regression (breakage of existing functionality) can always occur. Regression testing can need to be repeated if a bug fix is discovered during regression testing and a new build with a fix is made.

What is automated regression testing?

Automated regression testing is simply taking and automating regression test cases that have been written down and are currently being run manually. Some or all of the test steps can be automated. Automation also includes setting up infrastructure to automatically run regression tests at key software development life cycle stages. These include pushing a change to a repository and running continuous integration with quality gating configured.

Why automated regression testing matters?

If an issue is discovered with the software under test during regression testing and a fix is required, another round of regression testing must occur after the new build is ready. If a follow-up issue is found, another round of regression testing may follow again. Regression tests are run at least once on every release. This means that the same tests need to be run frequently, and that makes automation of regression testing important. The significant time savings result in a high return on investment.

7 benefits of automated regression testing

1. Improved test coverage and accuracy. With automation, it's possible to get high test coverage in short times and with repeated runs.
2. Increases the ROI. Due to the high frequency of having to run regression tests, manual testing is expensive, and automated regression tests provide a high return on investment.
3. Early detection of bugs. Since automation takes care of time-consuming regression testing, new features, and exploratory testing can be the focus, and automation helps detect regression faster.
4. Cost-effectiveness & resource optimization. Automated regression tests make releasing high-quality and robust releases more efficient and allow engineers and testers to focus on more original work.
5. No Human Intervention. Automated regression testing is more consistent than manual regression testing and assures that the quality is maintained at the same level on each release.
6. 24*7 running tests. Comprehensive regression testing can still take a long time, even if automated. Since testing can occur at all hours of the day, the quantity of testing that can be done with automated regression testing outperforms manual testing.
7. Reduces turnaround time. By taking advantage of parallelization, it's possible to greatly reduce test time without adding significantly more cost.

What are automated regression testing tools?

Automation regression tools are any software that helps to write and run automation regression testing. Writing automated regression tests usually requires using a general test framework. Popular general test frameworks include:

• JUnit, TestNG for Java, Maven
• Robot, pytest for Python
• Mocha, Jasmine, Jest for JavaScript
• NUnit, xUnit for C#
• RSpec for Ruby
• XCTest for Swift

For web front-end, more specialized tools exist, including:

• Playwright
• WebdriverIO
• CodeceptJS
• Nightwatch
• Protractor
• Cypress
• TestCafe

You could also build a custom framework on top of Selenium, but the above tools are very powerful.

For native mobile apps:

• XCIUTest for iOS
• Espresso for Android

You could also build a custom framework on top of Appium, but there are an increasing number of powerful mobile automation tools.

For API endpoints, specifically consider:

• Postman
• Rest Assured

Or, as with web and mobile tests, consider writing a custom test framework. For example, in Node.js, use Axios to make requests to your API endpoints and validate responses.

How to do automated regression testing?

1. Have test cases written down and run through them manually

With any manual or automated testing, the most important thing is writing out test cases. A popular method of structuring test cases is in groups of test suites for each area of the system under test. Each test case can be as simple as a step-by-step explanation of the actions to be taken and the expected outcome. A popular method of writing test cases uses the BDD (Behavior Driven Development) template with a three-part structure:

• GIVEN
• WHEN
• THEN

With this approach under GIVEN, the test author provides context about the test case, and under WHEN, outlines the actions to take. Under THEN, detail the outcome expected. For example:

• GIVEN: As a new user, I navigate to the Sign Up view.
• WHEN: I enter my email address and input a password.
• THEN: A new account is successfully created.

Writing out well-formed comprehensive test cases is the first step to regression testing. Consider using the Tesults manual testing features, such as Lists and Runs, to document your test cases. After test cases are documented, run them manually a few times to determine which test cases have to be run most frequently, which ones take the longest to run, and which ones make the most sense to automate with respect to return on investment of time. Regression tests must be run often, so automating tests while initially taking development time will save vast amounts of time in the long run. Automation will help speed up releases and ensure that your team releases a robust, high-quality build every time.

2. Automate the test cases that will offer the best return on investment

Now that you have a comprehensive list of test cases to run on each release to check for regression and have run them manually several times, it's time to start automating test cases. Start by automating the test cases you have identified that take most of your time to run or parts of them.

Automating test cases used for regression is the same as automating any test cases. A wide range of test frameworks are available for every programming language to create unit, integration, system, or end-to-end test cases. Frameworks exist for specific use cases, such as front-end; there are web browser and mobile test frameworks to choose from.

Here is a sample of popular test frameworks Tesults lists:

• JUnit
• TestNG
• pytest
• Robot Framework
• NUnit
• xUnit
• MS Test
• RSpec
• Mocha
• Jasmine
• Jest
• Playwright
• WebdriverIO
• CodeceptJS
• Nightwatch
• Waffle
• Postman
• Protractor
• Cypress
• TestCafe
• XCTest
• Espresso

In some situations, you may need to write a custom test framework yourself. Popular languages Tesults lists support for include:

• Java
• C#
• Python
• Ruby
• JavaScript / Node.js
• Kotlin
• Swift
• Go
• PHP

3. Set up automated runs of automated regression tests

When testing locally on a developer machine running tests before committing changes is good practice. When opening a pull request for review the tests should be set up to run automatically via the continuous integration system such as Gitlab CI, Github Actions, Circle CI, Teamcity, Jenkins and others.

4. Set up effective test reporting

Reporting test results effectively matters for visibility, regression analysis, diagnosis, actioning fixes, and gaining assurance on meeting quality targets. It can mean the difference between getting a high return on investment in testing and a low one.

It may seem obvious that regression can be identified if all test cases are expected to pass, so if any test case fails in any test run, there is regression. This often works well. Usually, though, a more detailed analysis is necessary, and a detailed storage of test results history is required to analyze regression over time and identify patterns or subtle issues.

5. Follow up and address failures, issues and identify gaps in testing

Once steps 1-4 have been completed, it's time to finally analyze failures from reporting, identify regression, follow up on them, and address them. Once a new build with fixes is ready, repeat the cycle, view the results again, and continue to assign failures as needed for fixes. It is also worthwhile looking for potential gaps in testing based on the test suites and cases in the report.

Test scenarios that are a good fit for automated regression testing

• Frequently repeated and reusable test cases, such as highest severity test cases on every release
• Critical flows that can render the whole system unusable, such as authentication and authorization
• Test cases where the balance between investment in automation and return on time savings makes doing it a great return on investment - the frequency of repeated test case runs must be considered.
• Scenarios where human verification or supervision is not required. For example, where usability and UX do not need to be verified, purely API integration tests are an example.

Automated regression testing best practices

• Before automation, ensure manual test cases have been recorded (written out) with clear context, actions, and expected outcomes and that the coverage from a functional perspective is broad and comprehensive. Categorize tests by severity.
• Automate the tests that, if failed, would cause the highest severity issue (such as being unable to sign in or use core features) and that lend themselves to automation, i.e., require little to no human verification.
• Once automated tests exist, run them locally and as part of your CI system for every release.
• Automate regression analysis with a high-quality reporting system capable of storing test results data history. The reporting system should be able to identify regression over time, present error details, notify appropriate team members of errors, consolidate automated results data across your systems, and make manual test cases easy to record and view.

If you do all of the above, you will be releasing high-quality software every release and have assurance about the robustness of the product you are putting in the hands of your customers.

Enhance your regression testing with Tesults

Consider Tesults for robust regression analysis and reporting. It's free to try forever and simple to set up and use with most languages and test frameworks.

Conclusion

Regression means going backward and breaking features and functionality that worked before. There's no need for that to happen. Follow the steps outlined above to ensure you're releasing high-quality software every time.

FAQs

1. Is regression testing manual or automated? Testing for regression can be done manually or automatically. In the beginning, it makes sense to record and run test cases manually. Once you want to start releasing frequently, it makes sense to automate the highest severity test cases to save time and achieve a faster release cadence.
2. Can automated regression testing completely replace manual testing? Some test cases requiring human usability or UX testing are difficult to replace entirely with automation; however, the vast majority of test cases can usually be automated.
3. How do I choose the right automated regression testing tool for my project? The right test framework to select depends on the language your software is written in and whether you need to test back-end or front-end (web/mobile/other devices) to determine which framework or frameworks to use. In some cases, a custom test framework may be required.
4. What areas of regression testing can be automated? Test cases that lend themselves to automation for regression identification purposes require little to no human verification and are run at high frequency, i.e., on every release. Thus, the return on investment in automation is large due to the huge amount of time saved.
5. How can I undo automated regression testing errors? Your test reporting tool should identify the reasons for the failures of test cases and point developers on how to get started on resolving errors.