The Importance of Highly Visible Automated Test Results

Five years of experience as a senior and lead software development engineer in test and a further five as a product developer have taught me that there is no greater driver of engagement in automated testing than ensuring highly visible automated test results are made available to the entire team.

Here's why.

Visibility leads to transparency. If the results of your automated tests are buried away in an xml file on your build server or in the console output of your Jenkins job or some other hidden away location that requires the esoteric skills of your build or test engineer to get at, then automated tests are out of sight and out of mind for most of your team.

There are a number of problems with this approach even if you currently have good test coverage.

Without highly visible results there is no reference for the overall team as to what the current coverage level is and where there may be possible gaps as product development progresses. Well defined and well named test suites and cases provide a good first indication of areas that are being neglected and where to focus efforts on authoring new test cases.

Without highly visible results there is no understanding of the current health of the project across the team. In many teams, changes may be checked in several times a minute, and certainly several times an hour or day. A continuous build and test system keeps abreast of the impact these changes are having on the entire system and keeps a finger on the pulse of the health of the product. If only a core group of developers are aware of these test results you introduce several problems.

These select few team members can become burdened with triaging or understanding the source of failures. This can often be inefficient because it is often not their changes that have introduced the failure point. With high visibility and awareness of results by everyone you introduce team wide ownership. A more appropriate team member can quickly identify the reason for the introduction of the error, especially if the test case is well defined and a failure reason is posted with a call stack or appropriate assert messages.

Another problem with only a small core of team members being aware of failures is that team leads and management are unable to get an understanding of the current project health without asking the ones in the know. A quick glance at a monitor displaying a 100% pass rate should be all that is needed, it takes half a second and requires no interruptions. Furthermore, if the results are stuck at below 100% for too long and over too many build cycles the right questions can be asked and a framework around appropriate discussions to have has already been built.

Team members besides developers and engineers benefit from being aware of the status of test results. Rather than charging in with a new feature request or insisting on a meeting about something irrelevant to the current issues at hand everyone has some level of awareness when there are real existing problems with the current robustness of the product and allows for objective sensitivity about allowing team members tasked with fixing issues to get on with it.

Transparency leads to honesty. Some teams may think they ‘do automated testing’ and have all of their bases covered. Highly visible test results can blow any illusions about this reality out of water and keep teams honest. If you only see a paltry number of test results being displayed it becomes immediately clear that the alluded to test coverage does not exist and was a myth. This point is particularly important for teams new to automated testing or teams unhappy with their current coverage. There is no need to have general uncertainty about the reality of test coverage and the current status. Highly visible results displaying only a handful of results generates an impetus to improve the situation and most importantly ensures follow through by also serving to provide the reward of actually seeing the number of test cases clearly increasing over time in parallel to product complexity.

Highly visible automated test results immediately tell you if you would be wasting time and resources even providing the latest build to a manual test QA team without any further checking required.

Perhaps best of all is the peace of mind you have due to the certainty and confidence of knowing the health of your project is robust.

Bring your test results to the forefront and have your entire team have access to them. Display them on monitors and invite everyone to contribute to increasing the total coverage and quality of the tests you are running by having team wide awareness of the issues.

Whether you use a custom test framework and harness or an industry standard one, whether you roll your own solution for making your test results visible or you use Tesults, highly visible automated test results are the key to driving engagement and higher quality standards.