Cypress test reporting

There are two ways to push test results data from Cypress to Tesults. The first way is to use the Cypress Tesults Reporter, which utilizes the Cypress Module API, this is recommended. The second way is by using the Mocha Tesults Reporter.

Cypress Tesults Reporter

Recommended

Installation

npm install cypress-tesults-reporter --save

Configuration

The Cypress Tesults Reporter cannot be used like a typical mocha based reporter. You must start cypress from a 'runner' as it utilizes Cypress's Module API. Create a new file with this content:

runner.js

const cypress = require('cypress')
const tesults = require('cypress-tesults-reporter');

cypress.run({

})
.then((results) => {
  const args = {
    target: 'token',
  }
  tesults.results(results, args);
})
.catch((err) => {
 console.error(err)
})

Replace token with your Tesults target token, available from the configuration menu within Tesults.

Now you can simply run the file. For example if you have called your file 'runner.js' then run:

node runner.js

After the tests have completed, results will be pushed to Tesults. See Cypress's documentation for more details when using their Module API, you can pass in args to cypress or use the cypress.json configuration file as usual: https://docs.cypress.io/guides/guides/module-api.html#cypress-run

You can optionally include build related information to Tesults in the args with the target token, for example, if this is being run as part of a continuous integration system:

const args = {
  target: 'eyJ0eXAi...',
  build: {
    name: '1.0.0',
    result: 'pass',
    desc: 'Build description',
    reason: 'If failed, provide build failed reason',
    files: ['/Users/admin/files/build.png','/Users/admin/files/build.log']
  }
}
tesults.results(results, args);

Setup is complete.

Consolidating multiple test results submissions into one test run

If you execute multiple test runs in parallel or serially for the same build or release and results are submitted to Tesults within each run, separately, you will find that multiple test runs are generated on Tesults. This is because the default behavior on Tesults is to treat each results submission as a separate test run. This behavior can be changed from the configuration menu. Click 'Results Consolidation By Build' from the Configure Project menu to enable and disable consolidation by target. Enabling consolidation will mean that multiple test runs submitted with the same build name will be consolidated into a single test run.

Dynamically created test cases should have static names

If you dynamically create test cases, such as test cases with variable values, we recommend that the test suite and test case names themselves be static. Provide the variable data information in the test case description or other custom fields but try to keep the test suite and test name static. This way you will you benefit from a range of features Tesults has to offer including test case failure assignment and historical results analysis. You need not make your tests any less dynamic, variable values can still be reported within test case details.

Proxy servers

Does your corporate/office network run behind a proxy server? Contact us and we will supply you with a custom API Library for this case. Without this results will fail to upload to Tesults.

Have questions or need help? Contact us

Mocha Tesults Reporter

Not Recommended

The Cypress documentation explains that 'Because Cypress is built on top of Mocha, that means any reporter built for Mocha can be used with Cypress.' Unfortunately this is not quite true currently in all cases.

Tesults is waiting for these two issues to be addressed by Cypress:

  1. https://github.com/cypress-io/cypress/issues/1946
  2. https://github.com/cypress-io/cypress/issues/7139

The first issue requires that your Cypress tests are all contained with a single spec file, you can then follow the instructions below to use the mocha-tesults-reporter to report results from Cypress to Tesults. However in most cases, we understand that is not how test cases are laid out, you are likely splitting test cases across multiple files. We are paying close attention to the further developments by the Cypress team and will update documentation here once there is a better solution available.

The second issue means that Cypress will have the Mocha process exit early and results may not be uploaded. A work around is to have a second spec after the first one that runs for a while but this is a bad work around and that is why we cannot recommend this approach and suggest you utilize the 'Cypress Tesults Reporter' instead, outlined at the top of this page.

Installation

npm install mocha-tesults-reporter --save

Configuration

There are two ways to have Cypress upload results to Tesults. One involves adding properties to the cypress.json file, the other is to use the command line. Whichever method you use is up to you.

Option 1: cypress.json

Your cypress.json file must contain the properties below to have results upload to Tesults:

{
  "reporter": "mocha-tesults-reporter",
  "reporterOptions": {
      "tesults-target": "token",
      "tesults-config": "/path/config.js",
      "tesults-files": "/path-to-files/files",
      "tesults-build-name": "1.0.0",
      "tesults-build-description": "build description",
      "tesults-build-reason": "build failure reason (if failed)"",
      "tesults-build-result": "pass"
  }
}

Option 2: Command line args

cypress run --reporter mocha-tesults-reporter --reporter-options "tesults-target=token"

Back to Mocha

This completes Cypress specific configuration. Now continue viewing the Tesults Mocha documentation to complete setup.

Consolidating multiple test results submissions into one test run

If you execute multiple test runs in parallel or serially for the same build or release and results are submitted to Tesults within each run, separately, you will find that multiple test runs are generated on Tesults. This is because the default behavior on Tesults is to treat each results submission as a separate test run. This behavior can be changed from the configuration menu. Click 'Results Consolidation By Build' from the Configure Project menu to enable and disable consolidation by target. Enabling consolidation will mean that multiple test runs submitted with the same build name will be consolidated into a single test run.

Dynamically created test cases should have static names

If you dynamically create test cases, such as test cases with variable values, we recommend that the test suite and test case names themselves be static. Provide the variable data information in the test case description or other custom fields but try to keep the test suite and test name static. This way you will you benefit from a range of features Tesults has to offer including test case failure assignment and historical results analysis. You need not make your tests any less dynamic, variable values can still be reported within test case details.

Proxy servers

Does your corporate/office network run behind a proxy server? Contact us and we will supply you with a custom API Library for this case. Without this results will fail to upload to Tesults.

Have questions or need help? Contact us