Cypress test reporting

There are two ways to push test results data from Cypress to Tesults. The first way is to use the Cypress Tesults Reporter, which utilizes the Cypress Module API (https://docs.cypress.io/guides/guides/module-api.html#cypress-run), this is recommended. The second way is by using the Mocha Tesults Reporter.

Cypress Tesults Reporter

Recommended

Installation

npm install cypress-tesults-reporter --save

Configuration

The Cypress Tesults Reporter uses the Cypress Module API and so you must start Cypress from a 'runner' as described in Cypress's guide (https://docs.cypress.io/guides/guides/module-api.html#cypress-run). Create a new file with this content:

runner.js

const cypress = require('cypress')
const tesults = require('cypress-tesults-reporter');

cypress.run({
  // specs to run here
})
.then((results) => {
  const args = {
    target: 'token',
  }
  tesults.results(results, args);
})
.catch((err) => {
 console.error(err)
})

Replace token with your Tesults target token, available from the configuration menu within Tesults.

Now you can simply run the file. For example if you have called your file 'runner.js' then run:

node runner.js

After the tests have completed, results will be pushed to Tesults. That completes basic setup.

Screenshots and Videos

Any screenshots and videos taken using built-in Cypress functions (https://docs.cypress.io/guides/guides/screenshots-and-videos) will be automatically saved to Tesults.

Custom files

If you save files without using built-in Cypress functions (custom files) then these can also be uploaded to Tesults. Save custom files to a local temporary directory. At the end of the test run all custom files will automatically be saved to Tesults as long as you save files in the directory structure below and provide the top-level temporary directory full path in the args options alongside the target token

const args = {
  target: 'eyJ0eXAi...',
  files: '/Users/admin/temporary'
}

The directory structure to save custom files in for them to be automatically saved to Tesults:

  • expanded temporary folder
    • expanded Test Suite A
      • expanded Test 1
        • test.log
        • screenshot.png
      • expanded Test 2
        • test.log
        • screenshot.png
    • expanded Test Suite B
      • expanded Test 3
        • metrics.csv
        • error.log
      • expanded Test 4
        • test.log

Build data

You can optionally include build related information to Tesults in the args with the target token, for example, if this is being run as part of a continuous integration system:

const args = {
  target: 'eyJ0eXAi...',
  build_name: '1.0.0'
  build_result: 'pass'
  build_description: 'Build description'
  build_reason: 'If failed, provide build failed reason'
}

Result Interpretation

Result interpretation is supported by Cypress. Install version 1.3.0 or newer. If you use result interpretation we recommend you add these minimum mapping values:

passed
pass
failed
fail

Consolidating parallel test runs

If you execute multiple test runs in parallel or serially for the same build or release and results are submitted to Tesults within each run, separately, you will find that multiple test runs are generated on Tesults. This is because the default behavior on Tesults is to treat each results submission as a separate test run. This behavior can be changed from the configuration menu. Click 'Results Consolidation By Build' from the Configure Project menu to enable and disable consolidation by target. Enabling consolidation will mean that multiple test runs submitted with the same build name will be consolidated into a single test run.

Dynamically created test cases

If you dynamically create test cases, such as test cases with variable values, we recommend that the test suite and test case names themselves be static. Provide the variable data information in the test case description or other custom fields but try to keep the test suite and test name static. If you change your test suite or test name on every test run you will not benefit from a range of features Tesults has to offer including test case failure assignment and historical results analysis. You need not make your tests any less dynamic, variable values can still be reported within test case details.

Proxy servers

Does your corporate/office network run behind a proxy server? Contact us and we will supply you with a custom API Library for this case. Without this results will fail to upload to Tesults.

Have questions or need help? Contact us

Mocha Tesults Reporter

Not Recommended

The Cypress documentation explains that 'Because Cypress is built on top of Mocha, that means any reporter built for Mocha can be used with Cypress.' Unfortunately this is not quite true currently in all cases.

Tesults is waiting for these two issues to be addressed by Cypress:

  1. https://github.com/cypress-io/cypress/issues/1946
  2. https://github.com/cypress-io/cypress/issues/7139

The first issue requires that your Cypress tests are all contained with a single spec file, you can then follow the instructions below to use the mocha-tesults-reporter to report results from Cypress to Tesults. However in most cases, we understand that is not how test cases are laid out, you are likely splitting test cases across multiple files. We are paying close attention to the further developments by the Cypress team and will update documentation here once there is a better solution available.

The second issue means that Cypress will have the Mocha process exit early and results may not be uploaded. A work around is to have a second spec after the first one that runs for a while but this is a bad work around and that is why we cannot recommend this approach and suggest you utilize the 'Cypress Tesults Reporter' instead, outlined at the top of this page.

Installation

npm install mocha-tesults-reporter --save

Configuration

There are two ways to have Cypress upload results to Tesults. One involves adding properties to the cypress.json file, the other is to use the command line. Whichever method you use is up to you.

Option 1: cypress.json

Your cypress.json file must contain the properties below to have results upload to Tesults:

{
  "reporter": "mocha-tesults-reporter",
  "reporterOptions": {
      "tesults-target": "token",
      "tesults-config": "/path/config.js",
      "tesults-files": "/path-to-files/files",
      "tesults-build-name": "1.0.0",
      "tesults-build-description": "build description",
      "tesults-build-reason": "build failure reason (if failed)"",
      "tesults-build-result": "pass"
  }
}

Option 2: Command line args

cypress run --reporter mocha-tesults-reporter --reporter-options "tesults-target=token"

Back to Mocha

This completes Cypress specific configuration. Now continue viewing the Tesults Mocha documentation to complete setup.

Result Interpretation

Result interpretation is supported by Cypress. Install version 1.3.0 or newer. If you use result interpretation we recommend you add these minimum mapping values:

passed
pass
failed
fail

Consolidating parallel test runs

If you execute multiple test runs in parallel or serially for the same build or release and results are submitted to Tesults within each run, separately, you will find that multiple test runs are generated on Tesults. This is because the default behavior on Tesults is to treat each results submission as a separate test run. This behavior can be changed from the configuration menu. Click 'Results Consolidation By Build' from the Configure Project menu to enable and disable consolidation by target. Enabling consolidation will mean that multiple test runs submitted with the same build name will be consolidated into a single test run.

Dynamically created test cases

If you dynamically create test cases, such as test cases with variable values, we recommend that the test suite and test case names themselves be static. Provide the variable data information in the test case description or other custom fields but try to keep the test suite and test name static. If you change your test suite or test name on every test run you will not benefit from a range of features Tesults has to offer including test case failure assignment and historical results analysis. You need not make your tests any less dynamic, variable values can still be reported within test case details.

Proxy servers

Does your corporate/office network run behind a proxy server? Contact us and we will supply you with a custom API Library for this case. Without this results will fail to upload to Tesults.

Have questions or need help? Contact us