Github Actions

Running tests and reporting results within a continuous integration (CI) is a common task. If you use Github Actions and your workflow includes a job with a step that runs your tests, you may need to check the result to determine whether the pipeline is a success or a failure.

Tesults API results check

Some test frameworks automatically fail the process if the tests do not all pass. Others do not. For workflows that upload results to Tesults, the Tesults API results check Action in the Github Actions Marketplace can check the uploaded results and fail the job when required. Usage is demonstrated below. https://github.com/marketplace/actions/tesults-api-results-check

In the case of the Playwright test framework for example the step will report results to Tesults and then Playwright will automatically fail the job if all test cases do not pass. In this case you do not need to do anything else.

name: Playwright Tests
on:
push:
  branches: [ main, master ]
pull_request:
  branches: [ main, master ]
jobs:
test:
  timeout-minutes: 60
  runs-on: ubuntu-latest
  steps:
  - uses: actions/checkout@v3
  - uses: actions/setup-node@v3
  with:
    node-version: 18
  - name: Install dependencies
  run: npm ci
  - name: Install Playwright Browsers
  run: npx playwright install --with-deps
  - name: Run Playwright tests
    env:
      TARGET: ${{ secrets.TARGET }}
      BUILD: ${{ format('{0} {1}', 'Build ', github.run_number) }}
    run: npx playwright test

In the case above this assumes the playwright.config.js file contains reporter parameters like this:

reporter: [['playwright-tesults-reporter',
{'tesults-target': process.env.TARGET,
'tesults-build-name': process.env.BUILD,
'tesults-build-result': 'pass'}]],

This all works well for the Playwright test framework because it handles returning the appropriate process code for failing the job.

In the case where the test framework does not automatically fail the job, after tests have run and results data has been uploaded to Tesults, you would want to check results via the Tesults API, and then determine whether to fail the job or not.

To make this easy for Github Actions Tesults has made the Tesults API results check Action available in the Github Actions Marketplace:https://github.com/marketplace/actions/tesults-api-results-check

Using this action is demonstrated below.

- name: Check
id: check
uses: tesults/tesults-api-results-check@1.0.6
with:
  target: ${{ secrets.TARGET }}
  build: ${{ format('{0} {1}', 'Build ', github.run_number) }}
  api_token: ${{ secrets.API_TOKEN }}

The target, build and api_tokens are all required.

1. target - the id of the target (available from the Tesults configuration menu (https://www.tesults.com/config) and Tesults API)
2. build - the build name, this is self generated and should be unique for the test run, we recommend simply using github.run_number
3. api_token - the api token for the Tesults project, generated from the Tesults configuration menu (https://www.tesults.com/config)

This action will automatically fail the job if the pass rate is not 100%, however if you need the number of passes and total for any reason, the action outputs these values and they can be examined.

# Use the output from the `Check` step
- name: Get the number of passes and total
run: echo "${{ steps.check.outputs.pass }} pass out of ${{ steps.check.outputs.total }} total"

View the output in the Actions tab on Github.

Test Automation Reporting for GitHub Actions

Existing Tesults customers can also use the Test Automation Reporting for GitHub Actions Action to get richer test reporting directly in GitHub while continuing to upload the same test run to Tesults. It adds a GitHub Actions job summary with test counts, suites, failure details, source annotations, attachments and optional Check Runs. https://github.com/marketplace/actions/test-automation-reporting-for-github-actions

Keep the target token in your existing Tesults reporter configuration. Current Tesults test framework integrations can upload results to Tesults and write the local Action results file during the same test run. Add the Action immediately before the step that runs your tests:

- name: Set up test automation reporting
  uses: tesults/test-automation-reporting@v1

- name: Run tests
  env:
    TARGET: ${{ secrets.TARGET }}
  run: npm test

The target token remains part of your test framework reporter configuration, as in the Playwright example above; it is not passed to the reporting Action. The Action can also be used without a target token when only a GitHub report is required. In that mode, results are not uploaded to Tesults.

Supported frameworks, configuration options and complete setup instructions are available in the Action documentation.