RSpec test reporting

This a reference guide for setting up Tesults with your RSpec tests. If you are not using RSpec, view the Ruby docs for information about integrating with a lower level library.

Installation

RSpec Tesults Formatter makes setup easy. Add one dependecy to your project.

gem install rspec_tesults_formatter

Configuration

Once the rspec_tesults_formatter gem has been installed your results data will be pushed to Tesults if you follow the steps here to configure RSpec. By default the RSpec Tesults Formatter will be disabled. To enable the formatter you must provide a value for the tesults_target option. There are also other arguments that you can provide to report more detailed information, outlined below.

If you have not done so already in your RSpec folder, run:

rspec --init

This will create a .rspec file and a spec directory containing spec_helper.rb, if they do not already exist. Open up the .rspec file and add:

.rspec

--require rspec_tesults_formatter
--require spec_helper

Then in the spec_helper.rb file add:

RSpec.configure do |config|
  config.add_setting :tesults_target, :default => "token"
  config.add_setting :tesults_files, :default => "/path/to/files"
  config.add_setting :tesults_build_name, :default => "1.0.0"
  config.add_setting :tesults_build_desc, :default => "Build description"
  config.add_setting :tesults_build_result, :default => "pass"
  config.add_setting :tesults_build_reason , :default => "Failure reason if fail"
end

The token above should be replaced with the token for your Tesults target, available from the Tesults configuration menu. The other settings are explained in detail below in the 'Tesults Settings' section. Now when you run your tests, use the TesultsFormatter and your test results data will automatically be transmitted to Tesults:

rspec  --format TesultsFormatter ./*_spec.rb

Alternatively, you can add this line to your spec_helper.rb:

config.formatter = "TesultsFormatter" # This line MUST come after other tesults settings are set such as tesults_target

Then to use Tesults you can run your rspec command without needing --format:

rspec ./*_spec.rb

Supporting Multiple Targets

To consolidate test results data from multiple test jobs you can create multiple spec_helper.rb files, name each one after a target, for example spec_helper_target1.rb and spec_helper_target2.rb. Each spec_helper can specifiy a different target token. Then remove the line --require spec_helper from your .rspec file and instead use --require spec_helper when executing rspec in order to choose the file containing the target you want to use. In this case you have:

.rspec

--require rspec_tesults_formatter

Multiple spec_helper for each target

/project
  /spec
    spec_helper_target1.rb
    spec_helper_target2.rb

Command

rspec --require spec_helper_target1.rb ./*_spec.rb

Tesults Target

tesults_targetRequired

The tesults_target is the token value for the target you wish you to upload results to. You can retrieve your token from the Tesults configuration menu. Learn more about targets here

Basic configuration complete

At this point RSpec Tesults Formatter will push results data to Tesults when your tests run as long as the tesults_target is supplied to indicate which target to use.

Files

During a test run, save test generated files such as logs and screenshots to a local temporary directory. At the end of the test run all files will automatically be saved to Tesults as long as you save files in the directory structure below.
  • expanded temporary
    • expanded Test suite name
      • expanded Test case name
        • file 1
        • file 2

For example:

  • expanded temporary
    • expanded Test Suite A
      • expanded Test 1
        • test.log
        • screenshot.png
      • expanded Test 2
        • test.log
        • screenshot.png
    • expanded Test Suite B
      • expanded Test 3
        • metrics.csv
        • error.log
      • expanded Test 4
        • test.log

If not using test suites then omit the suite directories:

  • expanded temporary
    • expanded Test 1
      • test.log
      • screenshot.png
    • expanded Test 2
      • test.log
      • screenshot.png
tesults_filesOptional

Provide the top-level directory where files generated during testing are saved for the running test run. Files, including logs, screen captures and other artifacts will be automatically uploaded.

This is one area where the RSpec Tesults Formatter is opinionated and requires that files generated during a test run be saved locally temporarily within a specific directory structure.

Store all files in a temporary directory as your tests run. After Tesults upload is complete, delete the temporary directory or just have it overwritten on the next test run.

The default behavior of the RSpec Tesults Formatter is to set the describe value as the test suite, the context value as the test description and the it value as the test case name. Also be aware that if providing build files, the build suite is always set to [build] and files are expected to be located in temporary/[build]/buildname

Caution: If uploading files the time taken to upload is entirely dependent on your network speed. Typical office upload speeds of 100 - 1000 Mbps should allow upload of even hundreds of files quite quickly, just a few seconds, but if you have slower access it may take hours. We recommend uploading a reasonable number of files for each test case. The upload method blocks while uploading test results and files. When starting out test without files first to ensure everything is setup correctly.

Build

tesults_build_nameOptional

Use this to report a build version or name for reporting purposes.


tesults_build_resultOptional

Use this to report the build result, must be one of [pass, fail, unknown].


tesults_build_descOptional

tesults_build_reason=reasonOptional

Use this to report a build failure reason.

How RSpec test cases are reported in Tesults


This example test file includes one describe, three context and three it blocks. Tesults would report the suite as Test Suite A, the test cases would be reported as Test 1, Test 2 and Test 3. The context values will appear as the descriptions for each test case.

test_spec.rb
class Calculator
  def add(a, b)
    a + b
  end
end

RSpec.describe Calculator do
  describe 'Test Suite A' do
    context 'Test 1 description' do
      it 'Test 1' do
        expect(Calculator.new.add(1, 2)).to eq(3)
      end
    end
    context 'Test 2 description' do
      it 'Test 2' do
        expect(Calculator.new.add(2, 2)).to eq(4)
      end
    end
    context 'Test 3 description' do
      it 'Test 3' do
        expect(Calculator.new.add(3, 3)).to eq(6)
      end
    end
  end
  end

Note that if you omit the context blocks then have at least one context (or an additional describe) around all it blocks in the file to correctly report test suite names.

Consolidating multiple test results submissions into one test run

If you execute multiple test runs in parallel or serially for the same build or release and results are submitted to Tesults within each run, separately, you will find that multiple test runs are generated on Tesults. This is because the default behavior on Tesults is to treat each results submission as a separate test run. This behavior can be changed from the configuration menu. Click 'Results Consolidation By Build' from the Configure Project menu to enable and disable consolidation by target. Enabling consolidation will mean that multiple test runs submitted with the same build name will be consolidated into a single test run.

Dynamically created test cases should have static names

If you dynamically create test cases, such as test cases with variable values, we recommend that the test suite and test case names themselves be static. Provide the variable data information in the test case description or other custom fields but try to keep the test suite and test name static. This way you will you benefit from a range of features Tesults has to offer including test case failure assignment and historical results analysis. You need not make your tests any less dynamic, variable values can still be reported within test case details.

Proxy servers

Does your corporate/office network run behind a proxy server? Contact us and we will supply you with a custom API Library for this case. Without this results will fail to upload to Tesults.

Have questions or need help? Contact us