This a reference guide for setting up Tesults with your RSpec tests. If you are not using RSpec, view the Ruby docs for information about integrating with a lower level library.
RSpec Tesults Formatter makes setup easy. Add one dependecy to your project.
gem install rspec_tesults_formatter
Once the rspec_tesults_formatter gem has been installed your results data will be pushed to Tesults if you follow the steps here to configure RSpec. By default the RSpec Tesults Formatter will be disabled. To enable the formatter you must provide a value for the tesults_target option. There are also other arguments that you can provide to report more detailed information, outlined below.
If you have not done so already in your RSpec folder, run:
This will create a .rspec file and a spec directory containing spec_helper.rb, if they do not already exist. Open up the .rspec file and add:
Then in the spec_helper.rb file add:
RSpec.configure do |config|
config.add_setting :tesults_target, :default => "token"
config.add_setting :tesults_files, :default => "/path/to/files"
config.add_setting :tesults_build_name, :default => "1.0.0"
config.add_setting :tesults_build_desc, :default => "Build description"
config.add_setting :tesults_build_result, :default => "pass"
config.add_setting :tesults_build_reason , :default => "Failure reason if fail"
token above should be replaced with the token for your Tesults target, available from the Tesults configuration menu. The other settings are explained in detail below in the 'Tesults Settings' section. Now when you run your tests, use the TesultsFormatter and your test results data will automatically be transmitted to Tesults:
rspec --format TesultsFormatter ./*_spec.rb
Alternatively, you can add this line to your spec_helper.rb:
# This line MUST come after other tesults settings are set such as tesults_target
config.formatter = "TesultsFormatter"
Then to use Tesults you can run your rspec command without needing --format:
Supporting Multiple Targets
To consolidate test results data from multiple test jobs you can create multiple
spec_helper.rb files, name each one after a target, for example
spec_helper_target2.rb. Each spec_helper can specifiy a different target token. Then remove the line
--require spec_helper from your
.rspec file and instead use
--require spec_helper when executing rspec in order to choose the file containing the target you want to use. In this case you have:
rspec --require spec_helper_target1.rb ./*_spec.rb
At this point RSpec Tesults Formatter will push results data to Tesults when your tests run as long as the tesults_target is supplied to indicate which target to use.
Provide the top-level directory where files generated during testing are saved for the running test run. Files, including logs, screen captures and other artifacts will be automatically uploaded.
This is one area where the RSpec Tesults Formatter is opinionated and requires that files generated during a test run be saved locally temporarily within a specific directory structure.
Store all files in a temporary directory as your tests run. After Tesults upload is complete, delete the temporary directory or just have it overwritten on the next test run.
The default behavior of the RSpec Tesults Formatter is to set the
describe value as the test suite, the
context value as the test description and the
it value as the test case name. Also be aware that if providing build files, the build suite is always set to [build] and files are expected to be located in temporary/[build]/buildname
Caution: If uploading files the time taken to upload is entirely dependent on your network speed. Typical office upload speeds of 100 - 1000 Mbps should allow upload of even hundreds of files quite quickly, just a few seconds, but if you have slower access it may take hours. We recommend uploading a reasonable number of files for each test case. The upload method blocks at the end of a test run while uploading test results and files. When starting out test without files first to ensure everything is setup correctly.
Use this to report a build version or name for reporting purposes.
Use this to report the build result, must be one of [pass, fail, unknown].
Use this to report a build failure reason.
This example test file includes one
context and three
it blocks. Tesults would report the suite as Test Suite A, the test cases would be reported as Test 1, Test 2 and Test 3. The context values will appear as the descriptions for each test case.
def add(a, b)
a + b
RSpec.describe Calculator do
describe 'Test Suite A' do
context 'Test 1 description' do
it 'Test 1' do
expect(Calculator.new.add(1, 2)).to eq(3)
context 'Test 2 description' do
it 'Test 2' do
expect(Calculator.new.add(2, 2)).to eq(4)
context 'Test 3 description' do
it 'Test 3' do
expect(Calculator.new.add(3, 3)).to eq(6)
Note that if you omit the
context blocks then have at least one
context (or an additional
describe) around all
it blocks in the file to correctly report test suite names.
If you execute multiple test runs in parallel or serially for the same build or release and results are submitted to Tesults within each run, separately, you will find that multiple test runs are generated on Tesults. This is because the default behavior on Tesults is to treat each results submission as a separate test run. This behavior can be changed from the configuration menu. Click 'Results Consolidation By Build' from the Configure Project menu to enable and disable consolidation by target. Enabling consolidation will mean that multiple test runs submitted with the same build name will be consolidated into a single test run.
If you dynamically create test cases, such as test cases with variable values, we recommend that the test suite and test case names themselves be static. Provide the variable data information in the test case description or other custom fields but try to keep the test suite and test name static. If you change your test suite or test name on every test run you will not benefit from a range of features Tesults has to offer including test case failure assignment and historical results analysis. You need not make your tests any less dynamic, variable values can still be reported within test case details.
Does your corporate/office network run behind a proxy server? Contact us and we will supply you with a custom API Library for this case. Without this results will fail to upload to Tesults.