Visual Studio Unit Testing Framework (MSTest)

Read the C# docs for complete details about supported build and test case properties and a complete code example for how to apply values for each property.

Unlike all of the other test frameworks we document here, MSTest does not support the type of extensibility that makes plugin and codeless integration possible and you will need to add code to report test results data to Tesults.

Installation

The Tesults package is hosted on The NuGet Gallery: https://www.nuget.org/packages/tesults/

Run this command in Package Manager Console.

Install-Package tesults

Configuration

Tesults recommends using the [TestCleanup] and [AssemblyCleanup] decorators to implement methods that extract results data from TestContext to generate results data for reporting and upload.

A complete example is provided below, TesultsReporter.cs, ready to be copied and pasted for use with your system. Use as is or edit. To upload files uncomment the optional files section and provide paths to where files are saved for each test case. To map test logs, screenshots and failure reason logging to a test case, you could consider have a temporary directory for each test case to save to during tests.

Note that in order to use TestReporter.cs everyone of your Test classes will need to have this code added:

Add to each test class:

public TestContext TestContext { get; set; }

[TestCleanup]
public void TestComplete()
{
  TesultsReporter.TestComplete(TestContext);
}

TesultsReporter.cs

using System;
using System.Collections.Generic;
using Microsoft.VisualStudio.TestTools.UnitTesting;

namespace TesultsMSTest
{
  [TestClass]
  public class TesultsReporter
  {
      public static List<Dictionary<string, object>> testCases = new List<Dictionary<string, object>>();

      public static void TestComplete (TestContext testContext)
      {
        var testCase = new Dictionary<string, object>();
        testCase.Add("name", testContext.TestName);
        testCase.Add("desc", testContext.TestName);
        testCase.Add("suite", testContext.FullyQualifiedTestClassName);
        var result = testContext.CurrentTestOutcome.ToString();
        var properties = testContext.Properties;
        if (result == "Passed")
        {
          result = "pass";
        }
        else if (result == "Failed")
        {
          result = "fail";
          testCase.Add("reason", "Access the error from a file");
          // MSTest does not provide a way to get the failure reason from
          // TestContext unfortunately so you must save the test failure
          // reason (assert, exception etc.) to a file and then add it
          // to the reason attribute. We understand this is not ideal.
          // You may want to consider using the NUnit test framework.
        }
        else
        {
          result = "unknown";
        }
        testCase.Add("result", result);

        // Optional for file uploading
        // var files = new List<string>();
        // files.Add(@"C:\\path-to-file-for-this-test\\img.png");
        // testCase.Add("files", files);

        testCases.Add(testCase);
      }

      [AssemblyCleanup]
      public static void Upload ()
      {
        var results = new Dictionary<string, object>();
        results.Add("cases", testCases);

        var data = new Dictionary<string, object>();
        data.Add("target", "token");
        data.Add("results", results);

        var response = Tesults.Results.Upload(data);
        Console.WriteLine("Success: " + response["success"]);
        Console.WriteLine("Message: " + response["message"]);
        Console.WriteLine("Warnings: " + ((List<string>)response["warnings"]).Count);
        Console.WriteLine("Errors: " + ((List<string>)response["errors"]).Count);
      }
    }
  }

The target value, 'token' above should be replaced with your Tesults target token. If you have lost your token you can regenerate one from the config menu.

The upload method returns a response of type Map<String, Object> that you can use to check whether the upload was a success.

Value for key "success" is a Boolean: true if results successfully uploaded, false otherwise.

Value for key "message" is a String: if success is false, check message to see why upload failed.

Value for key "warnings" is a List<String>, if size is not zero there may be issues with file uploads.

Value for key "errors" is a List<String>, if "success" is true then this will be empty.


Caution: If uploading files the time taken to upload is entirely dependent on your network speed. Typical office upload speeds of 100 - 1000 Mbps should allow upload of even hundreds of files quite quickly, just a few seconds, but if you have slower access it may take hours. We recommend uploading a reasonable number of files for each test case. The upload method blocks while uploading test results and files. When starting out test without files first to ensure everything is setup correctly.

Consolidating multiple test results submissions into one test run

If you execute multiple test runs in parallel or serially for the same build or release and results are submitted to Tesults within each run, separately, you will find that multiple test runs are generated on Tesults. This is because the default behavior on Tesults is to treat each results submission as a separate test run. This behavior can be changed from the configuration menu. Click 'Results Consolidation By Build' from the Configure Project menu to enable and disable consolidation by target. Enabling consolidation will mean that multiple test runs submitted with the same build name will be consolidated into a single test run.

Dynamically created test cases should have static names

If you dynamically create test cases, such as test cases with variable values, we recommend that the test suite and test case names themselves be static. Provide the variable data information in the test case description or other custom fields but try to keep the test suite and test name static. This way you will you benefit from a range of features Tesults has to offer including test case failure assignment and historical results analysis. You need not make your tests any less dynamic, variable values can still be reported within test case details.

Proxy servers

Does your corporate/office network run behind a proxy server? Contact us and we will supply you with a custom API Library for this case. Without this results will fail to upload to Tesults.

Have questions or need help? Contact us