Device farms

I once setup and maintained a device farm consisting of 40 Playstation 4 development kits while working at a Sony Interactive Entertainment's London Studio. This enabled automated tests to be run in parallel to speed up test runs and was highly valuable. Maintaining the farm was a big job however and ultimately took up about a significant amount of my time. A GPU related crash caused by a shader change would need me to manually reboot kits rather than rely on the usual programmatic method. Not doing so quickly would potentially impact pending tests. There was a need to update firmware on kits regularly, look into any networking or power issues, swapping out problem kits as well as developing and maintaining allocation and distribution code. In general it was a huge distraction and it ate into development and test automation authoring time.

When possible I recommend using a cloud-based device farm. There are privacy and security reasons why this is sometimes not possible but where sensible the increase in productivity can be massive. Maintaining a device farm internally is a major undertaking, it is especially problematic when there cannot be dedicated team members assigned to the task.

While the video game industry has few options here, most teams writing end-to-end automated tests run them on PCs, Macs, Desktop browsers, mobile browsers and iOS and Android devices. There are a range of options available for these teams including BrowserStack, SauceLabs and AWS Device Farm.

Using a cloud-based device farm means your team is free to concentrate on writing bespoke test automation that no one else can do and should be the focus for most automated test developers. Using a device farm makes parallel scaling easy, and opens up options for running tests on a larger range of devices and browser versions.

Your test execution application can continue to run on either your local servers or on cloud servers and you usually only need to change a small amount of code, mostly whatever setups a device or browser interface, everything else such as Appium or Selenium calls can stay the same. There is also no disruption with respect to continuing to use Tesults for reporting results, files, managing test cases, assigning failing test cases and notifications.

By Ajeet Dhaliwal