Selenium testing in the cloud with BrowserStack

Hello fellow Jenkins fans, I'm back for another instalment of JenkinsHeaven.

This post is a little off topic but is so seriously cool it has to be covered.

In the past, to perform cross-browser, or better yet cross-platform testing, you would install the Selenium Grid plugin into your Jenkins master and have a number of physical (or virtual) machines with different browsers installed on them and set up some tags so that the Selenium Grid could find them. Once all this infrastructure was set up you would write your Selenium tests.

Side Note: We're all doing cross-platform testing as part of our builds right? If you answered "Wrong!" like Arnie in 1985's Commando. I highly recommend that you integrate cross-platform testing in all your web application builds before the 11 hours is up (if you're confused, watch the movie).

While the traditional approach of in-house testing slave infrastructure is still valid and might be necessary for your organisation, with Browserstack you can put all the infrastructure in the cloud. They take care of it for you. Browserstack makes it really easy. They essentially provide the functionality of Selenium Grid, the test slaves and tags.

There are two main ways of interacting with browserstack: Automate and ScreenshotsAPI. There is "Live" testing but in my opinion its only really useful for concept demonstrations. Its not programmatic like the first two and we're interested in the programmatic stuff. Responsive appears to me to be the same as ScreenshotsAPI.

They support a ton of browsers and platforms and you can even test non-public facing urls (like test environments) with the web tunnel functionality. DANGER WILL ROBINSON: Be careful to note that the browsers and platforms supported by ScreenshotsAPI is not the same as Automate. They are slightly different (at least that is the case at the time of writing).

Happily, all this leaves you to focus on writing your tests. With only a few tests written, the effect of configuring just some of the supported platforms means that in a short amount of time you find yourself with a test suite with hundreds of combinations. Tests X Browsers X Operating Systems. :D

I found that while they give you automate code examples (Python, Ruby, Java, C#, PHP, Perl, Node.js) which are a helpful bootstrap it takes a bit of work to understand exactly what you can and can't do. For instance, I tried for 2 days get a batch file to call the third-party ScreenShooter tool repeatedly. Even with the /w argument it won't work after the first iteration (at least I haven't been able to yet.). BrowserStack's support team advised me that their servers limit one screenshot call at a time. You won't find that in their documentation.

While the documentation on the BrowserStack website isn't too bad I feel it is lacking in some areas.

I wrote stacked-browsers to pull all the threads together and make your lives easier...that...and my current project needed it.

stacked-browsers implements different flavors of integration with BrowserStack: Java, CSharp and Ruby (via ScreenShooter. I am yet to get the Screenshots API Ruby Library to work). If you would like to implement testers for the other languages feel free to send me a pull request. The structured readme at each level of the source tree should explain everything from setup to building to running. Take note of the batch files I wrote to make these steps easy.

It is cool to execute the TestRunner from the command line and see the test session appear almost immediately on Browserstack.com on the Automate page with the name you specified.

Browserstack's example code has 16 degrees of parallelism. I noticed in the top corner of the account I was using that they afforded me 5 degrees of parallelism. I had mixed results with this. With 2, I only seemed to have issues with one of the one of the Samsung android platforms, with 5 set, I had 8 tests running at times and had more timeout issues. I also noticed after about 10 minutes of my session running that this dropped to 2 test running anyway. I concluded that BrowserStack was throttling my session. I asked about this but their support team are yet to confirm or deny.

For a quick win, I chose Java Parallel JUnit (scroll down to "JUnit test for running in parallel browsers" for the helper class) as the initial mode of choice, but I expect the other flavors would work equally easily. Overall, I am very impressed with how smoothly BrowserStack works and would recommend it to anyone wanting to do cross-platform testing.

DISCLAIMER: I am neither employed by BrowserStack nor being paid for these comments (I am not a Sydney radio host...you know what I mean). I just think it works really well.

Oh, and if anyone manages to write a batch file that successfully calls the ScreenShooter tool or implement a client for the Screenshots API Ruby Library (or any other improvements) send me a pull request for the stacked-browsers repo.

I hope you have found this useful.

UPDATE: Saucery2 For NUnit is the commercial version of Stacked Browsers.

Till next time...