Sunday, November 2, 2008

Selenium - Chemistry for your UATs

Off and on for the past few weeks I have been working on migrating my current project's suite of user acceptance tests (UATs) from WatiN to Selenium.

Why you ask? Several reasons.

  1. Our current WatiN tests are slow and brittle. Some of this is WatiN's fault, some our own
  2. WatiN doesn't have great support for AJAX testing
  3. WatiN only works in IE on Windows. It does this well, but our app needs to work in 7 browsers at the moment (IE 6 & 7, FF 2 & 3 on Mac & Win, Safari on Mac) soon to be 10 (Safari on Win, Chrome, IE 8).
  4. Our company is going to be creating a Selenium Grid with remote control nodes of various browsers on different platforms for all projects to be able to use.
  5. Currently the UATs don't run as part of any automated build
Thus, we are slowly trying to see what kind of effort it will take to convert our tests, rewrite/fix/upgrade where we can and need to, and automate as much as we can.

The really nice thing is the Selenium Grid and the ability to have a large number of RC's standing by in VMs for any project that wants to run tests. If you don't know about Selenium Grid, read about it here.

In our initial efforts to convert the tests we have found a few things to be true:
  • Selenium has some built in checks that help with testing/action on AJAX calls, but they aren't perfect and primarily rely on built in check-wait loops on elements being on the page or Javascript functions changing values.
  • Firefox 3 is not yet fully supported. A simple hack allows it to be used, however not all of the integrations and XPath work correctly
  • I had a fair bit of trouble getting Safari to work on the Mac with Selenium. Evidently there are some architectural issues the team is battling with related to proxies and libraries ... but I was able to get it to work eventually
  • The Grid/Remote Control environment works well but still has some bugs. Closing the window for a running RC on windows doesn't unregister the RC from the grid.
  • Selenium seems to be a small but noticeable amount faster than WatiN
  • Converting tests is hard when trying to ensure that the same exact thing is being tested before and after
Other than these issues, things are progressing. Three larger problems we are seeing and havn't come up with solutions for yet are:
  • How can we automate the UATs running on the continuous integration server to run all browsers in parallel (e.g. running each test against all browsers at the same time, then moving on to the next one) while still maintaining one build/failure point?
  • Is it possible to integrate the running of UATs against multiple browsers to the developer's work flow? Possible tooling support?
  • Is it possible to rework a test-runner to run all of our tests in parallel depending on the number of RC's of a particular type that are available (e.g. if we have a ton of FF 3 RCs, run several tests all at once instead of waiting for each other)
It looks like we are going to be in between contracts for about a week so we are hoping to come up with some solutions to these. I will post back if we do.

** If you have any suggestions on how to automate or parallelize the Selenium tests, check our my StackOverflow question.

2 comments:

Graham said...

Hey Benjamin,

I previously replied to your stackoverflow question and have recently posted on my blog about this. I know this is an old post but I'm just wondering how you solved your issues in the end?

Benjamin P Lee said...

Graham,

Thanks for the answer on stackoverflow and the follow-up comment. To be honest, we never found a great solution to this issue. I am no longer on that particular team, but currently they are running a few different builds (one for each main platform) with the UATs and any other browser/platform combinations on an on-demand basis. I will point them toward your article to see if it helps them any.

thank again.