22.5.12

Acceptance Test: What needs automation?

Fact:
As you can read here and here, Acceptance Testing is a technique for demonstrating the customer that features are implemented correctly and are doing what they are supposed to. They are black box tests and specify the feature from end-to-end (in terms of process and not architectural layers). Furthermore, they are supposed to be automated as they are used for the regression suite.

Problem:
Assuming we are implementing a product with a web-based graphical user interface, we therefore decide to use one of the various GUI automation tools like Selenium, Marathon or Watij. Motivated as we are, we start to write long test scripts for each of our acceptance tests. After some stories the customer wants lots of GUI changes and we need half a sprint just for fixing our tests and the more changes we have the more time we spent with test fixing.

Suggestion:
Martin Fowler argues in this article that it is definitely not necessary to automate 100% of the GUI part as it hardly pays off. Furthermore, also C. Lowell and J. Stell-Smith, the creators of Marathon, advise in this article to keep the testing pyramid in mind and keep the GUI tests around 10%.
Even though they are writing about test portfolios in general, I think this concept can be used for Acceptance Testing too: stick to the testing pyramid and stay with 10 % GUI tests.
Furthermore, it is probably not necessary to keep and fix all broken GUI tests. One possibilitiy, is to kick out tests that fail (because of GUI changes) certain times in a row.

So the question arises what and how to test? 
As before, test the business rules (acceptance criteria), but on API not GUI level.
Gojko Adzic approaches  in his (actually quite old ;) ) article to concentrate on the key business scenarios. That means, end-to-end tests for such scenarios (which should not be too many) are created and automated and further used as Smoke Tests. Other GUI related tests verify stuff like"are fields correctly displayed, when hitting this button", but not business rules.
Therefore, an architecture is needed that allows testing on another (architectural)  layer. As stated by Uncle Bob here, one approach would be to use the same services like the GUI and mimic the its behavior within the test. This, however, presupposes a correct separation of frontend and business layer exists. If there is no usable API, it is absolutely OK to build in test hooks, according to J. Whittaker.


Further Readings:
Brian Marick - When Should a Test Be Automated?
Behavior Driven Development - It raises the level of abstraction and thereby helps to reuse code, resulting in less broken code when dealing with GUI changes.






No comments:

Post a Comment