Do you want a framework or a solution?

This post was published on August 24, 2016

Reading through recent comments on this blog, as well as questions that are being asked in some of the LinkedIn groups I am a member of, something has started to strike me as odd. It seems like that in every other comment, someone is asking ‘how do I add so and so to my automation framework?’ or ‘how do I design a framework that does X, Y and Z?’. It isn’t the question as such that is the problem (if you can call me thinking of something as odd a problem in the first place), questions are good. You learn a lot by asking questions, and even more by asking the right questions. No, what gets to me is that all of these questions refer to automation frameworks. Why does it seem that so many people are so focused on ‘designing’ a ‘framework’? In this post, I’d like to explain why I think it’s better not to focus on frameworks in test automation, but shift your attention to something much more important.

First of all, what is a framework anyway? If you’d ask me, this is a framework:

But within the test automation realm, what does a framework constitute? Wikipedia defines a test automation framework as:

A test automation framework is an integrated system that sets the rules of automation of a specific product. This system integrates the function libraries, test data sources, object details and various reusable modules. These components act as small building blocks which need to be assembled to represent a business process. The framework provides the basis of test automation and simplifies the automation effort.

So, it’s an integrated system of libraries, modules and other stuff that sets the rules of automation. And there’s exactly the main gripe I have with talking and thinking about automation in terms of frameworks. Your framework shouldn’t be the one setting the rules for automation. Instead, your application under test, the quality of that application you want to have achieved when deploying it into production and the tests and checks that form the evidence of this quality should be the ones setting the rules. Test automation is merely one of the possible means to adhere to these rules.

When you start your test automation efforts by building a framework, either from scratch or by glueing together two or more existing tools and only then try and fit in all tests you want to perform, you’re approaching the problem from the wrong end. Trying to fit everything into an existing framework limits flexibility, as your prime concern becomes ‘how can I perform this test using my framework?’ instead of ‘what is the best way to execute this test?’. It’s a bit like buying a two-bedroom apartment and only then realizing you’ve got six children. And a horse. Sure, it might fit, but it’s not necessarily going to be comfortable. It’s the same with test automation. Building a framework (the two-bedroom apartment) and then trying to fit in all sorts of tests (your six children, plus yourself and your significant other) will probably not lead to the most efficient (comfortable) situation. And don’t get me started on that horse..

Consider the following scenario: you have created a number of automated tests for your web application using Selenium (for browser automation), Cucumber (for BDD support) and a tool such as TestNG or JUnit (for assertions and possibly for reporting). At some point in time, you’re asked to also start writing automated tests for the newly built API that exposes part of the application logic. What do you do?

1. Do you write some more tests that exercise the user interface, which in turn invoke the API ‘under the hood’?
2. Do you search for an API testing tool that fits best inside your framework, glue it all together, start specifying features and scenarios for the API tests and automate away?
3. Or do you look for the tool that best supports your actual API testing needs, use that and make it part of your overall testing activities, for example by making API tests a stage in your Continuous Integration or Continuous Delivery process?

If you chose #1, then I have three words for you (well, technically, four): don’t do that. If you opted for #2, you might have thought that API testing capabilities would be a nice addition to your framework, making it more versatile than it was before. But are you sure you can specify the behavior of your API in the same manner (i.e., describing the business value with the same amount of clarity) in Given/When/Then format? What if you discarded an API testing tool that fits the task at hand much better, just because it didn’t fit in your framework?

Option #1 and especially #2 are examples of what I’d like to call ‘framework think‘. While you think you’re building something flexible and reusable, you´re really limiting yourself by discarding solutions that do not fit the framework. Option #3, on the other hand, describes what I’d call ‘solution think‘, in my opinion a far better way of thinking and talking about test automation.

Test automation efforts should serve a higher purpose: solving the problem of giving fast and reliable insight into specific aspects of application quality. Every decision that’s made when implementing test automation, from the decision whether or not to automate at all down to selecting a specific tool to perform a certain test, should be made with this problem in mind. You should always ask: does this decision move me closer to providing a solution? And to what problem? The second question is equally, if not more important, since the perfect solution to a problem that’s irrelevant is just as wasteful as a solution that’s not right in the first place.

So, in short, here’s my advice: stop ‘framework think’. Instead, start practicing ‘solution think’. It might be only a change in mindset, but sometimes that’s all it takes to get you in the right direction.

Thanks to Alan Richardson for putting me on track to write this post with a quote from his 2015 Test Automation Day keynote. He also wrote a somewhat related post recently, which I think you should all read.

"