Why and how I still use the test automation pyramid

This post was published on September 6, 2017

Last week, while delivering part one of a two-evening API testing course, I found myself explaining the benefits of writing automated tests at the API level using the test automation pyramid. That in itself probably isn’t too noteworthy, but what immediately struck me as odd is that I found myself apologizing to the participants that I used a model that has received so many criticism as the pyramid.

Odd, because

Half of the participants hadn’t even heard of the test automation pyramid before
The pyramid, as a model, to me is still a very useful way for me to explain a number of concepts and good practices related to test automation.

#1 is a problem that should be tackled by better education around software testing and test automation, I think, but that’s not what I wanted to talk about in this blog post. No, what I would like to show is that, at least to me, the test automation pyramid is still a valuable model when explaining and teaching test automation, as long as it’s used in the right context.

The basis of what makes the pyramid a useful concept to me is the following distinction:

It is a model, not a guideline.

A guideline is something that’s (claiming to be) correct, under certain circumstances. A model, as the statistician George Box said, is always wrong, but some models are useful. To me, this applies perfectly to the test automation pyramid:

There’s more to automation than meets the UI
The test automation pyramid, as a model, helps me explain to less experienced engineers that there’s more to test automation than end-to-end tests (those often driven through the user interface). I explain this often using examples from real life projects, where we chose to do a couple of end-to-end tests to verify that customers could complete a specific sequence of actions, combined with a more extensive set of API tests to verify business logic at a lower level, and why this was a much more effective approach than testing everything through the UI.

Unit testing is the foundation
The pyramid, as a model, perfectly supports my belief that a solid unit testing strategy is the basis for any successful, significantly-sized test automation effort. Anything that can be covered in unit tests should not have to be covered again in higher level tests, i.e., at the integration/API or even at the end-to-end level.

E2E and UI tests are two different concepts
The pyramid, as a model, helps me explain the difference between end-to-end tests, where the application as a whole is exercised from top (often the UI) to bottom (often a database), and user interface tests. The latter may be end-to-end tests, but unbeknownst to surprisingly many people you can write unit tests for your user interface just as well.There’s a reason the top layer of the pyramid that I use (together with many others) says ‘E2E’, not ‘UI’…

Don’t try to enforce ratios between test automation scope levels
The pyramid, when used as a guideline, can lead to less than optimal test automation decisions. This mainly applies to the ratio between the number of tests in each of the E2E, integration and unit categories. Even though well though through automation suites will naturally steer towards a ratio of more unit tests than integration tests and more integration tests than E2E tests, it should never be forced to do so. I’ve even seen some people, which unfortunately were the ones in charge, make decisions on what and how to automate based on ratios. Some even went as far as saying ‘X % of our automated tests HAVE TO be unit tests’. Personally, I’d rather go for the ratio that delivers in terms of effectiveness and time needed to write and maintain the tests instead.

Test automation is only part of the testing story
‘My’ version of the test automation pyramid (or at least the version I use in my presentations) prominently features what I call exploratory testing. This helps remind me to tell those that are listening that there’s more to testing than automation. I usually call this part of the testing story ‘exploratory testing’, because this is the part where humans explore and evaluate the application under test to inform themselves and others about aspects of its quality. This is what’s often referred to as ‘manual testing’, but I don’t like that term.

As you can see, to me, the test automation pyramid is still a very valuable model (and still a useless guideline) when it comes to me explaining my thoughts on automation, despite all the criticism it has received over the years. I hope I never find myself apologizing for using it again in the future..