On supporting Continuous Testing with FITR test automation

This post was published on May 24, 2017

Last week I had the chance to participate as a contributor to my very first webinar. The people at SeaLights, an Israeli company that offers a management platform for Continuous Testing, asked me to come on the webinar and share my views on test automation and continuous testing. In this post, I’ll share some of the thoughts and opinions I talked about there.

Test automation is everywhere, however…
Test automation is everywhere. That’s probably nothing new.

A lot of organizations are adopting Continuous Integration and Continuous Delivery. Also nothing new.

To be able to ‘do’ CI/CD, a lot of organizations are relying on their automated tests to help safeguard quality thresholds while increasing release speed. Again, no breaking news here.

However, to safeguard quality in CI and CD you’ll need to be able to do Continuous Testing (CT). Here’s my definition of CT, which I used in the webinar:

Continuous Testing is a process that allows you to gauge the quality of your software on demand. No matter if you’re building and deploying once a month or once a minute, CT allows you to get insight into application quality at all times.

It won’t come as a surprise to you that automated tests often form a big part of an organization’s CT strategy. However, just having automated tests is not enough to be able to support CT. Your automated test approach should not just cover application functionality and coverage, but also:

  • A solid test data management strategy
  • Effective test environment management
  • Informative reporting, targeted towards all relevant audiences

And probably many more things that I’m forgetting here..

In order to be able to leverage your automated tests successfully for supporting CT, I’ve come up with a model based on four pillars that need to be in place:

From AT to CT with FITR tests

Let’s take a quick look at each of these FITR pillars and how they are necessary in CT.

Focused
Automated tests need to be focused to effectively support CT. ‘Focused’ has two dimensions here.

First of all, your tests should be targeted at the right application component and/or layer. It does not make sense to use a user interface-driven test to test application logic that’s exposed through an API (and subsequently presented through the user interface), for example. Similarly, it does not make sense to write API-level tests that validate the inner workings of a calculation algorithm if unit tests can provide the same level of coverage. By now, most of you will be familiar with the test automation pyramid. While I think it’s not to be used as a guideline, I find that the pyramid does provide a good starting point for discussion when it comes to focusing your tests at the right level. Use the model to your advantage.

The second aspect of focused automated tests is that your tests should test what they can do effectively. This boils down to sticking to what your test solution and tools in it do best, and leaving the rest either to other tools or to testers, depending on what’s there to be tested. Don’t try and force your tool to do things it isn’t supposed to (here’s an example).

If your tests are unfocused, they are far more likely to be slow to run, to have high maintenance costs and to provide inaccurate or shallow feedback on application quality.

Informative
Touching upon shallow or inaccurate feedback, automated tests also need to be informative to effectively support CT. ‘Informative’ also has two separate dimensions.

Most importantly, the results produced and the feedback provided by your automated tests should allow you, or the system that’s doing the interpretation for you (such as an automated build tool), make important decisions based on that feedback. Make sure that the test results and reporting provided contain clear results, information and error messages, targeted towards the intended audience. Keep in mind that every audience has its own requirements when it comes to this information. Developers likely want to see stack trace, whereas managers don’t. Find out what the target audience for your reporting and test results is, what their requirements are, and then cater to them as best as you can. This might mean creating more that one report (or source of information in general) for a single test run. That’s OK.

Another important aspect of informative automated tests is that it should be clear what they do (and what they don’t do). You can make your tests themselves be more informative through various means, including (but not limited to) using naming conventions, using a BDD tool such as Cucumber or SpecFlow to create living documentation for your tests (blasphemy? maybe.. but if it works, it works), and following good programming practices to make your code better readable and maintainable.

When automated test solutions and the results they produce are not informative, valuable time is wasted analyzing shallow feedback, or gathering missing information, which evidently breaks the ‘continuous’ part of CT.

Trustworthy
When you’re relying on your automated tests to make important decisions in your CT activities, you’d better make sure they’re trustworthy. As I described in more detail in previous posts, automated tests that cannot be trusted are essentially worthless. Make sure to eliminate false positives (tests that fail when they shouldn’t), but also false negatives (tests that pass when they shouldn’t).

Repeatable
The essential idea behind CT (referring to the definition I gave at the beginning of this blog post) is that you’re able to determine application quality on demand. Which means you should be able to run your automated tests on demand. Especially when you’re including API-level and end-to-end tests, this is often not as easy as it sounds. There are two main factors that can hinder the repeatability of your tests:

  • Test data. This is in my opinion one of the hardest ones to get right, especially when talking end-to-end tests. Lots of applications I see and work with have (overly) complex data models or share test data with other systems. And if you’re especially lucky, you’ll get both. A solid test data strategy should be put in place to do CT, meaning that you’ll either have to create fresh test data at the start of every test run or have the ability to restore test data before every test run. Unfortunately, both options can be quite time consuming (if at all attainable and manageable), drawing you further away from the ‘C’ in CT instead of bringing you closer.
  • Test environments. If your application communicates with other components, applications or systems (and pretty much all of them do nowadays), you’ll need suitable test environments for each of these dependencies. This is also easier said than done. One possible way to deal with this is by using a form of simulation, such as mocking or service virtualization. Mocks or virtual assets are under your full control, allowing you to speed up your testing efforts, or even enable them in the first place. Use simulation carefully, though, since it’s yet another thing to be managed and maintained, and make sure to test against the real thing periodically for optimal results.

Having the above four pillars in place does not guarantee that you’ll be able to perform your testing as continuously as your CI/CD process requires, but it will likely give it a solid push in the right direction.

Also, the FITR model I described here is far from finished. If there’s anything I forgot or got wrong, feel free to comment or contact me through email. I’d love to get feedback.

Finally, if you’re interested in the webinar I talked about earlier, but haven’t seen it, it’s freely available on YouTube:

"