Testing vs Checking

There has been a lot of discussion over the last couple of years about test automation and in particular the varying definitions of testing vs checking and how that applies to test automation.

I broadly agree there is a difference, here is my paraphrased understanding of each definition;

Testing – the art and science of conducting experiments and carefully observing the results, all the while making multiple evaluations against explicit and implicit expectations. A fundamentally human, (or manual if you prefer), exercise.

Checking – the deterministic evaluation of the outcome of an action or step such that a pass or fail is recorded.

But there seems to be an underlying theme to most of these discussions, almost a fear. It is as if someone has threatened the existence of manual or human testing.

I do agree that there has been a general drive towards more automation of ‘tests’, and that this has been largely associated with the adoption of agile practices. I myself have encouraged, and in some cases demanded, more investment in, and thus more, automation of tests in companies I have worked for. However, I have also encouraged and hired for manual testing, and have coached and mentored folks to be better exploratory testers (what I call brain engaged testing).
So I don’t subscribe to the fear that manual testing is a thing of the past or an unnecessary overhead. Perhaps this is why I don’t share in the what seems to be an attempt at a sharp delineation between automation and testing?

Like Michael Bolton, I do see automation as a tool and as something that supports testing.
I often use the phrase automation assisted testing, to refer to exploratory or other manual testing where the test setup or initial test data has been achieved using automated tools or scripts.

My preference is to develop automation code in a re-usable fashion, producing a library of re-usable code that is easy to ‘glue’ together in different ways such that different automated tests (or checks if you will) are achievable quickly and efficiently. But this approach also lends itself well to re-using these library ‘functions’ to assist with manual testing. If developed well then anyone with fairly basic coding skills should be able to combine some of these together in order to ‘drive’ a system under test to the point where you want to start your exploration or manual testing. Or as mentioned before, to prime the system under test with the exact data you want or need, in order to conduct the exploratory or manual testing you wish to execute next.

My Agile QA Manifesto and Testing Principles

My Agile QA Manifesto

With reference to the original Agile manifesto I present my thoughts on an extension for agile QA or an agile testing manifesto;

  • Prevention over goalkeeping
  • Risk based test coverage over systematic test coverage
  • Tester skill over test detail
  • Automation over manual (for checking/repetition)

While there is value in the items on the right, I value the items on the left more

Testing Principles

And to follow that, a set of principles I try to follow and try to instill into those that work with me;

  • Fail fast/provide fast feedback
  • Test at the lowest layer
  • Test first (TDD/BDD)
  • Risk based testing for efficiency
  • Focus on tester skill and domain knowledge
  • Drive for automation for repeated checking (regression)
  • Learn from your mistakes – don’t repeat them

Layers of Test Automation

Also referred to as the Test Automation Pyramid

The intention of this post is to get across the idea that your testing strategy should include many layers of testing.

I am talking mostly about automation here and will for the purposes of this post I will ignore the discussion around testing vs checking when it comes to automation, and therefore will continue to use the common terms; tests and test automation.

My first introduction to the formal concept of the ideal test automation pyramid was courtesy of Mike Cohn of Mountain Goat Software (I read his blog post on this many years ago).
The idea he discussed resonated so well with me that I have been trying to follow this strategy ever since. Of course I have experienced a few different companies with very different shapes to their automated testing. I intend to share some of those experiences with you, along with some ideas for how to adjust your strategy in each of those cases, and of course to help you avoid the mistake that Mike was referring to of forgetting about the middle layer.

The test automation pyramid concept has been adopted quite broadly and adapted for many different scenarios too. But it is definitely not a silver bullet and there are times when this approach is not appropriate for your environment, technology or simply the way you work.
That said, most of the companies, technology stacks and teams that I have worked with can and have benefited from this strategy.

So, what is it?
Well here is the most basic version of the pyramid that I typically draw on a whiteboard;


One of the variants that I will often draw, when I feel the need to point out that we still need to do manual testing, (preferably exploratory), is shown below. Because this manual testing is somewhat variable in size or content I add it as a cloud to the top of the pyramid. There are many others who use this style (I don’t claim to have been the first but I cannot remember where I initially saw this in order to provide appropriate credit).

Test Pyramid with Manual Test Cloud

But the variant I use most often is one where I split the integration section in two, and talk about code component integration and system component integration;

Test Pyramid with 2 Integration Layers

Having done that I feel that I really need to explain my layers more clearly;

Unit tests – tests that are designed to ensure the smallest divisible pieces of code (units or components) are working the way they were intended. These are typically written by developers (though I encourage QA folks with development skills to at least review if not write some of them). They are typically written to make use of a unit test framework. They are often written after the code that they are intended to test is written, though in most cases I would prefer them to be written first (in a TDD manner). They should be executable by a developer at any time and are typically the first tests run in a CI system (Continuous Integration System).
A web based application may have unit tests in more than one code base, for example you may have Javascript Unit Tests in addition to those in the back end or server side code or even API code.

Integration tests at the code component level – tests that are designed to ensure that the code units or code components that need to work with each other (one calls another, passes data onto another etc), do so in the expected way(s). These are typically written by developers (though again I encourage QA folks with development skills to review and perhaps add tests here too). These will also often make use of a unit test framework but will be typically run after the unit tests have run (and passed).

Integration tests at the system component level – tests that are designed to ensure that the system components that need to interact with each other can do so as intended. These may be written either by developers or QA folks with programming skills. These tests will be designed and executed against APIs or Windows services or any interfaces exposed between system components. Sometimes you may have 3rd party services or components involved in this layer, for example we are currently using some cloud based services in our application. Often the UI will be built on top of an API, and by focusing on testing at this layer you can more efficiently and more robustly test the variations and permutations of API calls. Thus providing a solid, (well tested or checked), API layer upon which to provide a much smaller set of UI tests, as these will just need to prove that the UI interacts as expected with all the code layers below, and that in turn they all interact together well, (you will have covered the broad variations in this in the layer below too). These tests will need to be run against a deployed build in the CI pipeline, as these will typically need the application to be installed/deployed in an environment similar to way it will be delivered in production. As such these are normally run after the code component integration tests have run and passed.

UI tests – tests that are designed to ensure the user interface works in the way that was intended. Keep in mind that the user interface is not necessarily a web page or a GUI, it could just as easily be a command line interface to a tool. Typically though we are talking about a web based UI or a GUI of some kind. Test automation at this layer is often expensive both to produce and to maintain over time. So the focus here should be to minimise these automated tests by relying on and building on the successes of the testing in the layers below. Focus here on simple end to end workflow through the UI, and ensure your tests focus only on the sections of the UI that you want to prove are working well. In other words utilise lower levels of testing to prime the system under test with appropriate test data etc. For example: using the API test code to enter test data or get the system into a certain state that you need to start testing a UI workflow from. These are normally the last tests run in the CI system and sometimes are not run in a continuous way at all. For example if your UI tests take 4 hours to run then you won’t usually be able to run them on every check in and will instead need to consider running them periodically say once or twice a day.
(We can talk about opportunities to reduce this time later but the best one is to simply reduce the number of tests you need to run at this level by ensuring you have most of the coverage you need in lower levels).

So, why are the layers ordered and sized the way they are?
Well, I typically think of the width of each layer being the number of tests. This provides a relatively easy way to measure to see if you are approximating the right shape. As with most metrics I would caution you using this too strictly as really you just want to see that you are trending the right way or are in a position to discuss why not, (and perhaps understand that you have valid reasons).
The reason they are layered in this order is really the building analogy, where the bottom layer of unit tests is really forming the foundational layer of tests for the rest to be built on. You want a very broad bottom layer (a large numbers of unit tests as the scope of each test is very small but the permutations and variations you need to cover may be broad).
As you move up the pyramid you will need less tests as the type of test increases in scope (covers more with one test) and because you don’t need to cover all permutations or variations as most should have been covered in the layer below.
You will have noticed in the definitions that I mention CI systems and when the tests will typically run. This is following the same pattern, you will only run the tests of a higher layer once the tests that are providing a foundation for that layer have run and passed. If there are failures you typically want to stop and resolve those issues before moving on.
It is also worth mentioning that the lower the layer the ‘cheaper’ the testing is, e.g. unit tests are typically quick to write, and very fast to run. So having lots of permutations or variations of say data or parameters tested at this layer is relatively cheap and easy, (much cheaper than at the UI layer). Thus this layer can provide a very solid foundation for the higher layers where you may only need to test one or two permutations or variations of data or parameters as you will know that the rest have already been covered and that the higher level test is more focused on proving interaction between parts of the system or the system as a whole.

One of my previous colleagues (Caroline), always preferred to think of the layers of testing as layers of a multi-tiered cake, like a wedding cake.
I prefer the pyramid shape myself so I continue to use that as my illustration.

Here are some of the shapes I have experienced and some approaches we have used to improve the situation:

Single Layer

That said One company I worked at did not really have a pyramid at all, it was more like a unit test cake with a manual smoke test cherry on top
This was a very developer heavy company where developers were expected to deliver production ready code, so they were expected to test their own code. Which typically meant they wrote unit tests and not much more.
If the code compiled and could be installed then it was largely assumed to be good.
The unit testing was not, in my humble opinion, great or consistently applied. The usual patterns and problems of some developers doing a better job than others and no or very little measurement of coverage.
The tests were also typically written after the code (so not TDD), meaning that the tests typically just confirm that the code does what the developer wrote the code to do, and are not trying to ensure that the solution in code is a robust one that will handle interesting or unusual cases appropriately.

If you find yourself in this situation and you have quality problems, (if this is working for you then no need to fix it), then I would suggest you try to find examples of product failures that are as a result of failures in system component level integration or code component level integration. For example an API that was accepting invalid input from the UI and failing as a result. Use these to encourage the developers to add integration tests, by helping them to understand the missing tests (the ones that could have exposed these issues early).
You will also need to seek management support to ensure new code written has code and system component level tests delivered with it as well as the unit tests. It should be fairly easy to monitor and show that this is happening and provide feedback on some of the issues these extra tests are exposing.
Once you start seeing automated tests running and passing at the code and system component levels you can then start to add UI level tests (probably best to start by automating those smoke tests).

Inverted pyramid

A common scenario, (in my experience, and the experiences that others have shared with me), is an upside down or inverted pyramid, where the testers have focused on adding automation at the UI layer, with very little being done at the lower layers. There may have been some automation focused on service or API layers. The developers have not been encouraged or managed to producing much in the way of unit tests so this is the smallest of all the layers.
Sometimes this happens when an organisation purchases an expensive test automation tool and wants to see a return on that investment, so focuses or manages the team to that, resulting in lots of UI centric automated tests.

Inverted Pyramid

The way to ‘flip’ the pyramid in this situation is to set the expectation that all new code needs to have unit tests delivered along with it, and any existing/legacy code that is changed should also have unit tests added (where this is possible and cheap enough to do – code that was not written with unit tests in mind can sometimes be very hard to add unit tests for. If new code is written using a TDD approach or at least with the intention that unit tests need to be delivered too then it will be more ‘unit testable’ by design). Again you will need management support or buy in for this, as some may question the value of the extra time or investment required in providing these tests. Try to find some existing issues that could have been easily and cheaply exposed at this layer, or pay attention to those that are exposed by your new tests and celebrate them. Assuming, that you see unit tests being added and passing, then you can start to encourage code and system component level tests by looking at important interactions in both those layers and focusing on those first (critical components at both code and system levels). You should also look at your UI tests and see if these can start to be refactored to either use more API or service level integration or perhaps even be replaced by tests at that layer.

Trapezoidal pyramid

An interesting variation is one that I call trapezoidal, since it feels like there are two trapezoidal sections of tests with a thin and narrow band of integration tests in between, in extreme cases perhaps none at all. So this is really depicting a reasonable amount of unit tests along with a focus on UI tests and very few integration tests of any sort. This, I feel, is the very problem that Mike was focused on with his original blog post, and it is a shame that this is still a pattern we can see today.

Trapezoidal Pyramid

For this company it seemed the automated testing was divided between dev and QA in terms of, “is it a unit test?” then dev will provide that, anything else must be a QA test and that typically means testing as the customer so a UI test.

There are many reasons why we seem to ignore the integration tests, here are a couple of the most common ones I have witnessed;

  1. How many people can adequately understand, and thus define or explain what an integration test is? (In my experience not many).
  2. Even if the team does have a good understanding it seems to be something of a “no man’s land” in that it is not always clear who should own it so it simply doesn’t get owned and thus doesn’t get done

So in order to combat this shape, the team(s) really need to focus on adding integration tests, both at the code component and the system component levels. This will again require investment and support, so as before try to identify the critical code components and system components and focus your efforts on these first, or simply start applying this with all new code and only tackle existing code if it is changing significantly. Once you see these system and code components being covered more efficiently and effectively using integration tests you can probably reduce the number of UI test variations that interact with these system and code components. It may be possible to divide the efforts here neatly between QA folks with programming experience who can tackle and get the benefit of a greater understanding of the system component integration points, and the developers who can more readily identify and develop tests for the critical code components.
Make sure your tests are identifiable in some way so that progress can be shown and measured and that issues found can be attributed to the appropriate layer in which they were found so that these successes can be shared and celebrated, providing validation of the efforts it took to add these.

Fork And Ambush

Fork and ambush is my short way of describing the outcome of non-collaborative work on a story or the implementation of a requirement. I used to see this all the time in more waterfall like environments, but I am sad to say I still see and hear of this in more agile environments too.

The scenario is as follows;

Someone, usually the customer proxy, in my current company this is a Product Manager, provides a requirement, in our case a story along the lines of “As a …. I would like … so that I can …”
The Developer would then take this story and go back to his or her desk and start developing the code to deliver the functionality for the story.
The QA or tester would take this story and go back to his or her desk and start thinking about test cases that should be executed against the story.

This is the fork (both going their separate ways to think about the story individually)

Some time later …

When the story is developed the developer notifies the tester and testing begins.

This is effectively the ‘ambush’ part.

The tester is often trying to find fault with the developer by hunting for bugs in the developed code.
Often a bug turns out to be a difference of interpretation (of the story) between the developer and the tester


In the worst case the customer proxy (e.g. Product Manager), comes along to diffuse the argument and informs them both that they are both wrong and what has been delivered is not what was required and the tests are also incorrect.

So, how can this be better?

Using a more BDD like approach, where the three amigos (Developer, Tester and Product Manager) discuss the requirement first.
Making sure each of them understands what is required (scope of work), who it is for (who the customer is), and why it is important to them.
They confirm this understanding by defining collaboratively the set of tests that will be used to prove what has been delivered is acceptable to the customer (what they needed and working in the way they need it to).
Then if the developer and tester fork at all then they both have a clear understanding of what is required.
The developer can ensure the developed code passes the acceptance tests.
The tester can ensure that in addition to passing the acceptance tests the developed code does not do anything unexpected and conforms to any non functional requirements that may have also been discussed etc.

The conclusion then is more likely to be a successful delivery as the work would not be accepted if the acceptance tests do not pass, and the developer and tester are on the same page as the Product Manager and can all see how they can work together towards the common goal.

Canary Fail

canary in a coal mineI was looking back over some notes from previous positions and I came across an instance where I had experienced what I called a Canary Fail. I am sharing this in the hope that some of you can learn from this rather than having to experience this for yourselves first hand.

This was quite simply a couple of situations where the customer found issues I knew were there but had not yet raised them with my team. I have always maintained that one of the primary purposes of a tester, in the context of discovering and providing information about the quality of the product, is to be the canary in the coal mine. In other words to be the early warning system, to raise the alarm to avert disaster.

Why is it then that I failed in this regard? Well, there are no excuses, I was wrong and I really should have known better. This was a lesson well learnt! Of course there are plenty of contributing factors that conspired to help me make poor decisions, and I am sure these will be familiar to at least some of you;

  • The release was time bound.
  • I had more areas to cover than I had time for.
  • We were creating release candidates, and thus introducing new changes and risks, quicker than I could mitigate (test) them.

The root cause of my canary fail was that I held off on reporting these issues because I was concerned I was wrong, that my understanding and thus my testing was flawed in some way and this would lead to me crying wolf! This in turn would result in me losing credibility as a tester and thus weaken my ability to advocate for defect fixes.

So the lesson I learned painfully was to raise a flag early and to talk through my concerns openly. Then to trust in the professionalism of my colleagues to forgive me when I am mistaken but to treat each flag I raise with respect and not discount my concerns even if I cried wolf last time.


Behaviour Driven Development – An Introduction

This is a presentation given by me and Marc Karbowiak  at a local test meetup group called YVR Testing on the 2nd April 2014

PDF version of slides: BDD Intro


Disclaimers first, I am not an expert in Behaviour Driven Design (BDD), in fact I am just starting down this particular learning path. I have however been testing, and to a lesser extent, automating tests for many years. So I have learnt enough to know that this approach is a great one to try, as I can see how it will help to address many of the issues we experience. In particular it will clearly help prevent the types of issues that arise from misunderstandings, assumptions and ambiguity in our requirements.
Marc and I wanted to share some of our early experiences and those of other and better folks that precede us in learning BDD, as we feel strongly enough about this approach that we want to encourage and inspire others to learn and adopt BDD practices.

(slide 2) Setting the scene

We have probably all experienced ambiguous requirements, which means we have probably all experienced problems in our products as a result.
Some simple examples;

  • Product Manager (PM) & Interaction Designer (IxD) require a text box to have a 100 character limit.
  • Test automator leaves the requirements discussion, goes back to his desk and writes tests that will drive the UI to write one character and assert that the UI shows character count of 1
  • Developer leaves the same discussion, goes back to his desk and writes code which will show 100 characters and count down every time a character is entered.
  • The test fails and then a discussion ensues to figure out which one understood the requirement appropriately.

I have experienced many of these types of situations, quite often where the PM is saying that neither the tester or the developer understood them correctly. In other words that both developer and tester misunderstood or mis-interpreted the requirements, and now both need to go and re-do or refactor their work to deliver what the PM really wanted.

(slide 3) Are you often testing at the end of the cycle?

Perhaps you are in a waterfall like development lifecycle, or a fragile lifecycle?
Do the testers know ahead of time what they will test? Did they work that out from either a written requirements document or a requirements discussion at the beginning?

Or are the testers working with the developers, product managers and in our case interaction designers on a regular (daily?) basis to ensure we are always on the same page and on track to deliver what is really required?

(slide 4) Are you in an agile like environment?

In which case do you all speak the same language? Do you have a domain specific language that is understood and used by all?
I have been in many requirements discussions, story kickoffs or similar where it really seems like we are talking different languages.
In my current role we have a lot of domain specific terms which are either overused, (used to mean more than one thing depending on context), or terms that mean different things to different people. We recently had a problem where one team used the term ‘system variables’ to mean a specific kind of data we store about a member of one of our insight communities, another team wanted to use the same term to refer to data we capture and store about a survey respondent’s computer system (e.g. browser & version, screen resolution and browser locale)
As a test why not ask 10 different people what a test plan is and see if you get 10 different answers.

(slide 5) Deadline approaching?

Does this mean you usually cut a few corners, rush your work, ignore some aspects of your process that perhaps you don’t think provide good value for the time they take?
So, why don’t we have time to do it right but we have time to do it twice?
(I forget where I first heard that phrase but it is a really powerful one for me)
We often cut corners or rush to meet a deadline, knowing really that we will have to come back and ‘fix it up’ or pay down some ‘technical debt’ later. And of course the cost of that will be higher than the cost of doing it the first time round.

(slide 6) A well known illustration of;

a) What the senior developer/designed designed
b) What got delivered
c) How it was installed at the customer site
d) What the customer really wanted

This is typically the result of some of the ways we have been working and the approaches we take.

(slide 7) So how can we be better?

We can adopt the test first approach – combine the red-green-refactor pattern of TDD with behaviours to get BDD
Defining a test first and then writing the code to pass that test provides many benefits;

  • Only write the code needed to pass the test (no waste)
  • Ensures code is testable – cannot pass test otherwise – requires observability etc
  • Test effectively documents the code
  • Safety net – test is added to CI system ensuring we never regress this code (if we do the test fails) – future code refactoring is done with confidence as we will know, as quickly as it takes to run these tests, if we have made a mistake

(slide 8) Where do I start?

Another way to describe the test first pattern is the Acceptance Test Driven Development approach.
This fantastic diagram is borrowed from (add refs and links)
The fours Ds
Follows the TDD red green refactor cycle as shown in the middle, i.e. this is iterative

(slide 9) Discuss

Ensure you have representation from all of the key roles to discuss the story or item of work. Typically we refer to the 3 amigos – Product Management (or Business Analyst or if possible the customer), Development and QA. At Vision Critical we also often have a 4th amigo – an interaction designer (IxD)
The discussion needs to unify the language, ensuring we are all discussing the same thing and have common and shared understanding
For example ensure we don’t mix development terms with product or customer terms, try to define a domain specific language (DSL) that we can all share and understand.

(slide 10) Distill

Discussion should then produce examples of the behaviour you want from the product or system. These examples are effectively how you will test you have developed what was really needed.
Use a ubiquitous language and structure to define these – Given – When – Then
Facilitates clear communication as well as structure that is easy to read and simple to follow

(slide 11) Develop

First develop the automation that asserts the behaviours (automate the tests first)
Then develop the code (production code) to pass those tests

(slide 12) Demo

Demonstrate the working code using the automated tests
Review the behaviour specifications with the customer or product manager
Add the tests to your CI system (keep running the tests to ensure fast feedback and a consistent safety net of tests)
Time to celebrate and perhaps retrospect on the story and capture anything you learnt and ideas for improvement so that you can apply all to the next story

(slide 13) Repeat the cycle for all the stories, learning an improving as you go.

(slide 14) Results

Hopefully you will have delivered what the customer really wanted and gained some additional benefits;

  • Executable specifications that can always be trusted to be true (otherwise your tests will be failing)
  • Automated regression tests that provide a safety net and fast feedback on those regressions – try to have these tests run as frequently as possible
  • Testable and thus maintainable code, not only can you look back at these tests in 6 months and know how the code works, but you also know the code is testable as it was written to pass tests

You will hopefully also feel proud of what you have achieved and will be recognised for that – if you are the only team doing this is will show!

(slide 15) What BDD is not

This all sounds great, so where do I get hold of this silver bullet or pink glittery unicorn?
Well BDD is not one of those, in fact it takes a lot of work to do it well, but it is worth it

(slide 16) How to get started?

There are a number of different BDD frameworks for the mainstream development languages, here are a few

  • SpecFlow is for .Net
  • Cucumber is mostly for Ruby
  • JBehave is for Java
  • Behat is for PHP

(slide 17) Gherkin anyone?

As well as being a pickled cucumber …
This provides the common and ubiquitous language that facilitates the simple and clear communication of behaviour
Here is an example of a feature, which contains a number of scenarios (tests for that feature)
The feature description here describes the feature and the context, in this case the problem the feature is trying to solve

(slide 18) Background

Background is a special keyword in Gherkin
In this case I am showing an example from the Cucumber Book that uses the background to setup the test preconditions – the Given for the following scenario(s)

(slide 19) Scenario

Here are the When and Then sections of this scenario
Very readable, understandable and clear

(slide 20) A question of style

A lot of folks (myself included) start writing scenarios in a similar way to how we would write test cases or code, by detailing all the steps that we need to execute in order to setup the product under test as well as perform the test and check the results.
This is an imperative style and it is not very readable, at least not when you are testing something more trivial than adding two whole numbers.
So we need to focus on a more declarative style, try to tell the story of the behaviours we want the product to have
This means hiding all of the details that are not relevant to the behaviour and keeping only the details that are important to the behaviour or the intent of the test

(slide 21) An imperative example

Lots of inconsequential detail here which means that the intent of the test is lost in the noise. We specify the email address and password but have to assume this means these are valid. Do we really need to know we clicked the login button? How does that help us understand if the product behaves correctly when we provide valid login credentials?

(slide 22) A declarative example

Hopefully you can all see this is much more readable and very clearly talks about what is important to the behaviour
This test is not about what makes a valid or invalid email address or password, it is about what happens when those are valid and the user is able to successfully login

(slide 23) DRY vs DAMP

Aim to tell a story rather than focusing on re-usability
An example here would be that we have steps like those in the imperative example;

Given I am on the login page
When I enter email as “example.user@mybiz.com”
And password as “Password1”
And I click the login button

Because these steps detail how I login we can easily re-use the same steps throughout the scenarios to login before executing more steps that are designed to assert a new behaviour, for example;

Given I am on the login page
When I enter email as “example.user@mybiz.com”
And password as “Password1”
And I click the login button
And I click the Start New Project button on the dashboard page
Then I should see a blank project
And I should be able to edit the project

Instead of re-using these steps this could have been written as;

Given I am on the dashboard page (it is not important for this behaviour to know what steps you took to login or what exact credentials you used)
When I start a new project
Then I should be able to edit my new project

(slide 24) Scenario Table Example

Using data tables to test with multiple values that will not read well if all written on one line of a Given, When or Then
These enhance readability by keeping the data clear but separate from the declarative and meaningful phrases

(slide 26) Scenario Outline Example

Sometimes you need to effectively test using the same steps but with a variety of test inputs, a scenario outline helps to avoid repeating steps each with different data values
In this case each row of the table is essentially one Given – When – Then scenario and will be executed sequentially

(slide 28) Hooks

These are really useful, they can simply call some code to setup or tear down your tests and are controlled using methods called Before and After

(slide 29) Tags

Use these to label your features and scenarios within features
You can then execute only those features or scenarios that are tagged a certain way
Or filter out tests with a different tag
We use tags to group tests by team, to run certain subsets of tests in a certain environment, and now we are trying to use tags as a way of recording and reporting test coverage by labelling features and scenarios by the code area that they cover

And that wraps it up for the presentation part.

Test Coverage Outlines

This is a presentation I gave at a local test meet up group called VanQ

PDF version of slides: Test Coverage Outlines


Test Coverage Outlines are just elaborate spreadsheets. They were originally inspired by Jonathan Kohl, as part of exploratory testing training at Sophos. A colleague, (Jose Artiga), and I built on the initial inspiration and came up with the first version of the test coverage outline. This was then adopted by the teams at Sophos, and of course refined through use and practice.

(slide 8) Simple spreadsheets to help;

  • Structure & simplify your test planning
  • Inspire & structure your test design
  • Be test professionals and not just test execution machines
  • Collaborate on testing
  • Show clear and concise status at any point in your testing
  • Discuss risk in terms of planned vs actual coverage

(slide 9) An example coverage outline

  • Each white background row is a test idea – a simple sentence or few words to convey an idea you want to cover by testing
  • Try to keep these simple like a heuristic that acts as a guide rather than a step by step procedure

(slide 10) Divide your test ideas up into sections

  • Example shows this as the black lines with white text
  • You can do this by functional area, test focus, heuristic etc (whatever makes sense for you in your context)

(slide 11) Divide your test ideas up into sheets

  • Again divide into functional area per sheet or perhaps copy your sheet to provide same/similar coverage against different builds or versions of your software

(slide 12) Configurations – show your coverage against different configurations

  • Example given is a common one – browsers and OS, but could equally be versions of HW, SW, Mobile device etc

(slide 13) Test types/levels – show your coverage at different test levels or different testing types

  • Matrix your test ideas against different test levels, for example unit tests, integration tests, automated UI tests, manual tests, exploratory tests
  • Even if you are not familiar with the unit tests you can suggest the developers gain some coverage of the test ideas with unit tests and fill out the column with you

(slide 14) Prioritise your test ideas

  • I use simple (high, medium, low) priority levels to organise, sort and filter test ideas by priority
  • Make sure you are executing/covering in priority order

(slide 15) Prioritise your configurations

  • I use a simple left to right prioritisation for configurations, i.e. the highest priority configuration to cover is the left most column

(slides 16 & 17) Use colour and conditional formatting

  • I use simple colours and conditional formatting to make updates easy and to show status and priority clearly
  • Make sure you also use a colour (I use x and grey) to indicate coverage that you are consciously not planning to cover

(slide 18) I use mind maps as a visual test design/planning aid

(slide 19) Start by outlining your test idea areas, use the outline to inspire your test ideas

  • I sometimes use heuristics to structure and inspire my testing, for example using a quality criteria heuristic like usability or performance I can think about test ideas that come under each of those ares
  • Creating templates (see examples) for common types of testing focus can help inspire testers as well as providing some base or consistent tests

(slide 20) Using test ideas rather than detailed test cases

  • enables and encourages variation, exploration and more ‘brain engaged’ testing
  • thus avoiding the pesticide paradox [Boris Bezier]

(slide 21) Collaboration

  • Using google docs or similar enables you to easily collaborate in real time with other testers/colleagues
  • Some of the easiest ways in which you can organise and facilitate collaboration is to allocate a config column, test area or even tabs
  • You can see who is accessing/updating the sheet and can see status of tests others are covering in real time too

(slide 22) Use colour to provide very easy and quick to read status

  • You can just look at a tab and immediately see if you have any fails or blocked tests

(slide 23) Colour coding also makes it easy to see coverage in terms of planned vs actual

Example Coverage Outlines

Feel free to copy and adapt these for you own purposes, hopefully these will inspire you to refine the outlines and share your ideas back with me and others;

Learning from the mistakes our customers care about

5 question marks

5 whys

As mentioned in a previous post I keep a close watch on customer defects. These are the issues that a customer cared about enough (or was sufficiently annoyed by) to contact us and tell us about them.
I am focusing on the issues here, not the feature requests or the how do I’s, though both can be also regarded as defects in ‘failing to understand or predict the customers needs’ or ‘failing to deliver an intuitive product’ respectively.

Being a big fan of prevention is better than cure, I like to investigate the customer issues and perform a root cause analysis or 5 whys on the reasons each issue escaped our attention.
Yes, I refer to customer reported issues as escaped defects, since they escaped our detection. It doesn’t matter how many stages in your pipeline or how may automated tests at different levels, or even how good the teams are, there will be some issues that escape our attention.
Technically, I also regard issues discovered late in our pipeline, after story acceptance and as part of our release process, as escapes too, as well as any issues we happen to find in production (before a customer reports them).

There are lots of quotes around learning from failure, and being doomed to repeat your mistakes if you fail to do so. I believe, along with many others, that true learning only comes from failure and understanding the reasons for it. However, we should take care to not make the same mistake twice as this indicates a failure to learn. So the reason for analysing these escaped defects is not to apportion blame or point fingers, it is instead to learn how we can prevent a similar class of issue in future. The preference here being to prevent the class of issue from ever being coded again. If that proves to be more expensive than the cost of impact of the issues then at least being able to prevent the class of issue escaping our attention again.

I was introduced to Lean software development techniques via Mary and Tom Poppendick  which led me to learn more about The Toyota Way where I learnt the 5 whys technique. Prior to this I had been using other root causing techniques or simply using my QA ability to ask difficult but relevant questions to achieve the learning and expose the actions.
The 5 whys technique is just so simple that it makes it easy for anyone to participate as well as facilitate, meaning that anyone can do this – you don’t need to be a QA or have a background in problem solving or root cause analysis techniques.

So, what does it look like? Well here is an example with some edits to remove any proprietary details (note 5 is simply a guide, you can use more or less whys);

Problem statement: Service proxy was updated in C# provider code but not in consumer code

Why didn’t we catch this?

  • tests run in the consumer pipeline were not sufficient to expose the issue
  • tests were not full contract tests – nothing testing the contract between producer and consumer
  • no communication between the producer and consumer teams on any changes made to the interface

Why didn’t the consumer pipeline tests expose the issue?

  • because the test only exercised the simplest possible scenario which did not get affected by the change (sec call returned minimum possible data)

Why were the contract tests insufficient?

  • contract testing is not very well understood by all teams concerned
  • tests were not reviewed by anyone except developers

Why wasn’t there communication between teams?

  • the producer of the service does not know who is consuming that service
  • tests didn’t relay information of endpoint changes
  • consumer tests were still passing (green)

Why didn’t the tests relay information of endpoint changes?

  • there were no tests asserting or checking the stability of the interface
  • the consumer was coded to de-serialise the entire response when really it only needed to check for ‘success’

Why was the consumer coded to de-serialise the entire response rather than just parse the value of interest?

  • because it was deemed easier to use a standard pattern to de-serialise entire response rather than write code to specifically look for just the value of interest

Some example actions that were taken as a result of this;

  1. Provide training in contract testing patterns
  2. Producer to add tests to notify producer team of interface change (trigger for investigation or communication of change)
  3. Consumer to provide contract tests for producer to run in producer pipeline to alert of breaking changes for consumer
  4. Audit all cross team interfaces/dependencies to negotiate and add any missing contract tests

The Streetlight effect



This phenomenon is called the streetlight effect. I often see this with test engineers, I have caught myself doing it! I find myself testing or exploring some area of functionality because I am familiar with that area, in other words I know how to test it. Or I catch myself spending most of my time focused on a certain type of testing or looking for a certain type of bug because I am comfortable in my skill, knowledge and experience in that type of testing – the light is better here!

Another way to characterise this is that you are staying in your comfort zone. So how do you avoid the streetlight effect or start to move out of your comfort zone?

Step 1 is recognising it. Try to catch yourself looking where the light is better. For example spot when you are testing or thinking about tests to run with a product and challenge yourself to test something you wouldn’t normally test or think about tests that you wouldn’t normally run. Even better challenge your colleagues to spot when you are doing it (and offer to do the same for them).

For example if you are the tester on an agile team and you usually test functionally from the UI, ask your colleagues to review your test cases (you are discussing them with the team anyway, right?), and challenge you if they are only testing the story or feature from the UI. Or if you are a test automator and are familiar with automating tests by driving the UI have your colleagues review your automation code (they are already doing that, right?), and challenge you to test at a lower level interface in the system instead.

Step 2 is to develop the ability and skills required to shed some light on the areas of darkness. Or put another way, to push yourself out of your comfort zone and into the learning zone where you will be exposed to new things and can therefore develop new skills and experiences which will broaden your circle of light.

learning_zoneSo if you are normally testing using the UI with a customer focus, try learning more about the back end code and try thinking about new tests using that new found knowledge, or try thinking about tests to run against an interface into a lower level of code.

I work with teams to broaden their test perspectives and to think about the product or area of code they are responsible for more systematically and holistically. Challenging them to think about what happens at all levels in the code and all the places where things could go wrong. I then help them think about different ways we can test or automate testing those areas that are most likely to be buggy. For example using test tools or automation to create test data, or inject it into a part of the system so we can effectively check how the system transforms or presents it later.

To be a great Quality Assurance engineer or a great tester you also need to be a continuous learner. Are you a genius already? Did you leave school, college or university with all the knowledge you would ever need?learn_more_alpha

There will always be something you can learn about the product you are helping to deliver with quality. Customers will teach you new ways in which they use the product (which can often surprise you). Your colleagues and other thought leaders in the domain and profession you work in will always provide you with new and different ways of looking at something or approaching what you do. So make sure you listen, learn and experiment, looking in places where the light is not so good.

So what will you learn next to increase your circle of light?