Learning from the mistakes our customers care about

5 question marks

5 whys

As mentioned in a previous post I keep a close watch on customer defects. These are the issues that a customer cared about enough (or was sufficiently annoyed by) to contact us and tell us about them.
I am focusing on the issues here, not the feature requests or the how do I’s, though both can be also regarded as defects in ‘failing to understand or predict the customers needs’ or ‘failing to deliver an intuitive product’ respectively.

Being a big fan of prevention is better than cure, I like to investigate the customer issues and perform a root cause analysis or 5 whys on the reasons each issue escaped our attention.
Yes, I refer to customer reported issues as escaped defects, since they escaped our detection. It doesn’t matter how many stages in your pipeline or how may automated tests at different levels, or even how good the teams are, there will be some issues that escape our attention.
Technically, I also regard issues discovered late in our pipeline, after story acceptance and as part of our release process, as escapes too, as well as any issues we happen to find in production (before a customer reports them).

There are lots of quotes around learning from failure, and being doomed to repeat your mistakes if you fail to do so. I believe, along with many others, that true learning only comes from failure and understanding the reasons for it. However, we should take care to not make the same mistake twice as this indicates a failure to learn. So the reason for analysing these escaped defects is not to apportion blame or point fingers, it is instead to learn how we can prevent a similar class of issue in future. The preference here being to prevent the class of issue from ever being coded again. If that proves to be more expensive than the cost of impact of the issues then at least being able to prevent the class of issue escaping our attention again.

I was introduced to Lean software development techniques via Mary and Tom Poppendick  which led me to learn more about The Toyota Way where I learnt the 5 whys technique. Prior to this I had been using other root causing techniques or simply using my QA ability to ask difficult but relevant questions to achieve the learning and expose the actions.
The 5 whys technique is just so simple that it makes it easy for anyone to participate as well as facilitate, meaning that anyone can do this – you don’t need to be a QA or have a background in problem solving or root cause analysis techniques.

So, what does it look like? Well here is an example with some edits to remove any proprietary details (note 5 is simply a guide, you can use more or less whys);

Problem statement: Service proxy was updated in C# provider code but not in consumer code

Why didn’t we catch this?

  • tests run in the consumer pipeline were not sufficient to expose the issue
  • tests were not full contract tests – nothing testing the contract between producer and consumer
  • no communication between the producer and consumer teams on any changes made to the interface

Why didn’t the consumer pipeline tests expose the issue?

  • because the test only exercised the simplest possible scenario which did not get affected by the change (sec call returned minimum possible data)

Why were the contract tests insufficient?

  • contract testing is not very well understood by all teams concerned
  • tests were not reviewed by anyone except developers

Why wasn’t there communication between teams?

  • the producer of the service does not know who is consuming that service
  • tests didn’t relay information of endpoint changes
  • consumer tests were still passing (green)

Why didn’t the tests relay information of endpoint changes?

  • there were no tests asserting or checking the stability of the interface
  • the consumer was coded to de-serialise the entire response when really it only needed to check for ‘success’

Why was the consumer coded to de-serialise the entire response rather than just parse the value of interest?

  • because it was deemed easier to use a standard pattern to de-serialise entire response rather than write code to specifically look for just the value of interest

Some example actions that were taken as a result of this;

  1. Provide training in contract testing patterns
  2. Producer to add tests to notify producer team of interface change (trigger for investigation or communication of change)
  3. Consumer to provide contract tests for producer to run in producer pipeline to alert of breaking changes for consumer
  4. Audit all cross team interfaces/dependencies to negotiate and add any missing contract tests

The Streetlight effect



This phenomenon is called the streetlight effect. I often see this with test engineers, I have caught myself doing it! I find myself testing or exploring some area of functionality because I am familiar with that area, in other words I know how to test it. Or I catch myself spending most of my time focused on a certain type of testing or looking for a certain type of bug because I am comfortable in my skill, knowledge and experience in that type of testing – the light is better here!

Another way to characterise this is that you are staying in your comfort zone. So how do you avoid the streetlight effect or start to move out of your comfort zone?

Step 1 is recognising it. Try to catch yourself looking where the light is better. For example spot when you are testing or thinking about tests to run with a product and challenge yourself to test something you wouldn’t normally test or think about tests that you wouldn’t normally run. Even better challenge your colleagues to spot when you are doing it (and offer to do the same for them).

For example if you are the tester on an agile team and you usually test functionally from the UI, ask your colleagues to review your test cases (you are discussing them with the team anyway, right?), and challenge you if they are only testing the story or feature from the UI. Or if you are a test automator and are familiar with automating tests by driving the UI have your colleagues review your automation code (they are already doing that, right?), and challenge you to test at a lower level interface in the system instead.

Step 2 is to develop the ability and skills required to shed some light on the areas of darkness. Or put another way, to push yourself out of your comfort zone and into the learning zone where you will be exposed to new things and can therefore develop new skills and experiences which will broaden your circle of light.

learning_zoneSo if you are normally testing using the UI with a customer focus, try learning more about the back end code and try thinking about new tests using that new found knowledge, or try thinking about tests to run against an interface into a lower level of code.

I work with teams to broaden their test perspectives and to think about the product or area of code they are responsible for more systematically and holistically. Challenging them to think about what happens at all levels in the code and all the places where things could go wrong. I then help them think about different ways we can test or automate testing those areas that are most likely to be buggy. For example using test tools or automation to create test data, or inject it into a part of the system so we can effectively check how the system transforms or presents it later.

To be a great Quality Assurance engineer or a great tester you also need to be a continuous learner. Are you a genius already? Did you leave school, college or university with all the knowledge you would ever need?learn_more_alpha

There will always be something you can learn about the product you are helping to deliver with quality. Customers will teach you new ways in which they use the product (which can often surprise you). Your colleagues and other thought leaders in the domain and profession you work in will always provide you with new and different ways of looking at something or approaching what you do. So make sure you listen, learn and experiment, looking in places where the light is not so good.

So what will you learn next to increase your circle of light?