Beware the Streetlight effect in your feedback mechanisms

Be on the lookout for the Streetlight effect, especially when it comes to a choice between fast feedback and slower feedback.

I recently worked with a junior developer to add a simple test that could be run before code commit by any developer and as a test before the pull request was merged (if the developer forgot to run the tests in their branch before committing 🙂 ).

The intent being to provide fast and reliable feedback to whoever was making a change so that, if the test failed, then they were still in context and could very quickly and efficiently resolve the issue and commit passing code

I checked back on progress and found the junior developer had been advised to add the test to the CI/CD pipeline at the image build stage.

A senior developer, who would have been the one to help teach the junior developer how to add a test to the CI/CD pipeline, provided this advice.

Image build is the last stage of the pipeline. After several building stages, testing and packaging culminate in an image ready to be deployed into production.

Adding the test at this final stage would be trivial to add and run.

However, if the build failed at this stage, it could be hours or even days after the code change that caused this test to fail was made. The developer who made the offending code change would be in a different context now and would probably not be the person who sees the image build failure.

  • Now you have very slow feedback.
  • Another engineer has to diagnose/dig into the reason for the image build failure.
  • This engineer then needs to find who made the code change and track them down to correct it.
  • The offending code engineer may not even be available at that time.
  • Meanwhile, the build is stuck, and if you needed to get this build into production to fix a customer issue, you would be blocked and very unhappy!

So, when you have a choice between something hard (like finding the key you dropped in a dark place or adding a test to the earliest possible point of feedback) versus something easy (like looking for your dropped key under a streetlight or adding your test to the end of the pipeline). Be aware of your choice (not finding your key as this is not where you dropped it or getting your feedback very late and at a point when it will be expensive and more time consuming to fix).

streetlight effect cartoon

Author: Stuart Ashman

I am currently working as the Director of QA at Vision Critical a market research software and services company. I have been working in a variety of roles involving testing and quality assurance for over 20 years. I started off testing flight deck instruments and progressed through GSM network operations software, Unix Operating Systems and Lights Out Management Firmware, into Anti Virus and Anti-Spam software and HW appliances, finally spending a short period of time testing cloud provisioning and control software before entering into my current position.