Just published a free mini-course to introduce folks to Behaviour Driven Development in 3 Easy Steps: http://tiny.cc/bddin3
With reference to my original post on The Streetlight Effect; I still see this effect today.
Recently, I was discussing the testing of a mobile app with an experienced QA, who is a very experienced black box tester. I am currently encouraging and coaching this tester to learn more about programming, code design, code architecture etc as I think any knowledge in these areas can make you a more effective and efficient tester.
This tester was seeing some problems when interacting with the mobile app UI and was going to raise a bug with the developer. I asked this tester if they were observing the mobile app. requests and responses to and from the backend API, i.e. the requests and data being sent and received either across wifi or cellular data between the mobile app. and our cloud-based backend server API. The tester was not sure how to look at that traffic. The streetlight effect – I will look here (the UI) for my keys (any issues) because I know how to look here (the UI). So I shared a how-to I had written years ago for Charles Proxy which will allow the tester to inspect the https requests and contents along with timings and more, by proxying all requests made from, and responses to, the mobile app. through his laptop. This enabled the tester to see the requests that were failing as well as the resulting error message – the tester was then able to raise a much more detailed bug and the developer was able to get straight to the problem and fix it quickly. (As opposed to a bug like, when I do this in the app, I don’t get the UI screen/data I was expecting to see displayed. Where the developer would have to first reproduce the problem, watch the traffic themselves to pinpoint the problem – taking much longer and possibly with some back and forth to clarify reproduction steps etc.)
Another example is another very talented tester who is also a strong coder so has great white box testing skills. This tester was digging into some performance issues. Trying to understand why a request for a large dataset from an API was taking so long (when the system was otherwise quiescent), and would often fail if there was any other activity (API GETs and POSTs to requests for data and to store data). We are using AWS and there is a myriad of tools and monitoring capabilities to learn and get your head around. This tester was able to extract the time taken to complete the request for data and plot this against the size of the data extracted. If you think about this visually, this tester is looking at this from a black box perspective, making a request knowing what was requested and extracting the start time of the request from the log, then extracting the completion/response time from the same log, then plotting this against the size of data returned. (The tester was increasing the data stored and thus retrieved between each test/request). In this case, being capable of white box testing and understanding the code and system architecture this tester knew that there were several key components involved in servicing this request, but was not observing any of them. In order to understand what is really going on, and in this case be able to pinpoint the parts of the system that was taking a long time to service the request and which ones would fail when there was any other activity. Most performance issues are as a result of some form of resource exhaustion, e.g. CPU, Memory, Input-Output (IO), threads, connections etc. So, we really want to be able to see how these resources are being consumed when we interact with the system, as this can lead us to understand that our CPU usage is spiking to 100% when we do something and thus cannot do more when more requests come in, or that our memory spikes up and never comes back down to the pre-request level once the request is complete – in other words some memory is not being freed leading to a resource leakage (we will exhaust this resource over time). In this case, learning how to observe the individual service docker container resources and the database resources will likely lead us fairly quickly to where the problem(s) or weak link(s) in our data (chain) are for the request we are making.
In conclusion, what we need to do more often is to ask questions like;
- What could I watch or monitor to see, in more detail, what is happening/going wrong?
- What is the data flow – what path does the data takes through our system and what components are involved?
- How is the system architected and how do all of the components communicate
- How can I observe the communications between the system components?
If any of these questions result in a “I don’t know” or similar, then ask your colleagues for help, you are likely to learn something new, even if that something is that your colleagues also don’t know the answers to some of these questions.
I have used this example as a workshop to introduce BDD to a wide variety of folks at different companies. I like this example as it is deceptively simple, everyone knows how to deliver a login page, right? The reality is that we all have different ideas about what should and shouldn’t be on a login page and how it should look etc. So it does serve as a simple but very illustrative example of how using a Behaviour Driven Design approach can really help to clarify requirements, and engage the thoughts, experiences, and knowledge of all the participants to ensure what you deliver will be what was really desired. Also, that it will be both testable and tested as the high-level acceptance tests are defined up front.
Introduce roles and abbreviations
PM – Product Manager, our proxy for the customer, bringing the ‘what the customer wants or needs’ definitions to the team
DL – Development Lead
IxD– Interaction Designer, bringing the UI look and feel, the usability and customer workflow understanding to the team. Helping to ensure we have a consistent style, content, and customer workflows.
QA– Quality Assurance person (either QAE or SET) who will be ensuring we deliver the story with high quality, building it right and building the right thing
Dev– Developer(s), responsible for the actual implementation of the story, the code that will provide the desired functionality.
Implementation team– typically composed of a developer and a QA person, but can include Interaction designer, development or QA pairs.
Amigos– the group of people required to analyse a story – typically the PM as the customer proxy and the implementation team
Introducing the Story
The BDD discussion begins
I am going to join a long list of people I consider thought leaders in our industry in a lament of how many people continue to see testing as a simple job, not a career, not a craft, not really requiring any skill at all.
But, like Lalitkumar Bhamare I felt I needed to talk about this, because …
I recently interviewed a candidate that had had two co-op experiences as a QA, he had been given requirements and design specs and asked to think of tests (purely black box) for these. He believed he was doing well and that the team was working efficiently when he could think of one or two cases in addition to the one obvious one to confirm the requirement was met.
He found bugs, 2 or 3 a day, he said with some pride.
It was obvious he did not think this was hard, required any skill, or was a career that he wanted, but he felt trapped and compelled to apply for a QA role as that was what he had experience in. He really wanted to be a developer.
I explained he could be in a very different environment where instead of being part of a waterfall like cycle like this;
- customer proxy passes requirements to the tester and developer,
- the developer develops code to meet the requirements,
- the tester thinks up a few test cases per requirement (he told us he came up with about 2.5 on average).
- The developer then hands code to the tester who then takes the code (product) and tests it by following the heavily prescribed test cases he just wrote. (Presumably without thinking much). Finds a bug or two, records that in the bug tracking system and moves onto the next test case.
- Then the game of ping pong begins as the tester throws the code (product) back to the developer to fix the bugs. The developer fixes the bugs and throws it back …
That instead, he could use his skill, (yes I did say the word skill), to help the customer proxy and the developer understand what he was thinking of testing before the coding started. That this might actually prevent the developer from missing the requirement to meet the other use case the tester had thought of. That the customer proxy may have an opinion on this and say, “no we don’t want that”, or perhaps, “yes we want to meet that need too, and now you have made me think of another one that we need to meet too!” (Catching a completely missing requirement)
He didn’t see it, couldn’t see himself doing that, couldn’t see a way that could work, at least not during the interview. To his credit he was thinking about it and it clearly confused him but he was not able to take this any further.
(I did also suggest that there was no reason for him not to apply for software development roles)
Many, many, others I have interviewed, some who have been ‘testing’, (I would say checking), for years, still have no thought that they could be better, that they could improve and develop their skills, that they could really help their team to avoid making so many costly mistakes and save their companies thousands if not millions in wasted efforts, time lost due to bug interrupts and context switches, re-builds, re-tests, whack-a-mole etc. In other words deliver real value and not just fill a role, warm a seat, expend some effort, etc.
A co-op student I worked with, (believe me I am not picking on co-ops at all, they have very little experience to draw on, but are also often not taught anything good about testing), visibly turned his nose up at doing some testing, saying, “I don’t want to just press buttons all day long”. This was his understanding and belief of what testing was. Not entirely sure what his experience was or where he got that, but I tried to disabuse him of that incorrect understanding of the role, but once again felt myself working against an entrenched idea.
So, I call out to the learning establishments to do a better job of preparing these young minds for the real work of professional software engineering. That good software developers do a lot of thinking and a lot of testing as part of everything they do, and that doing this well takes a lot of skill, experience and learning. And that good software engineering teams and leaders recognize the need for people on their team that think differently and are skilled in the art of helping the team expose issues as early and as quickly as possible, to help them deliver the right solution first time more of the time. Without the need for drama, missed deadlines, long hours, late nights, re-design, context switching to fix bugs found in code developed weeks ago, etc. That this, (software testing), is a career, a valuable role, and an essential part of any high performance team. That it also takes a lot of skill, experience and learning to do it well.
I also call out to all the companies out there that insist on continuing to think of testing as simply, ‘using the product the way a customer will’, and hoping that will catch enough issues. Believing that in doing this they are ‘testing the quality into the product’! They are both wasting and losing a lot of money, as they have to pay the cost of finding defects late in their cycle. They also need large customer support teams, and in some cases teams of developers focused on maintaining, (fixing), their colleagues code.
These companies are responsible for perpetuating the myth that testing is basic, unskilled and a simple role. That it is easy to automate what these testers do or that we can simply outsource this work to countries where peoples’ time is cheaper.
They are also often responsible for poor quality products, that will often miss deadlines, and require expensive maintenance programs or extensive and expensive re-writes every few years.
And finally I call out to all the people in testing or quality related roles out there to learn to do better, be better and thus change the perceptions of everyone around them to see that testing is a skilled role, it is a craft and thus requires craftsmanship, and that it needs to be part of every team and will improve the quality of everything we do.
But it’s not all bad …
I have employed and had the pleasure of working with some fantastically skilled testers, both talented exploratory testers, and test automators. Sometimes a few rare gems that are able to combine great skill in both testing and the development of tools and automation to help check software.
I have also had the great pleasure of working with lots of great software engineers and leaders who really understand and seek the value of testing. Insisting that they have great testers to work with them, that they and their software developer colleagues need to think about and execute testing at all levels and that they encourage all to learn more about and be better at ensuring the quality of everything we do.
I have been inspired, guided and encouraged in my career by many people, but most of this has come from what I think are some of the thought leaders in this area. Here are some links to articles, books and people that can help you too;
I recently got asked what I look for when recruiting for QA/test positions. So I decided I should share my thoughts both with those of my colleagues who are recruiting, and also those who are looking and maybe wondering what they need to be able to share in an interview.
I have broken this out into a small number of attributes or qualities I tend to focus on:
Passion for testing?
- ask questions about why/how did they got into a QA role or what keeps them in a QA role
- looking for someone who cares about quality and recognizes that it takes skill, that they have some of that and they enjoy using it
- ask them to tell you about an interesting bug they found and why it is interesting (was is challenging to expose, challenging for the team to resolve, an interesting chain of events type issue … is there genuine interest and passion in finding and solving problems here)
- what do they enjoy most about their current role?
- why do they want to leave?
- what does this tell me about what motivates them and what they may be looking for?
- do they understand the basic test design techniques of equivalence class partitioning and boundary value analysis – keeping in mind they may not know the terms, they may just do this intuitively
- ask example questions that have clear equivalence classes and boundary values or get them to explain how they would think about the test cases for a situation like this
- ask why they think of testing with those values to see if they have ever identified that they are applying a technique or a pattern to these types of problems
- you can also ask other basic questions like what are some of the important points to include in a good defect report, what they do when faced with ambiguous or unclear requirements
- do they know any test heuristics?
- do they perform exploratory testing? Can they describe what they do or give you an example? (Are they truly exploring or are they just doing ad-hoc and thinking they are the same?)
- how big is their team – if they are the only QA then they probably don’t have anyone to bounce ideas off or tell them there is another/better way of doing something
- are they all in the same office? – if yes how often are they talking with others on the team and why
- if some/all are remote – how do they communicate and what problems do they have communicating
- do they try to help developers resolve issues – ask them if they attempt to narrow down where, how or why a bug is occurring
- do they try to help PM/BA define requirements – ask them how the requirements analysis/review happens and how they provide feedback or ask questions to clarify requirements
- you are watching for any team vs us type attitudes or ‘not my role/job’ type attitudes
- do they understand their current process? Can they describe it briefly and succinctly? – if not then they probably don’t understand it well enough to explain it clearly to a new teammate, or it is too complicated for them to efficiently work with and they don’t know how to improve it
- do they do regression testing
- do they get to choose what testing is done and in what order?
- if yes, what do they do first and why?
- if not, then do they understand how the decisions on what to test and in what order are made?
- what happens when they find a bug in something they are testing? Do they stop and wait for a fix or pick up something else?
- if they could change one thing about their current process in order to improve it, what would they change and why?
Automation – only applicable if they have some automation experience
- what do they automate and why do they focus on automating that testing? Often get simple answers here varying from I automate what I am given/told to automate through to I automate everything – you are trying to understand if they know that there is judgement required here – choices to be made about what to automate in order to improve the efficiency of the team
- also common that QA automation is focused on UI – ask them if they have any experience of automating anything other than UI
- do they run their automation regularly? In a CI system?
- what problems have they had with automation – digging here to understand if they have built fragile automation or if they have learnt the basic mistakes and are experiencing more advanced problems like how to efficiently manage test data or parallelize tests more effectively
- ask them how they keep their skills current
- ask them to tell you about something they have learned recently that they found particularly interesting or useful
- ask them if they follow any QA/test/development folks blogs or twitter feeds and what they learn from that if they do
Why don’t we have time to do it right, but somehow we do have time to do it twice?
My paraphrase of a quote by John Wooden (quote number 5)
How come we often think it is better to rush into something we don’t understand and hack at it than to take the time to understand what it is we are really trying to achieve and then think about how we will achieve that before we start coding?
Do we not learn from our mistakes?
Do we not see, measure or understand the cost of halting a developer, who is by now working on another story, to get them to switch context and think back to the rushed coding they did on the previous story. To get them to diagnose the root cause of the bug we just discovered? Or see, measure or understand the time it takes to diagnose, resolve then rebuild and re-test this change through the entire pipeline? With a very real risk that when handed back to the tester that she will find another issue as the fix was rushed and we did not have enough coverage in our pipeline of automated checks to discover the regression that was introduced.
I have seen this pattern throughout my life and have been guilty of it myself. So what do I do about it?
Well I try to discipline myself, but what I do for my teams is to use BDD to ensure we have shared and common understanding of what story we are about to do, changes we are about to make, additions we are introducing. To ensure we all understand who needs these changes and why they are important to them. (Who the customer is and why they care). Then we, (the three amigos – slide 9), will be able to agree some high level examples of what ‘done’ looks like for this piece of work. These examples will be in the the form of tests that will adequately specify and thus prove we have delivered what was required. We call these the acceptance tests. They are defined before any coding is started. Hence the ‘driven development’ part of BDD. Where possible these tests are automated and are required to be passed, (via automation or manual checking), before the story is pulled by QA, (we use Kanban), to do a final exploratory test.
We are not perfect at this, but it really does mean that we can get a story accepted more often than not on its first pass through the pipeline. If it doesn’t we all understand the work well enough to learn from the mistake(s) and improve.
Following on from the UNUSUAL PAGE post, I have also created a heuristic for REST APIs, along with a simple mnemonic, which I think is quite memorable for a certain group of sci-fi fans 😃
My organisation is currently implementing an API first strategy, whereby we design and implement the API for any piece of functionality before developing any UI or consumer code for that interface. This provides us with the ability to separate concerns easily, improves testability and is in line with the current trend for micro services.
As with the UNUSUAL PAGE mnemonic I realised that the original heuristic was not that memorable and thus my team were not able to easily call it to mind when in a meeting room, designing the next REST API with their team.
So, with a bit of rephrasing I came up with VADER, (Verbs, Authorization, Data, Errors, Responsiveness).
As with the previous heuristic, I have updated the coverage outlines templates originally described and linked in a previous post.
Obviously not all of these branches or leaves will be applicable to your REST API and your context, and indeed the words I use here may mean different things to each of you, but that is sort of the point with a heuristic, it is a guide not a formula, optional not rigid.
Hopefully this will help and possibly inspire some of you to expand your thinking when you need to test a REST API or clarify the requirements around REST API design etc
Feel free to share back your own variations on this heuristic or even your own heuristics.
I have been meaning to share this for a while now.
I have been inspired by, learned from and generally challenged to think more and better by some of the folks that I consider to be thought leaders in testing, namely; James Bach, Michael Bolton and Jonathan Kohl. These are amongst the best thinkers in the testing profession. They are also some of the best at sharing their knowledge, for which I am eternally grateful. I am in some small part trying to mimic them by sharing some of my thoughts and experiences here.
So this is a little overdue homage to these giants upon whose shoulders I am standing.
When trying to come up with ways to help my QA team think more broadly, differently and holistically about risks and tests for Web UI pages I realised that the mind map that I had developed over time for this purpose was not very easy to remember.
This was fine if you used my coverage outline template, (now updated to UNUSUAL PAGE), because that includes both the mindmap and the spreadsheet sections from the mindmap, thus no memory required.
But if you were in a meeting room discussing the user workflow or code design of the latest UI change, or at the desk of the User Experience designer looking over some wire frames in preparation for a 3 amigo style BDD discussion, (designed to ensure we all had a common, shared understanding of the requirements), or a story kickoff where we wanted to think about design and code risks and tests to mitigate those. But you didn’t have a laptop in front of you with the template to hand, how would you mentally run through the different aspects to consider in the context of the work in front of you?
Thinking about how I normally expanded my thoughts around where things could/would go wrong and what sorts of things I should consider testing I realised I often used heuristics I learned from the folks mentioned above. These heuristics were normally memorized in the form of simple mnemonics. Looking again at my mindmap I realised I was not that far from a fairly easy to remember mnemonic, so with a little tweaking I came up with UNUSUAL PAGE (start with URL and go clockwise);
Obviously not all of these branches or leaves will be applicable to your page and your context, and indeed the words I use here may mean different things to each of you, but that is sort of the point with a heuristic, it is a guide not a formula, optional not rigid.
Hopefully this will help and possibly inspire some of you to expand your thinking when you need to test a UI page or clarify the requirements around Web UI design etc
Feel free to share back your own variations on this heuristic or even your own heuristics.
I will share some more that I have been practicing with my team.
There has been a lot of discussion over the last couple of years about test automation and in particular the varying definitions of testing vs checking and how that applies to test automation.
I broadly agree there is a difference, here is my paraphrased understanding of each definition;
Testing – the art and science of conducting experiments and carefully observing the results, all the while making multiple evaluations against explicit and implicit expectations. A fundamentally human, (or manual if you prefer), exercise.
Checking – the deterministic evaluation of the outcome of an action or step such that a pass or fail is recorded.
But there seems to be an underlying theme to most of these discussions, almost a fear. It is as if someone has threatened the existence of manual or human testing.
I do agree that there has been a general drive towards more automation of ‘tests’, and that this has been largely associated with the adoption of agile practices. I myself have encouraged, and in some cases demanded, more investment in, and thus more, automation of tests in companies I have worked for. However, I have also encouraged and hired for manual testing, and have coached and mentored folks to be better exploratory testers (what I call brain engaged testing).
So I don’t subscribe to the fear that manual testing is a thing of the past or an unnecessary overhead. Perhaps this is why I don’t share in the what seems to be an attempt at a sharp delineation between automation and testing?
Like Michael Bolton, I do see automation as a tool and as something that supports testing.
I often use the phrase automation assisted testing, to refer to exploratory or other manual testing where the test setup or initial test data has been achieved using automated tools or scripts.
My preference is to develop automation code in a re-usable fashion, producing a library of re-usable code that is easy to ‘glue’ together in different ways such that different automated tests (or checks if you will) are achievable quickly and efficiently. But this approach also lends itself well to re-using these library ‘functions’ to assist with manual testing. If developed well then anyone with fairly basic coding skills should be able to combine some of these together in order to ‘drive’ a system under test to the point where you want to start your exploration or manual testing. Or as mentioned before, to prime the system under test with the exact data you want or need, in order to conduct the exploratory or manual testing you wish to execute next.