Be careful what expectations your team hears from you

I will often say that “you will get what you manage to” I think this is my version of Peter Drucker’s “what gets measured gets managed,” and since his quote is somewhat contentious, I will explain what I mean by mine;
In my experience, people and teams will work to produce the results that they are managed to. In other words, if your manager/leadership focuses on the number of lines of code you write (i sincerely hope not!), then your team will focus on producing more lines of code. If your manager/leadership focuses on meeting deadlines, your team will work to meet the deadlines.

We all have deadlines, so this may not seem like a bad thing. But it can be if you are only focused on making the deadline, as this often leads to making choices or decisions that will help meet the deadline but will compromise other things like quality, maintainability, performance, etc.

A warning for managers and leaders, be very careful what you choose to manage your teams to and be aware of what your teams perceive you care about most. What do your people hear from you most frequently? If all they understand you talk about or worry about is deadlines, then this is what they will believe you care about most and will try to ‘solve that problem for you,’ sometimes by cutting corners or making compromises to meet it.

I have heard so many managers and leaders say that quality is the most important thing. It is priority number 1, only to have those very same managers and leaders continually asking the team when we will be done and why a feature is not ready yet. They do not ask how well we developed the feature or how we assessed the quality.

People and teams will also develop habits and become habituated to working a certain way and making individual decisions; these habits and working patterns can then be hard to change.

An analogy for this might be how you formed certain habits based on how your parents ‘managed’ you as children. For example, if your parents insisted you clean and put away all the dishes and cookware after a family meal and make sure the kitchen and dining room were neat and tidy. Then, you have probably formed that habit and still do that today as an adult. And you may be uncomfortable if you are in a situation where dishes are left in the sink or on the dining table, as this does not feel ‘right’ to you.

Translating this into the workplace; You will probably be uncomfortable with technical debt. You may prefer to clean up after you have finished some work. Perhaps by adding tests, making sure the build pipeline is green after you merge your code. You may also update the operations docs and playbooks to ensure anyone on-call knows how to spot issues in the new code you delivered – making sure that there is no metaphorical food left rotting and generating smells.

However, if it was normal for you to leave dirty dishes on the table or in the sink, or if you formed this habit once you were no longer being managed to clean up after a meal. Then you will be used to not tidying up after yourself, or others make a mess and will be more comfortable with things being untidy, undone, not clean, etc.

Translating this into the workplace; You will probably be comfortable with technical debt. You will prefer to move onto the next fun task after you have finished some work, perhaps signing off or leaving the office after you merge your code, not checking or caring if the build is successful and if the pipeline is green with your changes. You will put off or expect someone else to run tests, tell you if anything was wrong, and forget to update the operational docs and playbooks. Metaphorically speaking, leaving some uneaten fish to rot and generate smells.

So, as managers/leaders and teammates, we need to hold each other accountable and manage the expectations that avoid smells and encourage good code hygiene. Ensure that all code is reviewed, delivered with great tests (don’t just tick a box that there are tests, review and evaluate them), that the build pipeline stays green, that system tests continue to run without failure or regression. That if issues are found, they are dealt with quickly. That code and systems are easy to maintain and quick to diagnose so that operational costs are low and customer issues and incident interrupts are very infrequent.

Take the time to change, and start to build good habits today!

When is Root Cause Analysis not a Root Cause Analysis?

When you stop at the Cause and don’t do the Analysis!

I have heard this or similar too many times, “we performed the root cause analysis and found that we were missing a test case, we have added that case now.” or “we found the root cause was this line of code, we have fixed it now.”

I heard similar again today.

RCA Stamp

Whenever you have a reason to want to perform a Root Cause Analysis, remind yourself to be grateful – a leader I admire referred to incidents as learning gifts – we learn the most and the most effectively from failures.

At my first workplace, we designed and manufactured Flight Deck Instrumentation; you know that stuff you see in cockpits that pilots rely on to fly planes and get you to your destinations without issue. Well, I will often say I have never worked on anything since that could directly cost human lives if it fails. So, I am always grateful for these learning gifts as they will help the teams involved learn how to be better, annoy or frustrate our customers less, and delight them more. (And not harm them in any way)

Which leads me to the intentions you need to have when performing a Root Cause Analysis, a quick reminder;

  • Our number one goal is to ensure we learn all we can to ensure we can prevent or at least catch future occurrences of any issues like this.
  • An RCA is never about blame.
  • We have the right people involved – Ensure you have folks representing each area of the product, service, or code involved in the issue.
  • We are focused on prevention – Ideally, you are gathering to perform the analysis and have already found and resolved the specific issue(s) – now you are focused on analyzing to understand how to prevent similar problems from occurring or going unnoticed in the future.

You will also need someone skilled and experienced at running these types of discussions. The outcomes will depend on the skills, experience, and knowledge of those performing the analysis. An experienced facilitator will know how to get the best out of the participants and call for more or more in-depth SME help when required.

For example, the 5-whys technique is a popular technique to use in an RCA. However, it can easily fail to produce the desired results as;

  • It is easy to stop at symptoms
  • You can get stuck not knowing what would be another good ‘why’ question to ask
  • Sometimes you follow the why questions down a path to a single root cause and are not open to multiple root causes.

So, the next time you have an issue or incident that you can learn from, set you and your team up for future success by;

  • Enlisting the help of a skilled facilitator
  • Getting the right people gathered together
  • Remind everyone that is a no-blame discussion
  • Ensure everyone is full of curiosity
  • All are in learning mode

Then, see what you can discover together to prevent future problems and lead more productive and less interrupted (by issues) work lives.

A previous post with an example

Beware the Streetlight effect in your feedback mechanisms

Be on the lookout for the Streetlight effect, especially when it comes to a choice between fast feedback and slower feedback.

I recently worked with a junior developer to add a simple test that could be run before code commit by any developer and as a test before the pull request was merged (if the developer forgot to run the tests in their branch before committing 🙂 ).

The intent being to provide fast and reliable feedback to whoever was making a change so that, if the test failed, then they were still in context and could very quickly and efficiently resolve the issue and commit passing code

I checked back on progress and found the junior developer had been advised to add the test to the CI/CD pipeline at the image build stage.

A senior developer, who would have been the one to help teach the junior developer how to add a test to the CI/CD pipeline, provided this advice.

Image build is the last stage of the pipeline. After several building stages, testing and packaging culminate in an image ready to be deployed into production.

Adding the test at this final stage would be trivial to add and run.

However, if the build failed at this stage, it could be hours or even days after the code change that caused this test to fail was made. The developer who made the offending code change would be in a different context now and would probably not be the person who sees the image build failure.

  • Now you have very slow feedback.
  • Another engineer has to diagnose/dig into the reason for the image build failure.
  • This engineer then needs to find who made the code change and track them down to correct it.
  • The offending code engineer may not even be available at that time.
  • Meanwhile, the build is stuck, and if you needed to get this build into production to fix a customer issue, you would be blocked and very unhappy!

So, when you have a choice between something hard (like finding the key you dropped in a dark place or adding a test to the earliest possible point of feedback) versus something easy (like looking for your dropped key under a streetlight or adding your test to the end of the pipeline). Be aware of your choice (not finding your key as this is not where you dropped it or getting your feedback very late and at a point when it will be expensive and more time consuming to fix).

streetlight effect cartoon

How testable is your or your team’s code?

How can you design your code so that it is testable?

Are you struggling to test code because you did not write it with testing in mind?

If so, you and your team need to design your code with testing in mind.

We all need and want code that is easy to test to get faster, quicker, and cheaper feedback on our changes.

So, here are some tips and techniques to help improve the testability of your code;

First, what do we mean by testability?

A quick google will provide you with lots of definitions, but a simple starting point is; Testability is the degree to which and how easily you can test the software.

If the testability of the software is high, then testing should be easier and more efficient.

In other words, you can think of testability as “how easy is it to test?”

To gauge your software’s testability, consider how efficiently you can test it in terms of these two factors; the number of tests you need to run to be confident it functions as expected and does not exhibit any unexpected issues. And the level or layer at which these tests can be developed and executed.

A larger number of tests or the majority of your tests being at a high level (e.g., UI, E2E, system-level), as opposed to at the unit or integration level, would typically indicate that your code is not very testable.

So, how can we improve the testability of our code?

What are the three main factors?

  1. Observability – the extent to which we can see that the code functions as expected and not doing anything unexpected.
  2. Controllability – the extent to which you can control your software. Often by controlling the inputs, state, or data on which each component operates.
  3. Isolate-ability – the extent to which you can test your code in isolation. 

In more detail;

Observability

Can you observe the results of any decisions or state changes in your running code?

Good examples;
Code modules or functions that return clear state or status
Bad examples;
Code modules or functions that modify state or status but do not return or share anything observable with the calling code

What are you logging?

Good examples;
Effective use of logging levels for different information types, e.g., INFO, WARN, DEBUG, ERROR.
Bad examples;
Nothing logged
You are only logging errors.

Can you query state, status, or data easily?

Good examples;
State or status is stored or queryable at all times in a DB or via an API call.
Bad examples;
State or status is maintained only in the code’s running memory, requiring sophisticated runtime debugging tools to observe it.

Controllability

Can you efficiently drive different data values into your code?

Good examples;
Easy to call your code component and pass the data you want it to use
Easy to provide data via a defined interface or data source, e.g., an API or DB.
Bad examples;
No ability to pass data into your code
Data is provided by a dependency that makes it hard or impossible to control.

Can you quickly change the current state or status of your code?

Good examples;
Easy to call your code component and pass the state you want it to have at any point in time
Easy to provide state via a defined interface or source, e.g., an API or DB
Bad examples;
No ability to pass state into your code or to set state before your code is executed
The state is provided by a dependency that makes it hard or impossible to control.

Can you control your inputs?

Good examples;
Easy to change inputs or to use mock, fake, or stubbed inputs, e.g., switch out the source of input between real/live and fake/controlled ones.
Bad examples;
Inputs are hardcoded or otherwise unchangeable.

Isolate-ability

Can you isolate your code components and test them on their own?

Good examples;
Your code component can be easily called or executed on its own. Its’ inputs controlled, outputs observed independently of other code components, such as a unit of code that can be called and passed data or state, and that returns data or state that can be easily verified.
Micro-services tend to be easy to test in isolation.
Bad examples;
Code contains many dependencies that are hard/impossible to control, meaning that it has to be run in an environment where the dependencies are met/running.
Code monoliths tend to be hard to test in isolation.

How easy is it for you to mock, fake, or stub out any dependencies for your code?

Good examples;
Easy to inject data via in-memory DB, file, or substitute a URL or other resource pointer for a fake one
Bad examples;
Hardcoded dependencies.

Other attributes which tend to impact testability include;

Separation of concerns – the extent to which each code component you want to test has a single, well-defined responsibility.

Can you easily understand or define what each code component does or is what its’ responsibility is?

Good examples;
Clear and typically singular responsibility for a piece of code
The state is changed, or a decision is made based on data or conditions assessed by the code.
Bad examples;
Code monoliths or code blocks in which you make multiple decisions or changes of state.

Understandability or readability – how easy is it to understand the intent of the code and how you achieve that intent.

Can you quickly understand what the code is doing?

Good examples;
Clean code, naming helps make intention(s) specific and meaningful,  commenting to explain any complex or non-obvious code.
Bad examples;
Obfuscated code using short naming that is not explicit or clear. Abstractions that don’t help readability or clarity Enumerations that do not help readability or clarity.

Takeaway

The key takeaway here is to think about how you will test your code BEFORE you write it. Doing so will help you ensure your code is; observable – so that you can see the result, state change, and data that occurs when your code executes.
Controllable – that you can easily control the input data or state for your code to use or work with
Isolatable – that you can easily verify your code in isolation, that you do not need to execute multiple dependencies together to test your code

I often find myself working with teams and code that is was not developed with testability in mind, and this means it is hard to add unit tests without first refactoring the code to make it more testable. It can be hard to know whether or not the code components behave as expected without using or employing sophisticated and often expensive tools or debuggers to observe it or control state or data during execution. Because it is difficult to isolate the code components, you often need to exercise the code and run tests in an integrated or almost production-like environment. Meaning that your feedback cycle is much longer, the time between you making the code change and getting any feedback on whether that change was right or not.

Putting the cart before the horse

Or the solution before we understand the problem

How often do you jump to a solution or search for a tool before you really truly understand the problem you are trying to solve?

I know I do this often …

I am trying to help a friend, colleague, or someone I lead, by telling them a solution to their problem before they have finished telling me what it is.

I have gone to a meeting convinced I know what we are going to talk about and the solution I will propose.

I have also searched for test tools that can help me test something, without having really defined clearly what or how I need to test.

Put simply, we need to fully and clearly understand the problem or problems we are seeking solutions for, because we cannot possibly evaluate any solution well without understanding the what, how, and why it will solve the problem or problems for us.

I can now realize and recognize when I am putting the cart before the horse, so, I can also see others do this.

I am often attending meetings where there are one or more participants who are eager to share the solution they have to a problem we are not all agreeing exists, or at least not all understanding the same way.

In addition, I am often asked what I think about tool X or solution Y without being asked if I have understood what problem needs to be solved by either.

Worse, I am being told a solution for a problem that I have not fully stated needs solving, or that I have fully understood myself.

Or, I am being recommended or told to use a tool when it is not clear what part of, (if indeed), any part of the solution the tool will address.

Ideally the horse should come before the cart. We should clearly and fully understand the problem or problems we need to solve before we seek solutions.

So, how can we do that effectively and consistently?

Here are some simple suggestions;

Meetings:

If the meeting has no agenda – ask what the agenda is, what do we want to get out of this meeting? What are the desired outcomes?

If the meeting seems to be going off course – bring it back by asking if we are still trying to solve the problem we agreed at the beginning or if we are now solving a different one? It may be that some feel the initial problem has been solved, and we are moving on to another. Others may feel that in trying to solve the first problem we identified a second and are now busy trying to solve that. Either way, calling this out will hopefully lead to a (re)statement of the problem that everyone should be currently seeking and evaluating solutions to.

Emails:

If you are sent an email asking you to use, take part in, or help with solving something, but the problem statement is missing or unclear, ask for clarity. I will often try to help the email sender by asking convergent questions, for example;    

I think you are asking me to solve either problem a) or problem b), but I may have misunderstood and you could be asking about c) which I am unclear about, please can you let me know which it is?

If I have guessed correctly and it is either a or b then they can simply reply with a or b and do not need to take the time to explain either fully, as I am letting them know I believe I understand both well enough to proceed with whatever was asked in the original email.


In-person:

Often this will be your boss or a colleague tapping you on the shoulder, catching you in the corridor, or by the water cooler. Asking what you think of tool X or somebody’s solution or idea. If you are not 100% sure of the context or problem that the tool, solution, or idea is in relation to, ask for clarity. For example;    

I am not sure which problem tool X, solution, or idea Y is trying to solve, can you please remind me before I can offer you an opinion?

In summary:

Make sure you clearly understand the problem before seeking or evaluating any solutions.

Wait for the person to finish explaining the problem they have, or how they see it from their perspective.

Ensure every meeting you attend has a clear agenda and desired outcomes. What are the problems we are trying to solve, what are the objectives – so that we can all understand if the meeting is on track and if we should even be there.

Don’t respond to questions about solutions or ideas unless you are sure you are clear on the problems they are addressing – you will not be able to provide good answers or evaluations if you are not clear.

Observation

With reference to my original post on The Streetlight Effect; I still see this effect today.

Recently, I was discussing the testing of a mobile app with an experienced QA, who is a very experienced black box tester. I am currently encouraging and coaching this tester to learn more about programming, code design, code architecture etc as I think any knowledge in these areas can make you a more effective and efficient tester.

This tester was seeing some problems when interacting with the mobile app UI and was going to raise a bug with the developer. I asked this tester if they were observing the mobile app. requests and responses to and from the backend API, i.e. the requests and data being sent and received either across wifi or cellular data between the mobile app. and our cloud-based backend server API. The tester was not sure how to look at that traffic. The streetlight effect – I will look here (the UI) for my keys (any issues) because I know how to look here (the UI). So I shared a how-to I had written years ago for Charles Proxy which will allow the tester to inspect the https requests and contents along with timings and more, by proxying all requests made from, and responses to, the mobile app. through his laptop. This enabled the tester to see the requests that were failing as well as the resulting error message – the tester was then able to raise a much more detailed bug and the developer was able to get straight to the problem and fix it quickly. (As opposed to a bug like, when I do this in the app, I don’t get the UI screen/data I was expecting to see displayed. Where the developer would have to first reproduce the problem, watch the traffic themselves to pinpoint the problem – taking much longer and possibly with some back and forth to clarify reproduction steps etc.)


Another example is another very talented tester who is also a strong coder so has great white box testing skills. This tester was digging into some performance issues. Trying to understand why a request for a large dataset from an API was taking so long (when the system was otherwise quiescent), and would often fail if there was any other activity (API GETs and POSTs to requests for data and to store data). We are using AWS and there is a myriad of tools and monitoring capabilities to learn and get your head around. This tester was able to extract the time taken to complete the request for data and plot this against the size of the data extracted. If you think about this visually, this tester is looking at this from a black box perspective, making a request knowing what was requested and extracting the start time of the request from the log, then extracting the completion/response time from the same log, then plotting this against the size of data returned. (The tester was increasing the data stored and thus retrieved between each test/request). In this case, being capable of white box testing and understanding the code and system architecture this tester knew that there were several key components involved in servicing this request, but was not observing any of them. In order to understand what is really going on, and in this case be able to pinpoint the parts of the system that was taking a long time to service the request and which ones would fail when there was any other activity. Most performance issues are as a result of some form of resource exhaustion, e.g. CPU, Memory, Input-Output (IO), threads, connections etc. So, we really want to be able to see how these resources are being consumed when we interact with the system, as this can lead us to understand that our CPU usage is spiking to 100% when we do something and thus cannot do more when more requests come in, or that our memory spikes up and never comes back down to the pre-request level once the request is complete – in other words some memory is not being freed leading to a resource leakage (we will exhaust this resource over time). In this case, learning how to observe the individual service docker container resources and the database resources will likely lead us fairly quickly to where the problem(s) or weak link(s) in our data (chain) are for the request we are making.

In conclusion, what we need to do more often is to ask questions like;

  • What could I watch or monitor to see, in more detail, what is happening/going wrong?
  • What is the data flow – what path does the data takes through our system and what components are involved?
  • How is the system architected and how do all of the components communicate
  • How can I observe the communications between the system components?

If any of these questions result in a “I don’t know” or similar, then ask your colleagues for help, you are likely to learn something new, even if that something is that your colleagues also don’t know the answers to some of these questions.

A BDD worked example – login page

I have used this example as a workshop to introduce BDD to a wide variety of folks at different companies. I like this example as it is deceptively simple, everyone knows how to deliver a login page, right? The reality is that we all have different ideas about what should and shouldn’t be on a login page and how it should look etc. So it does serve as a simple but very illustrative example of how using a Behaviour Driven Design approach can really help to clarify requirements, and engage the thoughts, experiences, and knowledge of all the participants to ensure what you deliver will be what was really desired. Also, that it will be both testable and tested as the high-level acceptance tests are defined up front.

Introduce roles and abbreviations

First of all, I want to introduce the roles that will be part of the discussion along with the abbreviations for those roles used in this example. Each role can then speak their part as an example of how the discussion could go for this example.
  • PM – Product Manager, our proxy for the customer, bringing the ‘what the customer wants or needs’ definitions to the team
  • DL – Development Lead
  • IxD– Interaction Designer, bringing the UI look and feel, the usability and customer workflow understanding to the team. Helping to ensure we have a consistent style, content, and customer workflows.
  • QA– Quality Assurance person (either QAE or SET) who will be ensuring we deliver the story with high quality, building it right and building the right thing
  • Dev– Developer(s), responsible for the actual implementation of the story, the code that will provide the desired functionality.
  • Implementation team– typically composed of a developer and a QA person, but can include Interaction designer, development or QA pairs.
  • Amigos– the group of people required to analyse a story – typically the PM as the customer proxy and the implementation team

Introducing the Story

The Login Page
Bring the story into ‘In Analysis’
What does the story look like at this point? (This is an example using a tool called Mingle)

The BDD discussion begins

As is fairly typical at this stage, the story does not contain a lot of detail and is kind of vague in its description
We start the discussion:
PM or Dev Lead presents the story
The 4 amigos (PM + Implementation team (IxD, Dev, and QA)) discuss and ask clarifying questions to understand the story in detail, exposing and discussing any business risks as they go
PM/DL: This story is to deliver a login page. Fairly standard login page, username and password fields and a submit button. (The what). This will be the login page for our administrators. (The who). Once they login here they will have access to the dashboard and all the administrator functionality. (The why)
QA/Dev: Do we have a mockup?
IxD: Yep, looks like this (I encourage mockups to be cheap and for me, nothing beats a whiteboard diagram for cheap, flexible and efficient;
QA/Dev: So is the button text ‘login’ or ‘submit’?
IxD: I think ‘login’ is more intuitive
QA/Dev: Is it a username or an email address?
IxD: I was assuming it was an email address
PM: Yep, we will need to use an email address, we will want this to work with our single sign-on feature coming later and that will use an email address
QA: Can I assume we will use our standard code for validating an email address?
Dev: Erm, do we have a standard email address validation code?
QA: Yes, I believe the architecture team has a regular expression they standardised on
QA: Do we want to provide any client-side validation of the password? Or should we just send it to the server for validation against the username? i.e. should we ensure it is at least 8 characters long, contains at least one special character and at least one upper case character?
PM: No, we will have checked that when we set the password at admin user creation time or when they update it themselves. Let’s just have the server side validate it against the email address. Besides if we provide guidance on how a password will be composed then an attacker can update the dictionary they are using for brute forcing so that it follows the rules.
QA: How do we want to tell the user that either their email or password is not valid? Text on the page? Red? A popup? Do we want to clear the fields?
IxD: First I think we should have ghost text in the email address field to provide an example of a correctly formatted email address. For the error, I think we should have red text above both boxes, and we should leave the fields populated, let me provide an updated mockup;
QA: Do we want to show a different message for an invalid email address, i.e. one that fails email address validation rather than a check to see if that email address is a user in our system?
IxD: Yes, I think we should help the user to avoid typos, how about red text above the username field for this too. Here is an updated mockup;

QA: Does the error text and the text on the page need to be localised?
PM: Yes, we need to support the existing 14 languages for the Admin users
Dev: So, we should use the browser context to set the locale and display localised text if we support that locale and a fallback if we don’t?
PM: Yep, we might have a different setting later if we allow users to select a preferred locale that we store as part of their profile, but at login time we don’t know who they are so we should just use the browser locale.
QA: Cool, so the localised text will be supplied as part of the page render based on the initial request to the login page URL, including any text for error messages?
Dev: Yeah that’s the way we usually do it, so we can just send error codes back and the client side code can then render the appropriate message. Of course, the email address validation will be checked at both client and server so we can provide quick feedback to the user if they don’t provide a valid email address format in that field, but also guard against someone hitting the server directly with an invalid formatted email address.
QA: Nice, we should make sure we have unit tests covering the validation on both client and server then, I can add a single invalid test for each to show the error message (UI) and return code (server)
Dev: Should we have a ‘forgot password’ link and functionality?
PM: Yes, but I don’t think we have email functionality built yet, so we will defer that to a future story
QA: Should we have a timeout for responses from the server? i.e. how should we deal with the server being busy or unresponsive?
Dev: Yes, we should have a timeout value in the client code that will display a message to try again later, do we have a mockup for that?
IxD: Agreed, let’s allow 10 seconds for the timeout and I think we should show a message to try again later if we timeout or if we get a 50X back from the server. Here is a mockup for how that text should be displayed;
QA: Do we need to support logging in on mobile devices? i.e. should this page follow a responsive design pattern?
PM: Yes, we need to support tablets right now and may need to support phone devices in the future, if we go responsive now then both should work.
QA: But we will only need to test on tablets for now, right?
PM: Yep, we will add testing stories for phone device testing later if we need them.
IxD: Responsive should be easy enough, but I may need to think about the length of the fields and the text we will need to display, particularly in different languages.
QA: What about accessibility? Do we need to support a WCAG level for this?
PM: Hmm, well we should but I think we will defer that to a future story. Let’s try to keep it in mind so that we don’t have to re-design later
Dev: What about functionality to enable maintenance notifications? i.e. the ability to add text to inform admins of upcoming maintenance or outages?
PM: Again, I want to defer that to a future story, I will sync up with Production IT to understand the requirements for that
QA: Do we need to limit the number of attempts to login so we can avoid brute force security attacks?
PM: Hmm, yes I think we should allow 3 attempts and then lock the account for maybe 5 minutes?
IxD: Actually I think 5 attempts would be better
PM: OK let’s go with 5 attempts and 5 minutes wait time
Dev: Do we want to log each and every login attempt (both successful and unsuccessful) or just the ones that result in a lock on the account?
PM: Hmm, I think we may need to log all attempts along with success, failure or lock so that we can provide an audit log if the customer needs it or if we need to show anything for a security audit.
QA: How do we want to show the lock message when 5 unsuccessful attempts have been made?
IxD: I am thinking red text again, but below the 2 boxes and to the left of the button this time, here is an updated mockup;
QA: How are we determining 5 attempts to login? Attempts using the same email address? Coming from the same IP? Some form of session identifier e.g. cookie?
Dev: Well the simplest is to set a session id in a cookie when an attempt is made on the server side and then to count how many attempts are made with this session id
QA: I presume that means someone malicious could simply brute force by creating ‘cookie-less’ requests?
Dev: Yeah, maybe we need to think about that one some more or talk to the security team.
QA: What should happen if the user attempts another login when we have locked them out?
.
.

.

QA: So based on all of that, what do we need as examples to accept this story with? I am thinking something like this;
Given a valid email address and password when I select login then I should be authenticated and taken to the dashboard page
Given an invalid email address or password when I select login then  should see a message indicating that my login attempt was unsuccessful
Given a badly formatted email address when I focus outside of the email address text field then I should see a message indicating that I have entered an incorrectly formatted email address
Given I am entering an invalid email address or password for the 5th time, when I select login then I should see a message indicating that I must wait 5 minutes before trying to login again
Dev: Should we include an AT for the server busy/down error message too?
QA: Is that required for acceptance? It is not something the user is in control of or can directly impact (without removing their network connection)
PM: I agree, we should test for it but I don’t think we need to include that in the Acceptance Tests
PM: I want to be sure this will look and work well on a tablet, so can we make sure we test that?
QA: Sure, we can do desk checks on an iPad if you like? But we will automate the ATs using Selenium and test with our most popular customer browsers for the admin interface
PM: Great
IxD: I am a bit concerned with how the localised text will look, can we make sure we test that too?
QA: Sure, we will test that the locale gets set and fallback as expected and we will do some basic checks with pseudo loc to make sure we don’t have overlap, truncation etc, but how about we include some different languages in the desk checks and with iPad to make sure you are ok with the the look and feel?
IxD: Sounds good
Dev: What about an AT for the audit logging?
PM: This is not a formal requirement from our customers or security team yet, so I want it tested but it does not need to be an AT as it is not a must-have part of the specification.
QA: No need for any regression tests here as this is all new code and not dependent on anything else.
PM: Do you need a new environment to test with?
QA: We already have the pipeline setup so we can just deploy to that from the CI system and test there, so no, we should be ok.
PM: What do we think are the biggest risks?
QA: Well I think security is the biggest as this is a login page, but we will mitigate that with validation in the client and server side plus testing focused on circumventing security, including the lockout to prevent brute forcing. The next biggest is email validation, this is notoriously problematic as most email clients do not conform to the RFCs. We will mitigate this by using our standard email validation to be consistent and to have one place to change if customers complain. We can also monitor the audit logs to see if people regularly use different methods of commenting or ‘tagging’ their emails that we should allow for. I am not really concerned about performance (very little traffic between server and client) and we will do regular desk checks for usability and style including the error messages, localised text, and responsive design.
What does the story card look like now?
QA: We need to get more specific with our examples, we have captured the top level ideas and behaviours but we really want to provide examples (or specification by example) so that it is 100% clear how this will behave and so we know how we will demonstrate this to you for acceptance, so how about;
Given I have entered [email protected] and Password1 (valid pass) as the password
When I login
Then I should be presented with the dashboard page
Given I have entered [email protected] and Password1
When I login
Then I am presented with an error message in red text saying “Invalid email or password”
Given I have entered valid.email@notopleveldomain as the email address
When I change focus from the email field
Then I am presented with an error message in red text saying “invalid email address”
Given I am entering invalid credentials for the 5th time in a row
When I login
Then I am presented with an error message in red text saying “Too many failed login attempts, please wait 5 minutes before trying to login again”
PM: OK, we know the scope now and we have Acceptance Tests defined, so what do we think is the size of this story is?
At this point, we have clarified and agreed on the scope, and have a common understanding of what ‘done’ looks like in the form of some high-level acceptance tests. It is reasonable to guesstimate the size of the story at this point. But note we are much more likely to be able to guess more accurately once we have talked through the design in a bit more detail – the how we will solve this need in code discussion.

Testing as a career

I am going to join a long list of people I consider thought leaders in our industry in a lament of how many people continue to see testing as a simple job, not a career, not a craft, not really requiring any skill at all.

But, like Lalitkumar Bhamare I felt I needed to talk about this, because …

I recently interviewed a candidate that had had two co-op experiences as a QA, he had been given requirements and design specs and asked to think of tests (purely black box) for these. He believed he was doing well and that the team was working efficiently when he could think of one or two cases in addition to the one obvious one to confirm the requirement was met.

He found bugs, 2 or 3 a day, he said with some pride.

It was obvious he did not think this was hard, required any skill, or was a career that he wanted, but he felt trapped and compelled to apply for a QA role as that was what he had experience in. He really wanted to be a developer.

I explained he could be in a very different environment where instead of being part of a waterfall like cycle like this;

  • customer proxy passes requirements to the tester and developer,
  • the developer develops code to meet the requirements,
  • the tester thinks up a few test cases per requirement (he told us he came up with about 2.5 on average).
  • The developer then hands code to the tester who then takes the code (product) and tests it by following the heavily prescribed test cases he just wrote. (Presumably without thinking much). Finds a bug or two, records that in the bug tracking system and moves onto the next test case.
  • Then the game of ping pong begins as the tester throws the code (product) back to the developer to fix the bugs. The developer fixes the bugs and throws it back …

That instead, he could use his skill, (yes I did say the word skill), to help the customer proxy and the developer understand what he was thinking of testing before the coding started. That this might actually prevent the developer from missing the requirement to meet the other use case the tester had thought of. That the customer proxy may have an opinion on this and say, “no we don’t want that”, or perhaps, “yes we want to meet that need too, and now you have made me think of another one that we need to meet too!” (Catching a completely missing requirement)

He didn’t see it, couldn’t see himself doing that, couldn’t see a way that could work, at least not during the interview. To his credit he was thinking about it and it clearly confused him but he was not able to take this any further.

(I did also suggest that there was no reason for him not to apply for software development roles)

Many, many, others I have interviewed, some who have been ‘testing’, (I would say checking), for years, still have no thought that they could be better, that they could improve and develop their skills, that they could really help their team to avoid making so many costly mistakes and save their companies thousands if not millions in wasted efforts, time lost due to bug interrupts and context switches, re-builds, re-tests, whack-a-mole etc. In other words deliver real value and not just fill a role, warm a seat, expend some effort, etc.

A co-op student I worked with, (believe me I am not picking on co-ops at all, they have very little experience to draw on, but are also often not taught anything good about testing), visibly turned his nose up at doing some testing, saying, “I don’t want to just press buttons all day long”. This was his understanding and belief of what testing was. Not entirely sure what his experience was or where he got that, but I tried to disabuse him of that incorrect understanding of the role, but once again felt myself working against an entrenched idea.

So, I call out to the learning establishments to do a better job of preparing these young minds for the real work of professional software engineering. That good software developers do a lot of thinking and a lot of testing as part of everything they do, and that doing this well takes a lot of skill, experience and learning. And that good software engineering teams and leaders recognize the need for people on their team that think differently and are skilled in the art of helping the team expose issues as early and as quickly as possible, to help them deliver the right solution first time more of the time. Without the need for drama, missed deadlines, long hours, late nights, re-design, context switching to fix bugs found in code developed weeks ago, etc. That this, (software testing), is a career, a valuable role, and an essential part of any high performance team. That it also takes a lot of skill, experience and learning to do it well.

I also call out to all the companies out there that insist on continuing to think of testing as simply, ‘using the product the way a customer will’, and hoping that will catch enough issues. Believing that in doing this they are ‘testing the quality into the product’! They are both wasting and losing a lot of money, as they have to pay the cost of finding defects late in their cycle. They also need large customer support teams, and in some cases teams of developers focused on maintaining, (fixing), their colleagues code.

These companies are responsible for perpetuating the myth that testing is basic, unskilled and a simple role. That it is easy to automate what these testers do or that we can simply outsource this work to countries where peoples’ time is cheaper.
They are also often responsible for poor quality products, that will often miss deadlines, and require expensive maintenance programs or extensive and expensive re-writes every few years.

And finally I call out to all the people in testing or quality related roles out there to learn to do better, be better and thus change the perceptions of everyone around them to see that testing is a skilled role, it is a craft and thus requires craftsmanship, and that it needs to be part of every team and will improve the quality of everything we do.

But it’s not all bad …

I have employed and had the pleasure of working with some fantastically skilled testers, both talented exploratory testers, and test automators. Sometimes a few rare gems that are able to combine great skill in both testing and the development of tools and automation to help check software.

I have also had the great pleasure of working with lots of great software engineers and leaders who really understand and seek the value of testing. Insisting that they have great testers to work with them, that they and their software developer colleagues need to think about and execute testing at all levels and that they encourage all to learn more about and be better at ensuring the quality of everything we do.

I have been inspired, guided and encouraged in my career by many people, but most of this has come from what I think are some of the thought leaders in this area. Here are some links to articles, books and people that can help you too;

I would also highly recommend Rapid Software Testing training, this course will be one of the best investments you can make in your testing career.

QA Recruitment – what do I look for?

I recently got asked what I look for when recruiting for QA/test positions. So I decided I should share my thoughts both with those of my colleagues who are recruiting, and also those who are looking and maybe wondering what they need to be able to share in an interview.

I have broken this out into a small number of attributes or qualities I tend to focus on:

Passion for testing?

Try to understand if they are passionate about what they do
  • ask questions about why/how did they got into a QA role or what keeps them in a QA role
  • looking for someone who cares about quality and recognizes that it takes skill, that they have some of that and they enjoy using it
  • ask them to tell you about an interesting bug they found and why it is interesting (was is challenging to expose, challenging for the team to resolve, an interesting chain of events type issue … is there genuine interest and passion in finding and solving problems here)

Motivations

  • what do they enjoy most about their current role?
  • why do they want to leave?
  • what does this tell me about what motivates them and what they may be looking for?

Testing Basics

  • do they understand the basic test design techniques of equivalence class partitioning and boundary value analysis – keeping in mind they may not know the terms, they may just do this intuitively
  • ask example questions that have clear equivalence classes and boundary values or get them to explain how they would think about the test cases for a situation like this
  • ask why they think of testing with those values to see if they have ever identified that they are applying a technique or a pattern to these types of problems
  • you can also ask other basic questions like what are some of the important points to include in a good defect report, what they do when faced with ambiguous or unclear requirements
  • do they know any test heuristics?
  • do they perform exploratory testing? Can they describe what they do or give you an example? (Are they truly exploring or are they just doing ad-hoc and thinking they are the same?)

Teamwork

How well do they work with others
  • how big is their team – if they are the only QA then they probably don’t have anyone to bounce ideas off or tell them there is another/better way of doing something
  • are they all in the same office? – if yes how often are they talking with others on the team and why
  • if some/all are remote – how do they communicate and what problems do they have communicating
  • do they try to help developers resolve issues – ask them if they attempt to narrow down where, how or why a bug is occurring
  • do they try to help PM/BA define requirements – ask them how the requirements analysis/review happens and how they provide feedback or ask questions to clarify requirements
  • you are watching for any team vs us type attitudes or ‘not my role/job’ type attitudes

Process basics

  • do they understand their current process? Can they describe it briefly and succinctly? – if not then they probably don’t understand it well enough to explain it clearly to a new teammate, or it is too complicated for them to efficiently work with and they don’t know how to improve it
  • do they do regression testing
  • do they get to choose what testing is done and in what order?
  • if yes, what do they do first and why?
  • if not, then do they understand how the decisions on what to test and in what order are made?
  • what happens when they find a bug in something they are testing? Do they stop and wait for a fix or pick up something else?
  • if they could change one thing about their current process in order to improve it, what would they change and why?

Automation – only applicable if they have some automation experience

  • what do they automate and why do they focus on automating that testing? Often get simple answers here varying from I automate what I am given/told to automate through to I automate everything – you are trying to understand if they know that there is judgement required here – choices to be made about what to automate in order to improve the efficiency of the team
  • also common that QA automation is focused on UI – ask them if they have any experience of automating anything other than UI
  • do they run their automation regularly? In a CI system?
  • what problems have they had with automation – digging here to understand if they have built fragile automation or if they have learnt the basic mistakes and are experiencing more advanced problems like how to efficiently manage test data or parallelize tests more effectively

Continuous learning

Lots of people say they do this but do they really? And what are they trying to learn?
  • ask them how they keep their skills current
  • ask them to tell you about something they have learned recently that they found particularly interesting or useful
  • ask them if they follow any QA/test/development folks blogs or twitter feeds and what they learn from that if they do