Quality Engineering Basics: Risk Analysis
Part 6 of Quality Engineering Basics. Today's topic is risk analysis.
Last week, I wrote about exploratory testing, and the nightmare headline game. When thinking of nightmare headlines, and all the things that could go wrong with a product or software, the possibilities may seem endless. However, some things are more likely to go wrong than others, and some are more consequential.
While achieving a bug-free product may be ideal, it’s virtually impossible to eliminate all bugs. Instead, our goal is to minimize the risk of impactful bugs. These are the bugs that affect revenue, engagement, retention, or severely deteriorate the user experience, especially for key customer segments.
Consider a bug that affects 5% of free users versus a bug that affects 1% of your best paying customers. Which is more impactful?
This is where risk analysis comes into play, guiding us to focus our testing efforts on the most critical areas of the software.
Assessing Risk
A quick way to assess risk is to ask these two questions:
What is the impact if this doesn’t work as intended?
What is the likelihood that this won’t work as intended?
Another way to think of these questions is in terms of product risk (impact) and technical risk (likelihood).
To do this for a release cycle, make a list of your use cases or test scenarios. Then, for each, assign a value for impact with 1 for no or low impact, and 5 for high impact. Do the same for the likelihood. (Tip: a spreadsheet works great for this).
Let’s look at an example. In this, I used high (5), medium (3), and low (1), and multiplied to get the score for each item. The examples and scores are entirely made-up, based on possible use cases for a fictional online shopping website. When doing this for a real product, I use intuition, input from customer support and product (impact), guidance from the developers (likelihood of failure), code coverage analysis (likelihood of failure), and past history of bugs, among other things, to determine scores.
As you can see, making a purchase scored the highest at 25. This is where I’d start testing. Next is sign up as a new user and log in, both with a score of 15.
Beyond Functionality: A Holistic View of Risk
Risk analysis shouldn't be confined to functional aspects alone. Consider security, user experience, reliability, performance, and testability, just to name a few.
There is nothing worse than discovering your ability to test the product or software is severely limited. For instance, a project I was involved in required testing with Spanish mobile numbers—a challenging task from our base in California, illustrating the need for comprehensive risk assessment.
Lena Nyström's Would Heu-Risk it? is a great resource, and can bring a fun, interactive component to your risk analysis.
Conversation Starters:
What criteria do you use to evaluate the risk in your product or software?
How do you balance the impact and likelihood when evaluating risks? Are there specific criteria or metrics you rely on to make these assessments?
Join me next week for a related topic — test case prioritization.
Brie