Quality Engineering Basics: Automated vs. Manual
Part 7 of Quality Engineering Basics. Today's topic is a simple framework for prioritizing test scenarios and determining what to automate.
One challenge in testing is determining what to test. This involves identifying the use cases, user personas, type of testing, and whether to do manual or automated testing.
For now, let’s focus on determining what tests have the highest priority, and which should be automated vs. run manually.
Similar to last week’s risk analysis, we can take a score-based approach. In fact, we can reuse all the information from last week’s example, and add new columns for frequency (of use), manual effort, and automation effort.
Impact, Likelihood, and Frequency are scored using high (5), medium (3), and low (1). For Manual and Automation effort, these are scored using high (1), medium (5), and low (10).
Now, let’s look at our first test scenario, make a purchase, which scores the highest for risk (25). Here, it has a score of 50 for manual testing, and a score of 25 for automation. This is calculated using impact, frequency, and effort. Likelihood is used only for the risk score, as this will change over time as the codebase changes. I recommend recalculating your risk scores periodically to make adjustments to your automation suites.
My approach is to look first at things with the most risk, and then evaluate the manual vs. automation scores. In some cases, you might want to include a column for how likely the feature or scenario is to change in the future. This can be useful when manual and automation efforts score similarly. The less likely a feature is to change, the better candidate it is for automation.
To get started, start with a small set of scenarios, and then adjust the numbers used for scoring until you’re happy with the results. Then you can apply it to a larger number of scenarios or use cases.
Another option is to add the frequency of use into the risk analysis score. The framework is designed to be flexible, so adjust it to work for you and your needs.
Update: I’m sharing a link to my example spreadsheet for anyone who wants to make a copy.
Conversation Starters:
How do you decide what to automate vs. test manually?
How do you prioritize your testing?
Until next time,
Brie