Quality Engineering Basics: Quality Metrics

Part 9 of Quality Engineering Basics. Let's talk about metrics!

Metrics are helpful for informing stakeholders on a product's current quality state and as a tool for driving improvement. They can be used to improve quality, user satisfaction, developer productivity, and testing efficiency.

While there are numerous quality metrics, today I’ll cover some of the metrics I find most valuable.

Bug Metrics

• How many bugs are found post-release (in production)? What are their priorities?

  • This metric identifies gaps in testing. If you are finding critical bugs or lots of bugs post-release, why? It might be a good time for a bug retrospective!

  • Variations:

    • Escape Rate: of the total # of bugs reported, what % are reported post-release?

    • Change Failure Percentage¹ : the % of releases that result in degraded service and require immediate remediation, such as a hotfix, rollback, or patch.

• How long does it take to fix production bugs?

  • Often called Mean Time to Repair (MTTR), this tells you how quickly customer-affecting bugs are resolved. Another variation is Failed Deployment Recovery Time, another DORA metric.

  • The DORA metric implies that this is measured from the time of deployment, not detection. So, if you are tracking the elapsed time from bug report to fix, you might want to also look at how long the bug was in production before it was reported.

  • If this is low, does that make production bugs more acceptable?

• How many bugs are open? What are their priorities? How long have they been open? What features or product areas have the most bugs?

  • I like to look at bug clusters. Are there areas of the product (or code) with a high density of bugs? These can be grouped together, as fixing and testing them all at once is more efficient than fixing the same number of bugs across various features. Additionally, this can make fixing lower-priority bugs more appealing, as the impact on the user experience will be more significant. For example, fixing ten bugs related to feature A means you can now say feature A is greatly improved, vs. fixing ten bugs spaning ten features.

• How does the number of open bugs compare to closed/fixed bugs?

  • If you find more bugs than you are fixing, this is a sign of decreasing quality.

• Who’s reporting bugs? Testers? Developers? Customers?

  • Does this match your expectations?

  • Is it acceptable if your customers report more bugs than your engineering teams?

• When/how are bugs found?

  • Ideally, you want to find bugs as early as possible in the development lifecycle.

    

    Relative Cost of Defects, National Institute of Standards and Technology.

Product Metrics

• What problems are customers reporting through customer support? How satisfied are customers?

  • When customers repeatedly report the same issue, this can indicate a bug or a poor user experience. Is there confusion about how the product works? Are features missing or misunderstood?

  • If your product has zero bugs, but customers aren’t satisfied, do you have a quality product? I’d say no.

  • Customer satisfaction can be measured using Net Promoter Score².

• Sign-ups, retention, engagement, refunds, churn.

  • Yes, these are product health metrics, but quality can all directly influence these. Bugs can negatively impact all product metrics, and unexpected changes in these metrics can indicate something is wrong or broken.

Automation Metrics

• How often are automated tests run? When are they run?

• How long do they take?

• Pass/fail rates.

• What bugs are caught by automated tests? How many? Priorities? How does this compare to other reported bugs?

• How often do tests fail when there isn’t a bug? This is typically referred to as flakiness.

• Code/test coverage.

  • How much of the testing is manual? How much is automated?

  • At what level are your automated tests?

  • Are you aiming for a testing pyramid³? Or a testing trophy⁴? This metric can help visualize your testing coverage and uncover gaps.

Example Charts

Conversation Starters:

• If you could track only one quality metric, what would it be?

• Is it ok if automated tests don’t find bugs? Why or not?

Like this article? Click the ♥️ button to let me know!

Or leave a comment 💬 and share your thoughts.

xo,

Brie

PS. If you’d like to support my writing and my work on QUALITY BOSS, you can leave me a tip through Ko-fi.

---

International Women’s Day Special: If you’re a woman interested in coaching or mentoring, I have two one spot remaining (out of 5) for five free hours. Book a call with me here.

My areas of expertise and interest are leadership development, conquering impostor syndrome, values exploration, goal setting, and creating habits & systems. And, of course, Quality Engineering. 🐞

1. This is one of the key DORA Metrics.

2. Net Promoter Score asks how likely a user is to recommend your product or service. This can be a useful benchmark metric to compare over time.

3. Martin Fowler on the Test Pyramid.

4. Symflower on the Testing Trophy.