Quality Engineering Basics: Prioritizing Bugs

Part 8 of Quality Engineering Basics. Today's topic is prioritizing bugs.

Continuing with the theme of prioritization, today let’s dive into bug prioritization. While not always the responsibility of quality engineers or software testers, I have found that testers are almost always asked to set an initial priority for bugs they file. In my recent roles, I've been entrusted with defining priority levels, escalation processes, and SLAs for bugs based on their priorities.

Bug Prioritization

Effective bug prioritization relies on three things:

• A common understanding of which bugs fall into each priority level.

• A structured escalation process.

• Clear expectations on resolution timeframes.

Today, our focus centers on the first component: creating a common understanding of the criteria for assigning each priority level to bugs.

Depending on your organization’s bug tracking system, you may have separate fields for priority and severity field, or you might have only one field. In some cases, like at my last company, you might not even have a designated field. We used GitHub, and without a dedicated field, had to rely on labels–it wasn’t pretty.

Personally, I prefer one priority field, and to use a matrix to determine a bug’s priority.

On the y-axis (left), we have the scope of the bug. This is a measure of how many users are impacted by the bug, or for non-customer facing issues, the business impact (an example of the latter would be something like a logging bug where important business metrics aren’t captured correctly).

On the x-axis (top), we have the impact of the bug. This is also often referred to as severity.

If starting with this as a template, the next step would be to answer the following questions:

• Do the buckets of < 5%, 5-15%, 15-50%, and > 50% make sense for your product? At one company, we adjusted these to make the top most bucket bigger as our quality got better, and as we raised the product price. Previously, the top bucket was > 25%, and we moved it to 50%.

• Do you need to differentiate between free users and paid users?

• What is low vs. medium vs. high business impact? This can be in terms of revenue, cost, or person hours.

• What constitutes a cosmetic bug? A UX bug?

• What is your product’s core functionality (the main features)? What is secondary functionality (less used features)?

• What do you consider a workaround? If the customer has to contact support, and support can implement a workaround, you might categorize those bugs differently from workarounds the customer can do on their own.

• What are other considerations or types of bugs for your product, and how do they fit into the impact categories?

• Is 3 priority levels the right amount? Personally, I prefer 4, with an additional P0 for bugs that are extremely urgent.

Once you have the buckets and definitions, then apply the matrix to actual bugs and see where they land. Do you agree with the priority the matrix would assign? If not, adjust the values in each square of the matrix.

In my most recent role, we streamlined this process using a GitHub action to automate priority assignment. Bug reporters selected parameters corresponding to the scope and impact via dropdown menus in our issue template, which GitHub actions then utilized to apply the corresponding priority label and escalate urgent bugs to my team via Slack.

The matrix approach is great if you’re using something like Jira, and bug reporters have to select a single priority field, or a priority and severity field. If you’re using GitHub, and can layer in GitHub actions, there’s a lot more room to get creative. One approach would be to add additional categories, and then assign points to each bucket within each category.

For example, what if new sign-ups are really important for your product, and you want those bugs to be prioritized higher even if they affect a low number of users? Instead of putting a priority in each box on the matrix, you’d put a score from 0-100, with 100 being the highest priority. You might set 0-33 as low priority, then 33-66 as medium, and then 67-100 as high priority. Then, for issues impacting sign-ups, you could have a modifier like +25 points. Anything that would normally be a low priority bug is then bumped up to medium.

Conversation Starters:

• How does your team currently prioritize bugs, and is there consensus on the criteria?

• Have you encountered situations where advocating for a bug’s priority was necessary?

Until next time,

Brie