Quality Engineering Basics: Bug Reporting
This is part 1 of a new series, Quality Engineering Basics. In this series, I’ll cover some of the basics of Quality Engineering and Software Testing.
In my post, Getting Started as a Quality Engineer, I recommended getting familiar with finding and reporting bugs.
This post dives deeper into the topic of bug reporting, and what makes a good bug report. After reading this, you’ll have a solid understanding of the important pieces that make up a good bug report.
Over the years, one question I’ve often asked interview candidates is to tell me what information they feel is important to include when reporting a bug. A pretty basic question, but you’d be surprised by the range and quality of answers.
So, What Goes in a Good Bug Report?
I’m in love with all the things AI can do these days, so I asked ChatGPT for help to get us started.
Prompt: What goes in a good bug report?
ChatGPT: A good bug report typically includes the following key elements:
Clear Title: A brief, descriptive title that summarizes the issue.
Environment Details: Information about the system where the bug was found, such as software version, browser type, operating system, and device.
Steps to Reproduce: A step-by-step guide on how to reproduce the bug. This should be clear and specific so anyone can follow it.
Expected vs. Actual Results: Describe what you expected to happen and what actually happened. Highlighting the difference helps in understanding the bug's impact.
Screenshots or Videos: Visual aids can be very helpful in showing the issue, especially for UI bugs.
Severity and Impact: Indicate how severe the bug is and what impact it has on the user experience or system functionality.
Log Files or Error Messages: If available, include any relevant logs or error messages.
Additional Notes: Any other information that might be helpful, like workarounds you've found or conditions under which the bug does not occur.
Remember, clarity and detail are key in a good bug report. The goal is to make it easy for the team to understand and address the issue.
This is a pretty good answer. We’ll examine it in more detail in a moment.
During interviews, my follow-up question is to ask what’s the most important piece of information in a bug report.
There are two answers that I consider to be the best. Let’s see how ChatGPT does with this one.
Prompt: What's the most important piece of a good bug report?
ChatGPT: The most important piece of a good bug report is the Steps to Reproduce. This section provides a clear, step-by-step guide on how to replicate the bug, which is crucial for the development and quality engineering teams to understand and fix the issue. It ensures that the team can see the bug in action and verify it in their environment. Without clear reproduction steps, it can be challenging to pinpoint the cause of the problem and verify that it has been resolved.
Good again! This is one of my two preferred answers, with a clear, concise title being the other one.
Now that we have an example from ChatGPT of information that goes into a bug report, and identified what’s most important, let’s dig into more details.
Remember, as a quality engineer or software tester, the more information and context you can provide in your bug reports, the better. A well-written bug report makes it so much easier for your bug to be prioritized and fixed.
Title
This is your first opportunity to explain your bug. Often, this is the only part of the bug report that others might read. This must be clear, concise, and descriptive.
Let’s look at a few real-world examples, using Substack as our SUT (software under test).
I recently moved QUALITY BOSS from WordPress.com to Substack. Substack has a nice WordPress importer, but I ran into two pretty bad bugs.
For each of these, here are some options for titles.
Import Bug #1
Problem with Import.
Import didn’t work.
Only excerpts were imported.
WordPress Import via URL: imported excerpts only, instead of full posts.
Which one do you think is the best? Which one is more meaningful to a product manager or developer? Which one gives the reader context without requiring them to read the entire bug report?
Import Bug #2
Imported post is broken.
Unable to edit imported post.
WordPress Import via XML: Unable to edit a specific imported post.
Again, which one do you think is better?
Environment
This information is often overlooked or not reported at all. It’s really useful to include, even when it seems unnecessary.
For a web-based product or website, include:
OS including version #.
Web browser including version #.
Alternate browsers tried, if any, and the results. This helps the team understand if the issue is browser-specific or not.
It can also be helpful to include any browser extensions you have installed/active.
For mobile, include:
Phone Model and OS version #.
If a web bug, browser and browser version #.
For apps, app version #.
Steps to Reproduce (STR)
Steps to reproduce are a crucial part of any bug report. If you don’t know how to reproduce the bug, that’s ok. This doesn’t mean you should leave this out of your bug report! Instead, include as much information as you can, and be clear where the exact steps are missing.
For example, you might find that sometimes when you go to [feature area] of [app], the app crashes. However, it doesn’t happen every time, and you’re not sure what the exact sequence is to reproduce the crash. You might write something like this:
STR:
Exact steps to reproduce 100% of the time are unknown.
Navigate to [feature area].
App crashes [x] out 10 times. It seems to happen most often after [some action].
When something doesn’t reproduce all of the time, it’s really important to note that and to mention–as best you can–how often it does occur.
Another issue I often see is that people include too many steps. Let’s look at this example:
STR:
Create an account with Substack.
Create a newletter with title [x] and settings [y,z].
Publish a post.
Click on Dashboard → settings → import.
Enter [URL] into the URL field.
Click Get Started.
…
Steps 1-3 are probably not necessary. We can infer that the import function is only available to users with an account and a newsletter. Now, the exact settings of the newsletter and the fact that there is already an existing post are useful to include only if they affect the reproducibility of the bug.
So, going back to my previous bug examples, if import bug #1 only happened with certain newsletter settings, or only when there are existing posts (or only 1 existing post, or no existing posts, etc.), then this is useful to include. However, it would be better to include this as part of the description or the environment.
Such as:
Account/Newsletter: QUALITY BOSS
Details: 1 existing post
Settings: x,y,z
An excellent bug report contains all the information needed to reproduce the bug, and nothing more. As a tester, if you can identify the exact conditions where the bug occurs and the conditions when it doesn’t, this makes the developer’s job to fix it much easier.
Knowing what that is can sometimes be tricky, but you’ll get better over time as you learn more about the SUT.
Expected vs. Actual Results
Here is where you make your case as to why your bug is a bug.
For example, I went to the ATM and I took out $20. I received 2 $10 bills, but I was expecting a single $20 bill. Is this a bug?
It depends. Is the system designed to give out the highest denomination bill possible? Is it acceptable to give out smaller bills? When?
Let’s say I wrote this up as a bug. The developer (or product manager/owner) might come back to me and say “Nope, this is expected. It happens when the machine is out of $20 bills.”
Then I can retest, and I can verify if the machine is out of $20 bills. If it’s NOT, and I still get 2 $10 bills, then I have a legitimate bug.
Screenshots or Videos
These are a must for UI issues. For screenshots, editing the screenshot and marking exactly what you see that’s wrong can be incredibly powerful.
Also, for videos, trimming them to show only the relevant portion, or noting the timestamp of when the bug happens is extremely useful.
Severity and Impact
Some bug trackers have severity as a dedicated field. If so, use the guidelines provided to select the most appropriate value.
As for impact, I like to see this included in the description of the bug.
Log Files or Error Messages
These should always be included when available. If you see an error message, copying the exact text, or even better including a screenshot, is the way to go.
Additional Notes
ChatGPT mentions workarounds and scenarios when the bug does not happen. I prefer these as part of the description, along with the impact and reproducibility rate.
Description
ChatGPT neglected to include description in its list. This is a longer version of the bug’s title, and as mentioned earlier, can include things like impact, workarounds, and reproducibility rate.
Example Bug Report
Going back to our example bugs, here’s what a full bug report might look like.
Title: WordPress Import via XML: Unable to edit a specific imported post
Description: After importing blog posts from a WordPress XML file, I am unable to properly edit one out of ten imported posts. I was able to reproduce this consistently with multiple attempts.
Additionally, I tried duplicating the post as a draft and ran into the same issues trying to edit. I was successfully able to edit all nine of the other imported posts.
Workaround: I copied the content from the original source into a new post and deleted the imported post.
Impact: As a new user, it’s frustrating to have the platform not work as expected. Additionally, if edits are necessary for the post to be live on Substack, not all users are going to discover or implement the workaround and may abandon the platform.
Environment:
Mac OS version 1234
Arc Browser version 567
XML file (attached) exported from WordPress.com using WP version 6.0
Account/Newsletter: QUALITY BOSS
Steps to Reproduce:
Export WordPress.com posts using tools → export → posts, and save the XML file.
On Substack, go to Dashboard → Settings → Import.
Enter qualityboss.blog into the URL field.
Click [Get Started].
Select file to upload → select WP XML file from step 1. (Selection for “Update existing posts does not seem to matter)
Click [Import].
Confirm ownership by checking the box next to “Yes, this is my publication and I accept” and clicking [Next].
After the import is complete, attempt to edit the post “Getting Started as a Quality Engineer”.
Expected Results
I expect to be able to edit any part of the post, as usual.
Actual Results
In edit mode, for this post only, when pressing backspace, nothing happens. If I select or highlight some text, and type over it, the text is replaced with the newly typed text. However, it then reverts to the original text after a few seconds.
When trying to edit a link, the link controls (change | remove) do not appear.
Screenshots or Videos
Attached video showing the actual results.
Log Files or Error Messages
No logs or errors available. However, here I would attach the XML file unless there’s another place to add files.
What About Priority?
Depending on the company, and whether you’re reporting something internally or as a user, you may or may not be asked to set the priority of the bug.
Every company will have its guidelines for prioritizing bugs. This is a key part of a bug report, but as it is often not set by the bug reporter, I did not include it here.
What Else?
Depending on the situation, you might want to include additional information such as:
Number of affected customers.
Number and details of customers who have reported the issue.
Earliest version where the bug is reproducible. For example, if you’re using version 5 of a mobile app, does the bug reproduce in version 4? Version 1?
For my example bug, identifying what part of the imported content is causing the issue would be hugely beneficial. Going the extra mile is what transforms a good bug report into an excellent bug report.
This is part 1 of a new series, Quality Engineering Basics. Look for more posts in the coming months.
What topics would you like to see covered as part of this series?
Until next time,
Brie