Quality Engineering Basics: Exploratory Testing
Part 5 of Quality Engineering Basics. In this series, I’ll cover some basics of Quality Engineering and Software Testing. Today's topic is exploratory testing.
As I mentioned last week, exploratory testing is one of my favorite types of testing. You might be asking yourself, “What is there to love about exploratory testing?”
Well, to start, exploratory testing requires very little preparation, it’s suitable for time-boxed testing efforts, easy to learn, and it’s great for finding interesting bugs!
Exploratory testing is a great introduction to the world of testing. I’ve found that folks in product, marketing, customer support, and other non-engineering roles often find value in learning these techniques. At Pinger, I ran an exploratory testing workshop for members of the customer support team, to help them participate in our internal dogfooding1 sessions.
Additionally, a tester’s mindset, along with debugging skills can be useful when applied to customer issues, and helping solve a customer’s problem. For example, if a customer support agent can help a customer narrow down an issue to a certain use case, then they can advise the customer on alternatives while the bug is being resolved.
Whenever I talk about exploratory testing, I like to start with the nightmare headline game.
The Nightmare Headline Game
What’s the worst news headline you can imagine for your product?
I first heard of the nightmare headline game in Explore It! by Elisabeth Hendrickson. It’s very similar to the process used in pre-mortems2, where the goal is to imagine worst-case scenarios, identify risks, and establish risk mitigation.
It’s a very accessible way for anyone working on a product to think about risks, and how the product might fail. From there, the next step is preemptively making sure those failures won’t happen.
So, pick a product, either one you currently work on, or you can pick your favorite product or app.
What’s the worst news headline you can imagine for [product]?
Write these down, we’ll come back to them later.
What is Exploratory Testing?
Alright, let’s dive into exploratory testing in more detail.
Exploratory testing is often time-boxed, and test design and execution happen in parallel. This means that there is not a predefined set of test cases or test scripts to follow, and instead the tester uses their skills, experience, and intuition to go on a journey (or exploration) of the software.
Exploratory testing:
Encourages creativity and adaptability.
No predefined scripts; design & execute tests on-the-go.
Uncovers unexpected issues & real-world usability flaws.
Catches bugs scripted tests might overlook.
Leverage tester’s skills, experience, and intuition.
Exploratory testing is not:
Random testing.
Open-ended.
If we look at the definition of the word explore, it can mean “to examine or investigate something systematically” or, “to travel somewhere in search of discovery.”3
Framed this way, you can think of exploratory testing as a journey to investigate the software and to discover information, bugs, etc.
Test Charters
Going back to Explore It!, the author introduces the concept of test charters. I find test charters are a great way to get started with exploratory testing.
The basic format of a test charter is:
Explore [target] with [resources] to discover [information].
Explore [target]
With [resources]
To discover [information]
So, what the heck does this mean?
The target is what you’re exploring.
Resources are how and with what you’re exploring.
Information is what you hope to discover.
Let’s look at a real-world example. Let’s say we’re testing an email program, Gmail, Outlook, whatever app you use for checking and sending emails.
Explore composing emails
With various attachments
To discover file compatibility issues.
In this charter, the goal is to explore composing emails with various attachments to see if we find any compatibility issues. Notice it specifies composing emails and not sending emails. One key with charters is to keep them small and focused. For this one, you might time box it to 20 or 30 minutes. Once you’ve explored composing emails with various attachments, the next charter might be for sending. Or, you might make a discovery that causes you to change your plan. That’s ok, and that’s part of what’s great about exploratory testing. For example, if you find an issue right away with attaching images, you might want to go deep on that and explore more. Or you might find nothing interesting, and move on to sending sooner.
Variations
To get the most out of test charters, it helps to think about variations.
There are almost limitless possible variations to explore. These include things like user interactions, sequences, timing, data, configuration, and so on.
Examples:
Formats
Time / timezone
Dates
Zip Codes
File types
Numbers
Things you can count or select
Zero, one, many
Too many or too few
Some, none, all
Input
Numbers
Character sets (accented characters, special characters, etc.)
Navigation
Keyboard
Mouse
Sequence
Relative positions
Beginning
Middle
End
Size
Timing, frequency, duration
Language
Going back to our charter – explore composing emails with various attachments to discover compatibility issues – here are some examples of how we can apply variations:
Formats:
Different file types such as .doc, .csv, .jpg, .pdf, etc.
Size: varying file sizes from 0 to very big. Can you find the upper limit on attachment size?
Things you can count: attaching zero, one, many
Relative position: attaching at the beginning, middle, or end (if possible)
Input: file names with special characters
Navigation: is there more than one way you can attach a file?
Sequence: varying the order of addressing the email, writing the subject and/or body of the email, and attaching a file
Recording and Reporting Results
One thing to remember is that we do need some documentation of our work. However, this does not need to be as detailed as typical test cases.
First, you need to be aware of your actions while exploring. If you find a bug, you’ll need to be able to reproduce it and/or explain the steps to reproduce in a bug report. It can be helpful to record your sessions, that way when you encounter a bug, it’s recorded.
Second, a report on our exploratory testing charter – explore composing emails with various attachments to discover compatibility issues – might look like this:
Test Charter: Explore composing emails with various attachments to discover compatibility issues.
Time: 30 minutes
Results: Overall, minimal issues were found. Explored [list of file types] of varying sizes.
Not Tested: Audio file formats were not tested as part of this charter.
Recommendations: Follow up with a charter on sending emails with various attachments, including audio file formats.
Bugs: bug-1, bug-2, bug-3
Building Test Charters from our Nightmare Headlines
Returning to the nightmare headline game, recall your answer to this question:
What’s the worst news headline you can imagine for [product]?
Let’s say you wrote down “Gmail users unable to send emails.”
What are some test charters you could write (and test) that would help prevent this headline from coming true?
Conversation Starters:
Have you ever participated in exploratory testing, either as a tester or in another role? What was the most surprising bug or issue you discovered during your exploration?
Think about the worst headline you can imagine for a product you use daily. What kind of exploratory testing could potentially prevent this headline from becoming a reality?
If you're not in an engineering role, how do you see exploratory testing being applicable to your area of work? Can you think of a situation where a tester's mindset would have been beneficial?
Exploratory testing often requires thinking outside the box. Do you have any unconventional testing methods or strategies that have worked well for you?
I hope this dive into exploratory testing sparks curiosity and encourages you to explore the uncharted territories of your products. Remember, the goal of exploratory testing is not just to find bugs but to understand the product deeply and from new perspectives.
I'd love to hear your thoughts, experiences, or any nightmare headlines you've come up with. Feel free to share in the comments below or reach out directly. Let's keep the conversation going!
Stay curious and keep exploring,
Brie