Death by a thousand bug reports – Part 3: What makes for a perfect bug report

A perfect bug report is one that ticks all the right boxes. While this sounds boring and may take your testers a while to accomplish, it is actually their job and will save your developers a great deal of time moving forward.

In the ideal world, your developers should get well-prioritized tasks that are crystal-clear and do not need any further explanations. So it should take them as little time as possible to reproduce the reported bugs.

And this is perfectly doable. Your testers just need to take into account the following: who the bug report is intended for (the project manager, the test engineer, the developers, or the customers); the typical stages of your development cycle; your way of handling version control; the gist of erroneous software behavior; and what should be fixed.

Let’s analyze a bug’s life in the context of a typical development process.

First of all, bugs can occur at any step – during the design phase, gathering the requirements, prototyping, development, testing, or in production. Regardless of the step, a typical bug life cycle is as follows:

1. Somebody detects a bug -> the bug is new (or open)
2. The bug is under analysis or is being fixed (“in progress” status sounds very intuitive, doesn’t it?)
3. Resolution:

3.1. The issue was fixed, verified, and closed

3.2. The issue wasn’t fixed and is, therefore, reopened

3.3. Although the issue cannot be resolved, it gets closed (if it’s a duplicate, if the testing case is no longer relevant, or if it cannot be reproduced)

Pro tip #1: Everyone on the team should be familiar with the criteria used for moving on to the next stage.

Pro tip #2: Use your bug tracking system capabilities. The default fields suggest which information will help the Project Manager to correctly assess severity and which information will help developers to quickly figure out what’s causing the problem.

Here’s the best way to use your bug tracking system’s (such as JIRA or Bugzilla) default fields effectively:

1. Say the gist in the summary

Any software bug report needs a summary. That is, it needs a concise title that contains essential information about the bug. The title must provide answers to the following questions:

-Where?

-What?

-Under what circumstances?

Bad title example 1: Problem with car alarm

Good title example 1: Alarm shock sensors are off by default in new app version

Bad title example 2: Search is broken in application

Why it is bad: it’s hard to tell what exactly is wrong with search. Is the search field missing completely? Is it being unresponsive (you can’t place the cursor into the field)? Or is it redirecting users to an error page?

Good title example 2: [Friends tab] Search by name field is missing in iOS

Why it is good: the summary contains a “prefix” – [Friends tab]– that helps you quickly understand which software module is affected. It says what the problem is exactly and points to the environment in which the bug occurs. You can use labels if you want, but that depends on whether this is convenient for your team and whether they’re willing to fill out another field.

A good title is important at all stages of the bug’s life cycle, and it’s especially important for a new bug, as it helps assign the correct priority.

2. Describe the bug’s environment

As a client, I have not the slightest desire to check if my car alarm sensor values got reset for all mobile app installations for all possible devices, as mentioned in the earlier example. But I’m sure that it happened on my Samsung Galaxy S7 Edge Android 8.0.0. And I can see that not only by looking at my app settings (where the right toggle has suddenly moved to “off”) – I can tell by looking out of the window and seeing that someone has damaged my Toyota’s bumper.

Any software bug lives in a particular environment. It is found in a certain browser, operating system version, on a screen of a particular resolution, etc. While these are quite boring details, it’s essential to specify them when writing a bug report. If you don’t, a more conscientious programmer may waste a lot of time trying to reproduce the bug in the wrong environment, while a more negligent one may give up and say the bug couldn’t be reproduced after the first shot.

In your bug report, provide these details: operating system version, browser, the server and the build version (if applicable), and say whether the bug was present in previous builds.

For example: Android 8.0.0, Edge, qa/staging/production

3. Specify the affected and fixed version

*Affected version: the software version where the bug was discovered.

*Fixed version: having learned the affected version, we can easily figure out which version should be released to users and also help ourselves with Release Notes preparation in the future (if you know what I mean 🙂 ).

You need to decide who is going to be responsible for entering these versions and how you’re going to use this information (say, for writing release notes). If the person who closes the bug does not specify the fix version, the fix may be lost and become impossible to find.

Tip: It’s convenient to search for fixed bugs by software version if your testers set up a respective filter in the bug-tracking system.

4. Write logical and exhaustive “steps to reproduce”

Sometimes the bug may be real, but the steps one should take to reproduce it may be not enough or confusing. Hence, developers may have a hard time reproducing the bug.

It’s important to be a bit of a maniac and act like a witness at the crime scene when describing the details of how you encountered the bug. If you describe your actions in the wrong sequence, or if you point the QA/developer to the wrong screen in the app, this may affect the bug’s reproducibility.

Things that make reproducing the bug much easier:

• The steps are described in simple words (without jargon) and are clear even to a non-techy person.
• All entities are called exactly the way they are called in the app (for example, if you write “Configuration” or “Settings” instead of “Preferences”, it will be much harder to reproduce, fix, and verify the bug – especially for new employees).
• The steps are an ordered list with numbers.
• The steps are concise and trigger the bug in the shortest way possible.
• Your QA engineer uses formatting wherever it makes sense (they make use of different fonts, bullet points, text color, etc.).
• The bug report contains all the necessary screenshots and videos.
• If the bug is triggered by a specific type of a user, the user’s role in the app is specified (e.g., registered user, guest, or admin).

It’s also nice when there’s sufficient rapport between your QA and development team and they are aware of each other’s preferences. To give you a personal example, I once worked with a programmer who preferred to look at the screenshots and rarely read the bug description. If the screenshots were confusing, he’d have difficulty reproducing the bug.

Be mindful of the programmers’ time and avoid overwhelming them with endless screencasts (anything over 10-15 seconds is too long). The problem should be easily understandable at a glance, without much thinking or studying the attachments.

Steps to reproduce play an important role throughout the bug report’s life cycle. Some teams choose to re-open closed bug reports (instead of creating new ones) when there have been minor changes to the case.

5. Define bug priority and/or severity

Priority and severity are quite different things. However, you can say that both serve to determine the level of discomfort your users will experience if the problem remains unfixed. Priority is normally set by the project manager – a person who can assess how urgently the error needs to be fixed. Severity reflects the damage the bug causes to the application (traditionally, it’s set by the bug report author or Head of QA).

One thing to remember is that priority is usually linked to your business goals. For example, an issue with the signup process may look high-priority compared to an issue with the background color. However, if we’re talking about an internal app used by a finite number of users, and they all have already signed up, the issue with the background color may be more urgent if the background was changed to some irritating hue during the redesign, for example.

So, it depends. Quite often, priority and severity become one hybrid concept. And that’s okay, as long as everyone on your team knows what each status means and finds the classification convenient.

The main point is that it should not be a mere formality – the priority/severity status should in fact help prioritize tasks.

There are some generally accepted statuses that pretty much everyone understands:

*critical

*medium

*minor

Sometimes statuses like “blocker” and “trivial” are also used.

Possible reasons to escalate priority/severity are:

• the problem affects a large number of users
• there is no workaround for the problem
• the problem breaks key functionality
• the app owner may intervene and say the issue is insignificant (or vice versa), since they know the business side better than the developers.
• paid vs free features, paying vs non-paying users

Early in the bug life cycle, it’s important that the error is easily reproducible. Bug reproducibility often affects how correctly you can prioritize the issue. For example, if the issue appears only if you enter the query, wait exactly 17 seconds and hit Search – it may not be a critical issue compared the situation when you get an error all the time.

6. Describe actual vs expected behavior

While describing the bug, specify how the application would act if it didn’t exist. This information is called expected behavior.

It’s important that expected behavior is described precisely. Wherever you can, link to “oracles” (mechanisms of identifying expected behavior) you used to learn the desired test result. These may be your requirements, an older program version, asking a project manager or a business analyst, and possibly other methods.

The only time it’s useless to describe the expected result is when you use generic phrases like “it should work correctly.” What is “correctly” in this case? Your expected result should be a list of concrete criteria that should help your testers quickly determine if the bug has been fixed or if the issue should be re-open. To determine whether you’ve described the expected result well, cover the ticket’s description and summary with your palm and read just the expected result. Does it make everything clear?

In conclusion

An ideal bug report should not be overly verbose, but on the other hand, it should contain enough information for the programmer/client/whoever to quickly identify, prioritize, fix, and verify the bug. The main idea is to satisfy the needs of the person for whom the bug report is intended.

Happy ploughing through the bug reports! We hope you will now find it much easier to do 😉