Go Play, Find Bugs

One of our Senior Testers wrote a blog detailing how she found an obscure bug. When I was a software tester, I often said that – even though you spend a large amount of your time writing Test Cases and running them; the majority of bugs I found were actually performing actions off-script.

The reason for this is that if you have a certain requirement, the developer writes enough code to pass that requirement as it is written. A better developer may even write some kind of automated tests to cover that scenario to prove that it works, and it won’t break in future. Therefore, running a manual test that describes that behaviour won’t find a bug now, and it won’t if you run that test in the future (during regression testing).

Being able to freestyle your steps means you can come up with obscure scenarios and experiment, and do way more testing than you would following a strict, heavily-documented process.

This was the main problem I had working as a Software Tester. Managers wanted the documentation and if you told them you had been testing without it, you sometimes got told to stop, or spend time writing Test Cases for ALL the additional scenarios you came up with. All that does is encourage people to be lazy and do the minimal amount of testing, which consists of just the basic scenarios.

You also get into scenarios where if there is a bug in live, it’s easy to make stupid claims in hindsight. I remember a colleague being absolutely furious with the criticism. They had done loads of testing but there was a bug in live in a very specific scenario:

“I’m disappointed in the level of testing” – Test Manager

Here is our Senior Tester’s blog:

I found a deliciously elusive bug last week. The feeling a tester gets when this happens is joy at your good luck, like satisfaction at solving a fiendish puzzle, and relief at preventing harm. We feel useful!
 
The bug was to do with online visibility data. My team is implementing the ability to right-click items and set Online Visibility. Sounds simple in theory - but data is complicated and the code base is large.
 
How was I going to approach this? It was an intimidating piece of work – and I was tired. My normal process would be to come up with some ideas for testing, document them, then interact with the product, make notes, fill out the report. But that day, I just couldn’t face doing the documentation and planning I would normally do before the testing. I decided to just test, not worry too much about documentation, and have fun.
 
I sought out a Record with a rich data set and played around, selecting things, deselecting them, selecting parent entries, child entries, single entry, multiple entries. I didn’t have any defined agenda in mind except to explore and see what would happen.
 
One minute in, I was rewarded with a beautiful crash!
 
I hadn’t taken a note of my steps – but I knew I could probably find the path again. I set up and recorded a Teams meeting with myself, as I didn’t want to have to pause to note down every step I took – that would take a long time and risk my mindset changing to a formal, rigid, structured view – which I didn’t want. I needed that playful freedom.  The system crashed again! As there were so many variables at play, I didn’t know what the exact cause was, but I now had proof that it hadn’t been a magical dream.
 
I spent the rest of the afternoon trying to determine the exact circumstances in vain. I spoke to the programmer, and showed him my recording. He took the issue seriously, and tried to recreate it himself. We both struggled to do so, and decided to wait until the morning.
 
The following day, we got on a call and went over the recording again. What exactly had I done before the crash? I had selected the parent entry, then two child entries, right clicked but not changed anything, deselected the parent, selected another child, unselected it, selected a different child, selected the parent again and then right clicked and changed the Online Visibility -  crash. We tried that again on the developer’s machine, on the same type of report, break points at the ready. Crash! Got it!
 
The developer eventually narrowed it down to two conditions: child entries had to have a GroupingDisplayOrder index beginning with 1, and the user had to select the parent entry after its child.

It seemed sheer luck that I had found this. But was it luck? No. It was luck by design – I had created a rich data set, and done lots of different actions in different orders, been creative and diverse in my testing. And it had only taken a minute to yield results!
 
So what did I learn? Reflecting, I noted my preference for highly structured documentation – of tables with colour highlighting, documenting each test in high detail, strictly in order, changing one condition at a time. The result of this was that I tested in a highly formal, structured way to fit the documentation, and only did informal testing as an afterthought. And yet I had most often found bugs during the informal testing! 
 
I had made a god of documentation and lost sight of what mattered most. If you need me, I’ll be testing. And trying not to make too many pivot tables.

What Are Software Testers Really?

The same tester once came out with this quote

“testers are ultimately critics. Developers are artists. Testers are there to give information. What you do with that information is up to you.”

That’s quite an interesting perspective. I think it mainly comes from the idea that Testers can find 10 bugs but maybe you decide that you will only fix 6 of them, a few you might fix later, then 2 you think aren’t a problem, or so unlikely to happen – it’s not worth the effort and risk to fix it.

“we are the headlights of the car, driving into the darkness”

Software Testers In Game Development

Another interesting discussion on the role software testers play is from Mark Darrah who has worked on games like Dragon Age Origins. He does seem to agree with this idea that the Testers are merely critics.

Mark Darrah – Don’t Blame QA

When encountering bugs during gameplay, it’s often misconceived that the quality assurance (QA) team is to blame. However, it’s more likely that the QA team identified and reported the bug, but it remained unresolved due to various factors. For instance, a more critical bug could have emerged from the attempted fix, leading to a strategic decision to tolerate the lesser bug. Additionally, project leaders may assess the bug during triage and conclude that its impact is minimal (affecting a small number of users), opting to proceed with the game’s release.

Such scenarios are more common than one might expect, and they typically occur more frequently than QA overlooking a bug altogether. If a bug did slip through QA, it’s usually not the fault of any single individual. The bug might result from a vast number of possible combinations (a combinatorial explosion) of in-game elements, making it impractical to test every scenario. Your unique combination of in-game items and actions may have simply gone untested, not due to negligence, but due to limited resources.

Complex game designs can introduce bugs that are difficult to detect, such as those that only appear in multiplayer modes. Budget constraints may force QA to simulate multiplayer scenarios solo (a single person playing all four or eight different players at once), significantly reducing the scope of testing. 

Furthermore, bugs can be hardware-specific, and while less common now, they do occur. It’s improbable that QA had access to the exact hardware configuration of your high-end gaming setup.

The term ‘Quality Assurance’ (QA) can often be a misnomer within the development industry. While ‘assurance’ suggests a guarantee of quality, the role of QA is not to ensure the absence of issues but to verify the quality by identifying problems. It is the collective responsibility of the development team to address and resolve these issues.

Understanding the semantics is crucial because language shapes perception. The term ‘QA’ may inadvertently set unrealistic expectations of the role’s responsibilities. In many development studios, QA teams are undervalued, sometimes excluded from team meetings, bonuses, and even social events like Christmas parties. Yet, they are expected to shoulder the criticism for any flaws or bugs that remain in the final product, which is both unfair and inappropriate.

Developers, it’s essential to recognize that QA is an integral part of your team. The effectiveness of your QA team can significantly influence the quality of your game. Encourage them to report both qualitative and quantitative bugs, engage with them about the complexities of different systems, and heed their warnings about testing difficulties. Disregarding their insights can lead to overlooked bugs and ultimately, a compromised product.

For those outside the development sphere, it’s important to understand that if you encounter a bug in a game, it’s likely that QA was aware of it, but there may have been extenuating circumstances preventing its resolution. Treat QA with respect; they play a pivotal role in maintaining the game’s integrity.

Remember, a strong QA team is the bulwark against the chaos of a bug-ridden release. Appreciate their efforts, for they are a vital component in the creation of seamless gaming experiences.