Hi

After getting into a pointless discussion about testing I thought I'd post some thoughts here as a fresh thread as it raised a few interesting thoughts.

Coming from a commercial testing background I'm quite methodical when I sit down to test. I grab a pen and paper and go through the game as thoroughly as I can making notes as I go.

I also make every effort to do things that the coder wouldn't have thought I'd do. Pressing keys all over the place, going to places I shouldn't be, hitting escape when it's halfway through something, just generally trying to cause chaos and being as annoying as I could. It's this that often reveal the juicy bugs.

Trouble is I find it difficult to test my own products to the same standard as I tend to subconsicously avoid doing things that may break it. Does anyone else have this problem?

As BigFizz develops multiple projects at once because we have several coders it's not too bad as I just test our other games. We also use a bugs database so we can keep track of what's been fixed and comments. Our classification system is as follows.

(A) - Show stoppers. These crash the game, make it hang and have to be fixed as soon as possible.

(B) - Serious. These are ones that really should be addressed before final release. These can be anything from the debug text on screen to missions not being completeable. Even though the coder knew that the debug text would have to be removed it's in the database so they don't forget at any point.

(C) - Minor. These are things that aren't too serious but if there's time or they're quick it wouldn't hurt looking at. Things like fades not quite fading down totally or artwork that looks cropped or missing sfx.

(D) - Comments/Suggestions. These are basically, "it would be good if it...", "how about if this did this instead?", "would it be better if this behaved like this", "that graphic looks poor". And so on. Basically just personal preference or ideas.

The class isn't meant to be how severe the bug actually is but more of what level of importance should the team give it.

The tricky part is knowing what to classify each item as. It's obvious if the game crashes when you click on a button it's an 'A'. But what if the coder has put some 'adult' artwork to liven his day up while waiting. For me this is a 'B' - it has to be fixed before final release. While it's not strictly a bug, after all it's there intentionally it needs to be addressed.

I've had testers who have an idea for a new weapon and they've listed it as a class 'A'. When asked why their response was, "I really want it in the game so I thought you'd do it if it was A". That's how not to classify them.

The other thing is the personal preference part. Basically anything below A could be classed as personal preference if the game continues to function. "I like the artwork cropped", "I like how it doesn't fade right down", "I like how it doesn't play a sound effect". etc

Each bug has a status attached to it, being...
Open - Bug is still there.
Fixed - Bug is gone and has been confirmed in test.
Ignore - Not going to pay attention to it - may come back to it later.

And there's also comment fields that can be added to. As quite often you need clarification on what the tester meant. ie, and this was a real bug, "when leaping the catamaran from water the textures to the walls then change" - (this earnt him the nickname 'yoda' )

So. Any suggestions on improving classification? Any ideas on improving the testing process? How do you handle testing when it's your own game? Is it worthwhile using an external testing company? Any nightmare stories of testing or testers? Is this post too long? :)

Scott

I used to work for a major game publishing house as a tester and if memory serves me right, the paper I had to fill out have these catergories:

- bug class: A, B C (usually determine by testing leads).
- bug category: graphic, sound, etc...
- repeatability: % of times it occurs.
- bug description: what happens.
- steps to reproduce bug: list steps you have to do to repeat the bug (keep it short and simple).
- result: what happens at the end of the final step.
- expected result: what did you expect it to happen.
- comment: tester comments.

and for regression testing the status of the bug breaks down to:

- confirmed: bug's still in.
- fixed: bug's gone.
- confirmed revised: bug came back.
- still testing: still testing.
- not testable: can't test it.

hope this helps. :)

I used to work for a games developer as programmer (and still do freelance work).
Each bug report would consist of;


Bug type: Bug, Suggestion, Implementation, Design Change
Bug subtype: None, Display, Sound, Crash, Spelling, Programing, Data Problems, Known Bugs, Menu System
Priority: Low, Medium, High
Status: Pending, Fixed, Suspended, Re-Test, Partially Fixed, Known Bug, Duplicate, Next Version, Patch, Verify, Closed
Bug Descripion: What happens and how to reproduce it
Test Comments: anything else the tester has to say, often used if the bug isn't fully fixed and is bounced back to pending
Developers Comments: important information from the developer


hope this helps! ;)

to be honest I don't see anything really wrong with your system, testers putting in suggestions as a bug is just part of dealing with inexperienced testers.
I always found that the best places always had the Head of Test filter through any new bugs before they went anywhere near the developers.
It's a small thing but it saves a lot of time and tempers :D

Hey, good thread topic. For my games I write down the bugs in a text file. Right now for one game I have over 200 documented bugs: some fixed, some not, with different priorities, etc.

Clearly I'm outgrowing a simple text file. What I need to do is move to a real bug tracking tool, but I really don't need the aggravation of setting one up.

I've looked at Gnats and Bugzilla. At work I use Footprints but it isn't free. Anyone got any suggestions for a bug tracking tool that's really easy to get going ?

Thanks for the replies. Class D was pretty much there for suggestions, was just annoying when suggestions were put into class A bugs. Even more so when the producer would ask how many class A's are there and the tester would say there's 6. One stressed producer later and we're having to explain that there's actually none just testers being stroppy.

I remember having to pull a couple of all nighters to hit a deadline, burnt the CD (it was PS1), tried it, handed it to the tester and went home for much needed sleep. Got back in the next day and got absolutely barracked by the manager for it not working. I went over to the machine with it in. Switched it on, sure enough it didn't boot. Opened the PS1, they'd put it in upside down. Aerrghhhh...

You could try writing a quick database yourself and just write it to your own file or an SQL database. Shouldn't take much effort really and you can the fine tune it to your requirements.

At one place I worked we just used a word doc, then we switched to access and generated our own layout, this was probably the easiest system we used. Later on a head of testing with lots of experience was brought in, he wanted to switch to Excel for some reason. Never quite figured out why.

I agree with that, Access is very good for making a reasonably strong bug tracking system which everyone can use easily.

Excel? that's just nasty!

Thanks for the replies. Class D was pretty much there for suggestions, was just annoying when suggestions were put into class A bugs. Even more so when the producer would ask how many class A's are there and the tester would say there's 6. One stressed producer later and we're having to explain that there's actually none just testers being stroppy.



Clas D seems to cloud the testing process, I suggest you leave it out. Or better still have a seperate doc, headed Apprais al nad suggestions. Give this to them after they have completed testing, after all it's a totally different entity to testing.


I remember having to pull a couple of all nighters to hit a deadline, burnt the CD (it was PS1), tried it, handed it to the tester and went home for much needed sleep. Got back in the next day and got absolutely barracked by the manager for it not working. I went over to the machine with it in. Switched it on, sure enough it didn't boot. Opened the PS1, they'd put it in upside down. Aerrghhhh...

ROFL

Alfie

Trouble is there's times when testers suggestions are very useful. Previously we've had lots of good ideas, suggestions, UI improvements come through the D class. We'll probably keep it for now as the more feedback we get the better it is. It only takes a few seconds to read it and ignore it anyhow :)

With you getting those benefits you are right to keep it in.

Alfie

A - crash/gameplay/Text Standard (Sony,PS2, XBox, own company standards)
B - Gameplay/Text/Graphic
C - Graphic
D - Comments

Bug Summary: Game crashes when User presses the A Button in Options Menu
Severity: A,B,C,D
Reproduction: 10 out of 10, or use a once - %100
Status: (Open, Closed, Duplicate, As design, Known Shippable/not a bug, In Regression, Can not test, Can not reproduce, on hold)
Priority: (low, medium, high)
Build: (Build No.)
Location: (What area does the game occur)
Test Plan: (What test plan does the bug occur in)
Assigned to: (Who is the bug currently assigned to)
Submitted by: (Bug author)
Keywords: (Words that can help find the bug)
Notes: (Attach images, avi, crash dumps, logs, dxdiag...etc.)
History: Keeps track of who the bug is sent to, any new notes added, and last modified..etc

There seems to be two main styles of writting the acutual bug. One is the list method, and the other the if/then method.

[List Method]
Game crashes when User presses the A Button in Options Menu

Steps to Reproduce:

1) Start up the game
2) In Main Menu > Options Menu
3) Press A Button

Results: The game crashes.


[if/then]
Game crashes when User presses the A Button in Options Menu

On startup, Main Menu > Options Menu, if the player presses the A Button, then the game crashes.

In level 5, Boss Fight, with difficulty on Easy, if the player...etc

----
In QA, the bug is submitted by the tester/developer and is sent to another tester/lead who is in charge of checking the bug. The bug is then sent off to someone in development who will then assign it to a developer that will eventually fix the issue.

Normally at the beginning of QA they will allo every type of Severity, A,B,C,D, as the development progresses, they will not allow minor C, and D bugs to be submitted. Near the end of the process, they only want gameplay B, and A crash bugs.
----

The most common bug database I've seen has been created using FileMaker Pro. I've also heard of Bugzilla, DevTrak, and custom developed bug DB.

I think when we did work for Sony they had (at the time) a bug system almost identical. I think they later changed "Class D" to simply "Suggestions" --- since it was pretty frustrating to refer to all the stupid suggestions by testers who couldn't find any real problems with the game "Bugs." :)

I found that methodology worked pretty well. I've used some others, but it seems the more categories you put in, the more difficult bugs are to classify.

What we did was have the producer over the title work with the test lead for re-classification of bugs that didn't make sense. We had some testers who were notorious for doing just what you said... labelling purely cosmetic issues as "Class A," for example.

Some more classifications you may want to add for bug status:

* "Cannot Reproduce" - use this to send a bug back to testing to try and get them to figure out steps to reproduce the bug. Don't abuse this one as a developer, but this is a handy one to use when the testers get sloppy with their bug reports... like "I walked through a wall." HELLO? Which wall, please?

* "Fixed - Verify" for sending a fixed bug BACK to testing so that they can verify it really is fixed.

* Make certain you have a field for version identification, and make sure your testers are hard-core about filling that field in. We had so many problems with testers using old versions of software - especially for regression tests. It was amazing how often they'd claim bugs weren't fixed because they were using a week-old version!

* Have a field not just for "comments", but "steps to reproduce." 90% of the challenge of fixing a bug is finding exactly where it is... let your QA people help you in this respect. They can often do things on-demand that a developer just can't imagine.

I've had experience with this as well. Since I'm a developer and my wife is a software tester. We both worked for a software company that I have to agree put alot of importance on quality testing, and it showed in the final product.

One thing we did, that I didnt' see mentioned was that we used a combination of priority and severity. The bug may have a "high" priority, but it's severity may be "cosmetic". This allowed our developers to prioritize their days and be much more productive. Another example, may be that you have a crashing bug that is very nasty, but it's very difficult to reproduce. That would have a priority of "Low", but a severity of "Critical".

Priority categories:
Fix if Time
Immediate
High
Medium
Low

Severity:
Critical
Major
Routine
Minor
Cosmetic
Other

In any system, what's most important is that it works for you instead of against you. If you're having problems with testers filing suggestions as class A bugs, then change the terminology so that everything is an issue instead of a bug, and seperate category and priority. That way, they can file their suggestion and still mark it with priority 1, but at the end of the day, the project manager knows it's a suggestion, and can just ignore it.

After getting into a pointless discussion about testing...

I thought you provided some great/thorough feedback in that thread. I hope you'll kick the tires on our upcoming title, when we have tires to kick. :)

During our early tests for Inago Rage (http://www.dejobaan.com/inago/index.htm), we had two ways a tester could classify a ticket -- it was either a suggestion or a defect. I'd then go in and summarize it, (to make sure I understood what was going on), and assign a severity (for bugs) or consensus (for suggestions). I feel that ticket severity should be set by the designers or test leads rather than the individual testers, for the sake of consistency.

At my day job we have very comprehensive testing procedures, testing and issue management software, ISO quality control.. the works! Well.. given the commercial scale of the projects we undertake, such a heavy handed approach is practical.

As I'm working on the game on the side, I try to keep things as simple as possible i.e. I log bugs in the IDE's task bar together with TODO's, HACK's (temporary implementations) etc. To me, the important thing is to keep track of bugs and solve them ASAP before doing new features (usually).

I also plan to have extensive play testing both within the team and also to involve external testers although it will be an informal affair.

My take on the subject of testing, is that if you have the resources, plan for it and structure it accordingly... but do not go beyond your means!