During lunch today I was pondering (not daydreaming, mind you) various aspects of unit testing and Test Driven Development (TDD) in general, and it struck me that TDD seems like it is, inherently, a good fit for developing software projects that one is going to manage using a system as described in the book "Getting Things Done".

For instance, here is a quote from Martin Fowler describing what he likes about TDD (the full article can be found at Test-Driven Development (http://www.artima.com/intv/testdriven.html) ):

Martin Fowler: The thing I like about taking small steps and writing tests first is that it gives me a simple to do list of the things I've got to do. At each end point I have a certain amount of closure. I say, OK, this stuff works. Check it in. It's all there. It does what it does and it does it correctly.

This seems to go along nicely with having the kind of well defined next-actions described in "Getting Things Done". Of course, it is still possible to define discrete, programming related goals for oneself without using TDD, but unit testing seems like a particularly good fit for this way of thinking.

How many of you use TDD in your overall software development projects, and, more specifically, how many of you use it for game related projects? Why or why not?

My day job is doing "Automated QA" for a major software corporation. We've been using the XP methodology, and while I'm not a total eXtreme Programming proponent, the focus on testing REALLY opened my eyes and turned me into a convert - to the point where I'm creating the same kind of tests for my game development projects.

It's got plenty of additional benefits besides the big one you mentioned. It can encourage you to write cleaner APIs into your code (so you can call them not only from the game, but from your testing suite) - this is a pretty low overhead cost for making more maintainable code.

It also aids IMMENSELY in regression tests. How often have you fixed one bug only to find you broke three other things? A comprehensive test suite will catch 90% of these problems. This is of great value during development, but of even greater development when you are creating post-release upgrades to your software.

As far as it GUIDING development - it can. I'm not quite as much a believer in that as Fowler & the other XP advocates, but it can be handy. With games there are a lot of things that aren't easily machine-readable (like sounds, graphics, etc.), but it should be easy to test out things like collision, AI, state change triggers, etc. This CAN help you "blueprint" certain major operations of your game, and help you keep track of how much is left to be done. Sometimes it's easy to get caught up in just the graphics & basic interaction, and we forget that there's a ton of gameplay bits that need to be built. I *do* find it's easier to do development after I've gotten the interface defined and the stubs created - whether this was forced via testing or some other methodology is irrelevant. That's just an easy way to tackle a big task.

Error-checking during development is another GREAT area for tests, particularly when working with a team. You can run checks on all the content (textures, models, level files, etc) to make sure they are valid BEFORE being packaged in to the latest development "build". I've seen some automation where Excel spreadsheets were auto-generated every night with a list of all "bugs" found (and bugs can include missing files that have never been created, but are expected at some point). The bugs can include things not parsing correctly, textures & art of incorrect format (like not in dimensions that are a power of 2), & so forth.

Finally, since bugs cost more to finish late than early, you really get some savings by getting some tests written from the get-go before you actually write the code to make those tests pass.

Used correctly, I think you can really speed up development this way.

Coyote,

All excellent points. I'm encouraged to hear that unit testing is also working well for you in your game development activities. In my own experience, TDD has been a relatively simple change in coding habits that, like many other things in life, required a period of adjustment and frustration, but that has ultimately led to better experiences and productivity.

I am going to ask a moderator to move this thread to the technical forum. It was a close choice between this forum and that one when I posted, but I think it makes more sense there.

I have a sinking suspicion I used to work for the company that now employs Coyote.

I now work for a contracting company that does a lot of contract work for them ;)

Anyway, at work we've been experimenting with Continuous Integration as a test paradigm.

We start with writing test case matrices, but we don’t actually build the tests first and use them to drive development.

However the written cases do serve to drive the requirements for the actual app.

I spent today working with an automated build tool called NAnt (or Ant for .NET). It's pretty slick, one of it's neato features is that you can use NUnit or CPPUnit to automate your unit testing via NAnt's build scripts.

Combine that with a tool like CruiseControl and a repository like CVS and you can have automated builds with full unit testing every time someone checks something in. In this way your application is continuously going through integration testing. It's a pretty cool way of doing things.

This is the first time I’ve been at the foundation stages of making a build environment, and I must say it’s pretty fascinating. I’d love to set up automated build scripts that I could kick off from RainfallStudios admin site by clicking a button.

I've become somewhat unenamored with XP lately. I'm in the final stages of a largish project that I did with XP, and I was fairly disappointed with the process. I think that XP has the right ideas, just taken a little too far. That said, unit testing is the one thing that really stood out, and I will continue using that. Probably not in a "test first" manner, but just knowing that I have hundreds of unit tests that all work without a hitch squelches a lot of anxieties.

I'll have to read some of the Fowler stuff. What you describe is basically the way that I've been working. From my current user story (could be a feature or whatever) I generate a set of tasks (in the GTD sense) and tackle them one at a time. The only thing I'm not really sure about is whether or not to generate all of the tasks at once, or just figure out each new task as I finish the one I'm on.

@Dan:

I have actually found writing tests first to be liberating. As I said, it took a while for it to become a habit, and the transition into it was frustrating at times, but overall it has been a big win.

@Siebharinn:

I have not yet participated in an XP development situation, therfore I have no real basis to say whether I like it or not. However, from the outside (read: this is just my opinion), much of XP, including the focus on unit testing, seems to be designed to keep the quality of programming consistent across non-homogenous groups with regard to talent and experience. Thus, it is, out of necessity, concerned more with programmers of lesser experience or "natural talent" within a team than it is with the best programmers in that team.

This might actually cause tension and/or dissatisfaction because many programmers could either feel stifled or burdened by duites for which they are a poor match. This tension, which could (lots of coulds, I know) then lead to reduced productivity or team unity, might even be overlooked by management because the quality of the code would still be relatively consistent according to some predetermined, middle-ground metric. It is like playing a numbers game... you are more likely to have a greater number of average (compared to some standard outside of your team) programmers than exceptional programmers, so if you can maximize the effectiveness of the average programmers, and you have enough of them, the overall increased productivity will probably outweigh any decreases in the effectiveness of your best programmers. Of course, this assumes that the metric you are using for programmer effectiveness naturally falls in the middle of some range that goes from the production of shoddy code to the production of ground-breaking, "every hacker wants to steal it" code, so what I am saying may not even be applicable across a wide variety of situations that lie outside of my own experience. Still, this does seem to be a valid point to me, peering in from the outside, even though XP proponents would likely characterize these things quite differently. It is also possible that I am completely off the mark, but that just doesn't happen very often ;)

I am curious about specific things in the XP process that didn't sit well with you... are you willing to provide more detail?

I use the test sometimes, or test as being written.

If I find myself testing some code, I might as well do it in a unittest. The idea that the framework should make it easier(or as easy) compared to any other way you can do tests. So if writing unitests seems like it is adding up front work, rather than saving you up front work, try fixing the framework.

If I ever find a bug, might as well add it to the unittest as it is an easy way to test for it.

I also do visual tests, and input tests. These are not automated, but can require user input. Good for writing up graphical effects etc. I don't use the unittesting framework, but the application frame work to make a mini application.

Tests are good for multiple platforms too(including multiple video cards). It is much easier to find the source of problems if most of your games effects have seperate tests for them.

Unit tests are also good when you change library or language versions. Especially when you are not in control of those libraries or languages.

Another good use of unittests are to profile things. Add a test to make sure that a certain function only takes eg 5ms. Then you check that it beats that. If it doesn't then you spend more time optimizing.

If you find something running slowly you might as well write a unittest for testing the speed of something.


Have fun!
http://www.holepit.com/

My problem with XP centered around the fact that the project seemed to take longer than it should have, and I got on a lot of people's nerves in the process.

The lack of a clear design meant I spent a lot of time refactoring when I found a better way to do something. I rewrote some modules three times to accomodate new functionality. I think that I spent *more* time backtracking than I saved in not doing design. Spending a week or two coming up with a functional and detailed spec would have saved months of refactoring.

Lack of communication also hurt. Since there is no formal written docs, the only way for someone to get up to speed on something is for me to sit down and explain it. That ended up being a big time sink. Plus, miscommunication caused some problems. With no document to refer back to, people would act on what they thought they heard (or what the speaker thought he said).

There was a great deal of difficulty explaining to people outside of our group where we were and what we were doing. Magement can't read UML specs, but they can read a functional document and have some idea about whether or not it will meet their needs. One board member that I had to interact with frequently referred to my user story cards as "flash cards" and gave all sorts of hints that he thought this was a fly-by-night operation. It didn't help when a problem was discovered and it was pointed out that we could refactor ("What does that mean?" "It means we rewrite that part." "Why didn't you write it correctly the first time?").

Testing was a bright spot in the process. But even that wasn't perfect; bad tests can run successfully and still not be helping you.

XP provides answers to some fundamental problems. The difficulty lies in the fact that many of the answers are...well...extreme.

This was an educational process for me. (There are no failures, only chances to learn. I learned a lot.) There are definately things that I will take from XP, like testing, very short iterations, and providing business value as soon as possible. But I want a more formal system.