I don't know if this is pretty common knowledge, but I had no idea that movie production companies do focus group style movie screenings before releasing a movie. I heard something along the lines of a premiere but I didn't know they actually do focus groups to see if people like the movie. I just watched some "bonus" material on the Count of Monte Cristo DVD where the Director and Editor of the movie were talking about the cut scenes.

They mentioned that they shot multiple versions of one scene, and the audience didn't like the one they used in the screening. They didn't know that there were other versions obviously, but they just mentioned that it should be different, which happened to be how one of the other versions was shot. So they changed it to the other version, and in the next screening the audience reaction was much better.

How is this related to games? Well, all kinds of companies with all kinds of products do focus group product testing (McDonald's for new kind of burger for instance), and obviously business software companies conduct focus groups, surveys, interviews with their clients to make sure the end product is what they want. As game developers we don't have a single customer, but I think that doing focus groups with members of your target audience before releasing a game would make a lot of sense for game companies.

Not for indies so much, since focus groups do cost a hefty amount of money (if you want to do it professionally anyway) in relation to the income you get from small games, but large publishers can benefit from this a great deal I think. They do all kinds of QA of course already, but how many of them conduct focus groups with users to find out what parts of the game are good, what parts need to be changed, etc. while the game is in development? And large scale beta tests don't count because the feedback level is minimal (mostly bug reports) and the publisher isn't planning to change much once the game is in beta anyway. I think trying harder to "give the people what they want" would make the industry a lot more profitable.

All large companies do focus groups. It's part of marketing and research. I was inited to a 3D0 focus group about 5+ years ago on Battle Tanks. My friend went to a Sim City 4 focus group and was paid $100.

You basically play the game, fill out a form and talk to some of the developers. In the case of my 3D0 focus group, the game was already finished, but they wanted to find out what should go into a sequal. I got paid $20 cash and all I could eat pizza and coke :).

I would think it's even easier for indies to do focus groups. Just release an early demo, post it on a message board such as here or any game forum, and ask people to reply for suggestions.

John

Oh so they do it to an extent. Ok I didn't know that. The impression I get from the retail game industry is that they do whatever seems cool to them, not what the users would really want. That's why we get so many crappy games coming out. But if they do it even somewhat that's a good sign.

In The Invisible Touch (http://www.amazon.com/exec/obidos/ASIN/0446524174/davidrmsoftwa-20), by Harry Beckwith, points out:

1. "Research changes its own results." Also known as the "Heisenberg Principle". People who know they're being watched behave differently.

2. "Researchers tend to find what they are looking for."

The book really beats on focus groups as a useful marketing or research tool.

-David

Sometimes (http://www.penny-arcade.com/view.php3?date=2002-10-18) things can get out of hand though. :)

During the development of SimCity3000 and The Sims, Maxis used a number of focus groups and "kleenex testing" to see if the game was on target. [Kleenex testing is where you find a group of people that have never played your game (previous versions included) and ask them for their impressions of the game after playing it for a while. It's "kleenex" because you use them once and then never again.] The results of these groups did change the games' designs to some extent, but the bulk of the changes focused on the user interfaces (making them more intuitive, simpler, etc). Usually, by the time you have enough of a game to bring in focus groups, it's too late to make any major changes to the game (unless your game is utterly awful).

"The impression I get from the retail game industry is that they do whatever seems cool to them."

This is pretty much true. I hear people saying all the time that the industry needs to pay more attention to market research and the needs of the consumer. As it is real work even on usability testing (ie. making sure people can figure out the interface) is a pretty new thing; the stabs at focus groups and other market research is laughable. At several companies I've had people tell me that their "market research" mostly amounts to scoring the answers from people registering their games.

And don't even try to tell me that the people who write TV ads for games pay any attention to the market.

Before I got in to games, I was a "user-centered design" consultant (the irony for me is that I got out of this in 1994 because I thought it was becoming an assumed part of product development, and thus no longer a good consulting area!).

UCD is up-front; usability is after the fact (well, that leaves out a lot of nuance, but that's the basics). You can test out product concepts -- including in terms of games the theme, gameplay, interaction methods, UI layout, art style, player goals, etc. -- long before the game is done, and even before you move into production. In fact the single biggest difference between games and other forms of software is that in other software you have to discover and conform to the user's goals and tasks. In games you have to think up those goals and tasks and make them compelling enough that the user will want to do them -- that's a much taller order to fill.

The earlier you begin bringing potential users into the process, the more likely you are to have a positive effect on the product's development. But at the same time, the earlier you do it, the more difficult the data is to interpret (and by far, the easier it is for an executive with a prejudiced opinion to make the data fit that opinion rather than vice versa).

If you wait until the game is basically done, your ability to change things is greatly constrained, and thus the likelihood that you'll really listen to customers who say "this sucks" is greatly reduced -- they aren't giving you data you can do anything with. Trying to fix a product with late user-testing is known as "putting lipstick on a cow." No matter how you dress it up, it's still a cow.

Originally posted by Fenix Down
Oh so they do it to an extent. Ok I didn't know that. The impression I get from the retail game industry is that they do whatever seems cool to them, not what the users would really want. That's why we get so many crappy games coming out. But if they do it even somewhat that's a good sign.

The problem with the mainstream games industry is that they don't do what they think is cool. Guys like EA and Activision do extensive focus group testing. The problem with focus groups is that if you're not careful you get a game that appeals to the lowest common denominator. The Tony Hawk series is a good example of this. Sure, the Tony Hawk series has some good games, but there hasn't been a lot of innovation since the second title. The reason is that focus groups generally focus on what they like from previous titles and how it can be improved in the next. So you just get improvements on the existing formula, and that doesn't make for ground breaking entertainment.

I remember reading that The Sims rated extremely poorly in early focus group testing because they "didn't get it." I guess that means if you're working on a game with a proven formula then a focus group session to tweak it may be good, but if someone is trying to come up with something truly new and revolutionary then focus groups probobly aren't the way to go.

Personally I think it would be great if developers just did what they thought was cool, but since they're not the ones footing the bill (usually) they're limited. Publishers want something that's going to sell, they aren't interested in doing the "cool" thing unless it shown that they can make money on it. Can't blame them I guess.

Originally posted by alchemist
You can test out product concepts -- including in terms of games the theme, gameplay, interaction methods, UI layout, art style, player goals, etc. -- long before the game is done, and even before you move into production.

I agree with you, but how often is that done? Even a large publisher probably wouldn't be willing to commit the time and money needed to do this, which I think speaks to the lack of maturity of the software industry in general and computer games development in particular. From what I've seen, most games are still developed by "gut instinct," not careful pre-production product testing (my current game included). Game developers as a whole are getting more sophisticated, but I think they have a way to go before they reach the point of pre-production testing. And, as the recent "market research vs. art" thread showed, I'm sure there are plenty of people here who would find such pre-production testing to be anathema to them. I'm pretty sure nothing like this ever happened at Maxis; are there places where this kind of testing takes place?

Originally posted by Gmicek
The problem with the mainstream games industry is that they don't do what they think is cool. Guys like EA and Activision do extensive focus groups sessions. The problem with focus groups is if you're not careful you get a game turns that appeals to the lowest common denominator.

Isn't that sort of the mantra of the successful shareware developer too? All the successful indies I've talked to or read about have said the same thing: come up with a decent game with a viable market, release it, and refine the next release based on feedback from the previous one. The only difference between the mainstream industry and the indies is that the indies do their iterating out in plain view.

Originally posted by kerchen
I agree with you, but how often is that done? Even a large publisher probably wouldn't be willing to commit the time and money needed to do this, which I think speaks to the lack of maturity of the software industry in general and computer games development in particular. From what I've seen, most games are still developed by "gut instinct," not careful pre-production product testing (my current game included). Game developers as a whole are getting more sophisticated, but I think they have a way to go before they reach the point of pre-production testing. And, as the recent "market research vs. art" thread showed, I'm sure there are plenty of people here who would find such pre-production testing to be anathema to them. I'm pretty sure nothing like this ever happened at Maxis; are there places where this kind of testing takes place?

Focus testing is more extensive than you might think. They may not have a guy look at a screen with nothing other than different interface designs for a few hours, but they will ask the person about other products that have been specifically singled out because of various elements of their design. The games business is a multi billion dollar industry, and there are a large number of companies that do nothing but organize and examine focus groups for different industries, including game focused focus group companies.

I know a few people that work in production and marketing at EA and they feel conflicted over focus groups. Marketing loves it because it gives them a bigger budget and helps the chances of larger sales figures. The production people I talk to feel it's a double edge sword because it tends to stifle risky, but potentially innovative, concepts, but it also has the potential to add to their products sales figures. People in it for "the art" seem to really dislike them, those looking for the next paycheck don't mind.

One drawback to focus groups (from what I've been told) is that you can't just grab some gamers and tell them to talk about the game. You have to carefully screen the people in the group to best reflect the opinions and mind sets you're looking for. This is why focus groups have the potential to be disasterous. if you had a bunch of people who only played first person shooters and had them involved in focus testing on a game like Rollercoaster Tycoon we would have a very different product.

Originally posted by kerchen
Isn't that sort of the mantra of the successful shareware developer too? All the successful indies I've talked to or read about have said the same thing: come up with a decent game with a viable market, release it, and refine the next release based on feedback from the previous one. The only difference between the mainstream industry and the indies is that the indies do their iterating out in plain view.

I guess it then comes down to that whole "market research vs. art" debate. Someone looking to just make money on their game will most likely do ok with some well organized focus group tests, but it doesn't mean the game will be any better.

I'm not a developer, I'm a gamer, and I'm only interested in good games that innovate and challenge me. That might better illustrate where I'm coming from.

Originally posted by Gmicek

I guess it then comes down to that whole "market research vs. art" debate. Someone looking to just make money on their game will most likely do ok with some well organized focus group tests, but it doesn't mean the game will be any better.

You're contradicting yourself though. If a game sells better that means it IS better. If you don't like something doesn't mean it's bad. It's a matter of personal opinion. If a game is selling well, then there are people who like it, otherwise it wouldn't sell. I don't see games as art though. To me they're a form of entertainment. There are simply "safe" games that you know will sell based on market research, and "risky" games where you tried to be more innovative but you don't know whether your target market will respond well or not.

That's why you don't see a lot of innovation. In business, the idea is to minimize risk, and doing "safe" games is the best way to do it. Of course I'm not saying there's no room for "risky" games. If someone can afford to and wants to take risks they should. For the rest of us, the safe zone is our haven. :)

Originally posted by Fenix Down
If a game sells better that means it IS better. If you don't like something doesn't mean it's bad. It's a matter of personal opinion. If a game is selling well, then there are people who like it, otherwise it wouldn't sell. And so you could just as well look at it as a bunch of Jazz music fans vs. Pop music fans. One is popular, one is artsy. But obviously it doesn't necessarily one is better than the other. Same thing anywhere really. There are artsy movies and there are market-made movies, and there are pretty bad and pretty outstanding instances of both.

And as a rule of thumb, artsy-oriented developers will forever be at least slightly contemptious of the popular-oriented developers. Such is the power of money.

- Air

Originally posted by Fenix Down
You're contradicting yourself though. If a game sells better that means it IS better. If you don't like something doesn't mean it's bad. It's a matter of personal opinion. If a game is selling well, then there are people who like it, otherwise it wouldn't sell. I don't see games as art though. To me they're a form of entertainment. There are simply "safe" games that you know will sell based on market research, and "risky" games where you tried to be more innovative but you don't know whether your target market will respond well or not.

I define better as being the quality of the game improved. Is Myst a good game? I'm sure some would say yes, but many gamers blasted the game for it's thin story, horrible control, slow engine, and tired puzzles, but for years it was the highest selling game of all time. The Deer Hunter series of games sell extremely well, as do the various Barbi fashion games, are they good games? I think many would agree that they are not good games, but they sell better than games like Grim Fandango (which sold horribly. About 60,000 units after 12 months in release) , or Shogo (which sold approximately 30,000 last I read a couple years ago).

Originally posted by Fenix Down
That's why you don't see a lot of innovation. In business, the idea is to minimize risk, and doing "safe" games is the best way to do it.

You're right. If making money is all a developer is concerned about then by all means they should have plenty of focus groups so they can make sure every person out there in middle America loves the game, divert money and resources from the development of the game to marketing, and cram in features for the sake of being able to hype them later.

However, I don't think many developers want this. That's why we're seeing so many independent developers come from the ranks of the "mainstream" game development community. Mark over at Inhuman Games, the gang from Octopus Motor, everyone at Digital-Eel, most everyone at GarageGames, Codo Technologies, Monkeystone, Tesseraction Games, White Knuckle Games.. the list goes all. All these people have left the world of game development at mainstream companies so they can make the games they want to make, not the games the marketing department and focus groups tell them to make.

Quoting from one of my favorite Game Design articles ("Game Development: Myth vs Method", Game Developer, June 2002): If you want to find out what features to put in your game, or what type of game you should make, the last thing you should do is conduct a focus test.

... Humans are pack animals.... Focus tests inevitably devolve into popularity contests ...

Are focus tests completely useless? Absolutely not. However, we believe in one very simple principle: A focus test can only tell you what not to do. Sometimes, that's incredibly useful.The article then suggests:No game should be released without having been formally and extensively gameplay-tested during at least two points in the development process, and perhaps four or five.

Gameplay testing is simply putting your game in front of consumers - mostly the same consumers to whom you expect to sell your game - and watching them play. We may not trust what consumers say in focus testing, but we trust completely what they do in gameplay testing.You can read the whole article in the June 2002 issue of Game Developer.

Originally posted by Gmicek
I define better as being the quality of the game improved. Is Myst a good game? I'm sure some would say yes, but many gamers blasted the game for it's thin story, horrible control, slow engine, and tired puzzles, but for years it was the highest selling game of all time. The Deer Hunter series of games sell extremely well, as do the various Barbi fashion games, are they good games? I think many would agree that they are not good games, but they sell better than games like Grim Fandango (which sold horribly. About 60,000 units after 12 months in release) , or Shogo (which sold approximately 30,000 last I read a couple years ago).


Hardcore gamers might not like those games, but the casual gamers whom there are a LOT more of then hardcore gamers like those games. They're also being largely ignored by the game industry because those kinds of games aren't "cool" enough. Again I must stress that whether a game is good or not is very subjective, and depends on who it was intended for. Obviously Myst wasn't intended for hardcore gamers. And it also proved that there are a lot more people who would buy a non-hardcore targeted game. Of course, game developers brushed that off because they'd rather work on games that they're interested in. Since most game developers are hardcore gamers, that means making games that appeal to hardcore gamers. So you end up with a huge number of games competing for a very small hardcore gamer market. That's why so many of these games don't sell a lot of units, and why Myst sold so many.

If people are having fun with them, then they're good games. My older brother loved Deer Hunter.

Originally posted by Ratboy
If people are having fun with them, then they're good games. My older brother loved Deer Hunter.

I guess that's where our opinions differ. Just because a game sells well doesn't mean it's a good game, in my opinion. Quantity doesn't equal quality. Just because the Golden Child(the Eddie Murphy movie) did extremely well in the box office doesn't mean it was a good movie. For the producers and marketing people it was I guess.

Originally posted by Fenix Down
Hardcore gamers might not like those games, but the casual gamers whom there are a LOT more of then hardcore gamers like those games. They're also being largely ignored by the game industry because those kinds of games aren't "cool" enough. Again I must stress that whether a game is good or not is very subjective, and depends on who it was intended for. Obviously Myst wasn't intended for hardcore gamers. And it also proved that there are a lot more people who would buy a non-hardcore targeted game. Of course, game developers brushed that off because they'd rather work on games that they're interested in. Since most game developers are hardcore gamers, that means making games that appeal to hardcore gamers. So you end up with a huge number of games competing for a very small hardcore gamer market. That's why so many of these games don't sell a lot of units, and why Myst sold so many.

See, you're talking about sales numbers. I'm talking about the quality of the product. The Deer Hunter games are of a pretty low quality level, even if you just look at production standards. And yet they sell well. Are they good games? Anyone who says something like "Well yes, they are because enough people bought it so it must make it good." is basically slapping a large number of developers in the face. Daikatana didn't sell very well, but it's sold more than Combat Mission, Strange Adventures In Infinite Space, and Buboids combined. Does that make Daikatana a better game? For someone just looking at sales figures it obviously does. But if we're looking at the quality of the game then no it does not. You could say that it sold better because it was marketed more, to which I would reply that that is my entire point. Marketing doesn't make it a good game. Sales figures do not make it a good game. There are plenty of games for casual gamers that are also quality games (Rollycoaster Tycoon and Monopoly Tycoon for example). They were marketed to high heaven, sure, but they were also quality products.

As for your final comment on Myst. It could easily be argued that Myst sold so well because of the technology it helped usher in (CD ROMs) and the countless OEM deals the publisher got for it.

Developers who are hardcore gamers doesn't mean that they just make products for hardcore gamers. Look at the developers here, does the statement stand up? Look at a list that shows more than a couple dozen game developers and you'll notice that most of them produce games meant for fairly large audiences. The reason it's easy to mistakenly say that casual gamers are ignored is because the games you read about on places like blues news, voodoo extreme, gamespot and in magazines like PC Gamers and Computer Gaming world are geared for the people who read that material, hardcore gamers, but it does not mean that games for casual gamers are not being made.

And by the way, Myst was not brushed off by developers. In fact, it created it's own genre of puzzle/adventure games that has spawned counless titles for both hardcore and casual gamers alike.

Actually, I was saying it's a good game because it keeps my brother entertained. Fun = Good, production values be damned.

Originally posted by Ratboy
Actually, I was saying it's a good game because it keeps my brother entertained. Fun = Good, production values be damned.
Ohhh, right-O. Sometimes it's great to fire up some cheezy enjoyment. Carnivores really did it for me. Not very similar to Deer Hunter, but you get my drift, hehe.

Let me just say that I don't believe that marketing and hype are enough to sell lots of copies of a game (or anything for that matter, look at the Segway). Daikatana is a perfect example. Hyped to high hell, with John Romero as the designer, and because it was so horrible it didn't get very far. Therefore when someone says "that game sold because of all the marketing" to me that's BS. Marketing can provide better exposure to a good game and increase sales, but you can market a bad game as much as you want and it won't get you anywhere.

Also, just because Deer Hunter wasn't a "high quality" game doesn't mean it didn't do what so many "high quality 3D vertex shaders and full motion video" games often fail to accomplish. And that's to provide entertainment (read: FUN) to the person playing it. That's all that matters. Just because you don't find something fun doesn't mean someone else won't. Believe it or not, most casual gamers won't find StarCraft very fun. Marketing doesn't sell a game, it's merely a tool for reaching your customers. I'm taking a marketing class right now, and in fact one of the things they teach you is that marketing is not the same thing as selling. It's a pretty complex topic.

Originally posted by Fenix Down
Also, just because Deer Hunter wasn't a "high quality" game doesn't mean it didn't do what so many "high quality 3D vertex shaders and full motion video" games often fail to accomplish. And that's to provide entertainment (read: FUN) to the person playing it. That's all that matters. Just because you don't find something fun doesn't mean someone else won't. Believe it or not, most casual gamers won't find StarCraft very fun. Marketing doesn't sell a game, it's merely a tool for reaching your customers. I'm taking a marketing class right now, and in fact one of the things they teach you is that marketing is not the same thing as selling. It's a pretty complex topic.

I can make a flash animation of unrine in a jar and someone will find it entertaining, but it doesn't make it good.

Anyway, I think the discussion has broken down, from what I can tell we're starting to talk about two different things. I'm saying that just because a game is successful doesn't mean it's a quality game. If someone out there cares to argue against that then they can but it's ultimately futile if you ask me. Popular does not equal quality.

Originally posted by Gmicek
I can make a flash animation of unrine in a jar and someone will find it entertaining, but it doesn't make it good.


Would people buy it though?

Originally posted by Gmicek

Anyway, I think the discussion has broken down, from what I can tell we're starting to talk about two different things. I'm saying that just because a game is successful doesn't mean it's a quality game. If someone out there cares to argue against that then they can but it's ultimately futile if you ask me. Popular does not equal quality.

And for the last time I will say that quality is a subjective thing. Just because you don't like POP music for instance, doesn't mean it's not quality music. I personally know people who love that stuff. Everything in entertainment is subjective. If something sells, then there's someone out there who thinks it's good. And that's all that matters in the end.