Steve Verrault's question posted to the thread Advertising Prices (http://www.dexterity.com/forums/showthread.php?s=&threadid=511) reminds me of a question I have regarding the testing of marketing plans. How valuable is it to do strict testing of any aspect of your plan? If you wanted to test what effect some change has on your downloads or sales, is it worth the effort to gather actual statistics on its impact, or is it sufficient to observe the results more or less anecdotally? For example, let's say you wanted to see if a yellow Buy Now button was more effective than a blue Try Now button. You could try it for a day and see what your sales are, but to really know the impact you'd have to leave it in much longer. It would be nice to test out a certain change on a subset of your customers. This would not only give you a control group for comparison, but it would also allow you to try out risky changes with little impact on your overall sales. Does anyone here do that, and if so, what methods do you use? I can think of a way of doing it, but it would have side effects. Let's say you set a "campaign" cookie when a visitor arrives on your site without one set. Most customers will have this set to 0, but to try a campaign on 10% of your customers, every tenth cookie you set, you give it the number of your campaign. Now whenever you display a page, you dynamically create a different page based on the campaign stored in the cookie. The campaign value must be noted whenever a download is requested (to track the success of the visitor-to-download rates) and when processing an order (to track the success of download-to-sale conversions). These naturally impose their own campaign-tracking difficulties depending on how your ordering process works. Focusing just on the cookie problems, you can see someone might visit your site from a different browser and get a different campaign. This in itself isn't a problem until they return to their other browser and see the original campaign again. I don't want customers feeling manipulated, especially if I'm testing price changes. Perhaps the campaign chosen also considers other identifying information as a factor, like your IP address. This would fix multiple browsers on the same computer or network, but wouldn't help with the home/work issue. Also it won't help a great deal if you have low sales to begin with; at least you'll have to run the campaign a good long while to get any relative statistics. Any thoughts?

I think testing what you do in a reasonable and scientific way is probably one of the most crucial things you can do when trying to improve your games and marketing. Methods vary of course, but figuring results out anecdotally is really quite difficult and you never really know. Imagine being able to say to yourself. I know for a fact that making the buy button yellow will give me 10% better results. That's a powerful thing.

Suppose you change a button color, as you say, and at the same time, unbeknowst to you the game makes it's way onto some magazine CD cover. Your sales go up 10%. Is it the button? You have no way of knowing without tracking. Basically it's a wasted effort.

How you track these things varies depending on the test you want to do. Sometimes it's sufficient to know your conversion rate and try two versions of a game, but even that is difficult to track because again there's always the possibility of an old version of the game making it's way to a cd or being hosted somewhere.

In terms of tracking you can use cgi paramters, cookies, etc.. You will need control groups though. So suppose you wanted to test the effect of no price on the buy screen vs the price clearly advertised. You could have the game choose one buy screen randomly and then when people click to go to the web page it's coded in the url

http://mycompany.com?buyscreentype=1

From that you could set a cookie with some info and then later track to see who bought. You'd also have stats of how many people bothered to click to the page. etc...

Anyway... the harder question is not so much whether or not you should test your hypotheses, but rather finding stuff you can easily test. Personally I'm pretty happy when i come across a meaningful test I can do that won't take months of my time to work out.

Actually there's a book about this. It's sort of a famous advertising book. It's out of date now and not that relevant to web marketing, but I'll mention it anyway. It's called "Testing Adverstising Methods" by John Caples

Another small note. If you want to test advertising phrasing google adwords is a great little lab to work with. It's worth using that service just to play with different phrasing and see the effect it has on click through rates.

Putting together an effective tracking system sounds like a bear. With the system you are using, can you also track the percentage of downloads that came from such and such a site (and keywords), and what percentage of those made a purchase? That is, can you follow a visitor through the entire process from first visit to sale? If you say "10% of my visitors come from this adword on Google, and you sell $X a month, you could say that one-tenth of that came from those Google visitors, but it would only be an estimation. How much more worthwhile is it to know that of those Google visitors, 25% downloaded one or more games and of those 5% converted into a sale?

How can you tell which part of the process is a higher priority to optimize? In some cases it must be obvious (say, if you make a game but don't tell anyone about it), but after you start getting results in all areas, which part has the greatest potential for turning sales with the least effort? I know people speak of the desired 2% conversion rate, but it seems to me that's only one piece of the pipeline. Are there rules of thumb for the other parts? This is the pipeline how I'm seeing it now, which doesn't really define cross-selling or repeat customers:

# unique visitors last month
    x
% from a particular campaign/source (Say, a Google Adword using your blue buy now button that only 10% of your customers see.)
    x
% that download
    x
% that buy (conversion rate)
    x
$ average product price for last month
    =
$ revenue last month for a particular campaign

>That is, can you follow a visitor through the entire process from
>first visit to sale?

Not generally, but I would really like to. The best I can hope is to try and track with some cookies, but a lot of sites will only let you link directly to a file, so you can't always track individual customers. What you try to do is track in those cases that you can and then make the simplyfying assumption that it's similar to your other non-trackable sales. I think it's inevitable that there will always be a certain amount of guesswork.

>How much more worthwhile is it to know that of those Google
>visitors, 25% downloaded one or more games and of those 5%
>converted into a sale?

If you can track it that closely then you can always know if google in particular is profitable for you. Otherwise you may be spending money on google and not getting any ROI. So it's the difference, potentially, of knowing you're making money or losing money. I'd say that's valuable.

>How can you tell which part of the process is a higher priority to
>optimize?

Steve P. has talked about this and I think he's got the right idea. It's effort vs reward. You look for the biggest reward and lowest effort. If changing you buttons to yellow makes a 5% conversion difference and it takes 10 minutes to do then you do it immediately. If redoing the user interface for your game takes 1 month of work and makes a 5% difference then you have to weigh that against the other things that make a 5% difference. And that's when it becomes beneficial to know what makes you 5% or 10% - That is.. if you're completely guessing then it's actually quite hard to know where to spend your time and effort. Can you know? Probably you can if you do some experiments. Anecdotally it's very hard. Basically the closer you are to the mark the better decisions you can make.

>I know people speak of the desired 2% conversion rate, but it
>seems to me that's only one piece of the pipeline.

Sure absolutely. 2% of 100 people is only 2 sales.

>Are there rules of thumb for the other parts?

Well you can sort of figure out the other parts yourself. That is... if we assume 2% is a decent conversion rate then you know you have to get 10k downloads a month to make about 4000$ (at 20$ a game) in sales. If you've done everything you can to get the game to as many people as you can and you're still only getting 1000 downloads a month then you'll only make 400$ and at that point you have to figure out if there's something you can still do to improve exposure or if your time is better spent trying to turn 2% into a 3% conversion.

I've seen enough variety in terms of effectiveness of advertising that the only thing I fully trust is solid empirical data. This is why setting up a detailed tracking system is a priority this year. I want to know things like what % of people who view a particular game page on our site download the game, and of those people, what % actually buy, and when, and how much do they spend over the course of a year, and how many times do they contact tech support.

Averages are a good start, but because of the number of products we have and the variation in sales, there's plenty of room for more optimization when you get behind the averages to the specific numbers.

For instance, right now our server periodically regenerates the top 5 list on our home page with nothing but hard empirical data. No human value judgments are involved. Games are ranked based on the ratio of total sales revenue / total downloads. This is not the perfect way to rank games, but what it does is encourage more downloads of the games that yield the most revenue per download.

We can even set cookies when sites link directly to the file (i.e. http://www.dexterity.com/download/dweepgolddemosetup.exe). The way to do this is just to set up an apache redirect to a PHP script that drops the cookie and then serves up the file download. This won't set a cookie when people use download managers, but it will work when people use a browser. Because each user will get a unique cookie, we can track everything each visitor does. When someone buys a game, we'll know what other pages on the site they visited and when, what demos they downloaded and when, what other games they bought and when, and (sometimes) where they came from. By recording all this data, we can do queries to find out what's really going on behind the scenes.

Absolute numbers are not as important as relative numbers. No measurement system is perfect, but if you're consistent in how you measure, you can test what causes a relative increase or decrease. And you can compare the relative performance of different products.