For those of you who've been following the great sales experiment thread (http://www.dexterity.com/forums/showthread.php?threadid=1459), I thought I'd share some preliminary results. I don't feel I have enough data yet for these results to be too significant, but perhaps some will find them at least interesting.

Odd-numbered visitors see both "download free demo" and "order now" buttons on each product page. Even-numbered visitors see only the "order now" buttons.

For the roughly 10 days this experiment has been running, total quantity of sales was split as follows:

46.2% odd (demo available)
53.8% even (no demo)

For the total dollar volume of sales, the split is:

43.3% odd (demo available)
56.7% even (no demo)

Based on total dollar volume, the even (no demo) group yielded 130.8% of the sales that the odd (demo available) group produced. So if these stats hold out, it suggests that getting rid of the demos could potentially increase total sales revenue by about 31%.

Again, I want to keep running the experiment for at least a few more weeks, since as was previously discussed, the short-term effects may differ greatly from the long-term effects. And I definitely don't feel I've collected enough data just yet. So it's very possible that in the weeks ahead, these figures will change a great deal.

But I know how hungry indies are for numbers, so hopefully these will be appreciated.

Another interesting metric to track would be the Yield per GB of downloads. Even if your total sales dropped slightly but your bandwidth was halved, it would still make sense to go with the new method as bandwidth is one of the primary costs for online distribution. Because of the yeild spread between your bandwidth costs and your gross revenue your net revenues end up being much larger over time.

That's both surprising and impressive - but Dan's comment on yield -vs- bandwidth makes a lot of sense. Definitely interested in seeing what happens after, say, two months of this in place.

One thing to keep in mind is that these results might only be meaningful with regard to an established company. It could be that demos play an important role in a company that has yet to build a reputation for itself, but that as word-of-mouth spreads and player reviews appear on the Internet demos become less important than they once were.

Heck, it's certainly possible that these results are only relevant with respect to Dexterity in particular. Since I don't know of anyone else doing these kinds of tests, I have no idea how great the deviation is. Others' results could fall within a very narrow range, or they could be all over the spectrum.

I did an analysis a few months ago and concluded that a metric like profit per GB of bandwidth just isn't a very good metric for us to use in decision making. The reason is that bandwidth costs are still a fairly small percentage of our total revenue... less than 5%. The bulk of our costs are associated with product development and product acquisition. So a more useful metric for us is profit per product, and in the long run, profit per franchise is even more useful. Plus these metrics are a lot more meaningful across a broad spectrum of distribution options, especially those where bandwidth isn't a consideration, like CD-ROM distribution or licensing.

Chronos: Trust me, I wasn't contemplating trying to do the same thing :-) If I had a serious installed user base, and lots and lots of traffic everymonth, I'd probably consider it. I don't, so at the moment I'm exploring other innovative avenues of marketing the games, plus the retail distribution stuff.

Im not really surprised by these results. It seems to me that if you had a number of people with demos coming to the site, maybe considering a purchase, they'd have an easier time, or would be less likely to try something else if the pages had only buy buttons.

That being said, if a significant number of trials/demo downloads occurs off the site directly I have a feeling the numbers will flip long term.

If most demo downloads occur on other sites like download.com then the numbers could stay. I guess I'll eventually have to do this experiment as well... In fact it makes some sense to me to consider redirecting all ads to a site that has demos whereas all links coming from demos already come to a site which is totally sales focused.

For fun, you might want to check out how the demo downloads work on my site. There's definitely a measurable percentage of people who purchase off the "why not just buy it now?" screen. (maybe 10-15% last check) Returns are still next to non-existant.

Original post by svero

In fact it makes some sense to me to consider redirecting all ads to a site that has demos whereas all links coming from demos already come to a site which is totally sales focused.


It seems easier to redirect the demos to a (new) site without buy links while keeping all the ads and the original site with the demos the same.

It would be pretty easy to tag visitors coming from links in the demos, such that they see only ordering links. This could be done just by storing their cookie in a database table and then checking for the cookie's presence in that table on each visit to the site. Our demos already tag their links, so it wouldn't be much extra work to do this. No need for two separate sites -- PHP & MySQL could dynamically customize the site for each visitor.

So many experiments... so little time.

Also one thing which you should consider: All your sales are made by "old" (been available for months/years) products. What happens if you put a new product (which nobody has downloaded before) there? What would happen to sales now? :)

Interesting experiment, must say - looking forward to see more results.

(and I agree with Midnight Ryder - this method of selling would need a good customer base)

Dunno, but I'm sure we'll find out eventually....

Here's something to think about...

This is Steve's first stab at selling without downloadable demos. I would guess that there will be many refinements to improve its conversion. So, with tweaks, it could swing even more to the non-download side.

I love to see market information like this :-).

Ther are two factors that I would suspect are accounting for it.

First, as someone mentioned, that the visitors buying already had the demo, either from your site or from somewhere else. In which case they are already familiar with the product (which is the point of a demo), wanted to buy it, and didnt get side tracked by 'new' demos.

Second, when someone visits your site and then comes back, does a cookie remember what they saw? What could be occuring is some people are return visitors, they are first numbered odd, then return to purchase it, and get numbered 'even'

Very nice work though :-). We are performing a multitude of tests in the same genre of this. I may be able to post some of that info at a later date.

Terin

A cookie does remember the odd/even distinction, so it's the same on every return to the site, unless something has wiped out the cookie or it's been blocked. Visitors who block cookies are tracked separately, so their data isn't part of the result set. The number of sales from visitors who block cookies is less than 1%, so it's not really significant anyway.

So that leaves the real question as:

How many of them purchasing had a demo beforehand
How many of them have previously purchased

Very cool. If you would like my assistance with it in some way let me know

Terin

Can you tell what percentage of people in each camp are first time visitors vs repeat visitors?

Yes, I can tell. Most of the customers at this time are first-time visitors. But this changes whenever we launch a new game or send out a newsletter, since those bring in a surge of repeat customers.

These statistics could also signify something no one has mentioned yet.

Perhaps there are two distinct classes of users: those who would buy without trying a demo and those who wouldn't. In that case, removing your demo could effectively reduce sales by up to 43%.

Originally posted by ivo
These statistics could also signify something no one has mentioned yet.

Perhaps there are two distinct classes of users: those who would buy without trying a demo and those who wouldn't. In that case, removing your demo could effectively reduce sales by up to 43%.

Maybe, but I'm under the impression that here we're just talking about removing the demo from "your site proper" -- that is... people could still get the demo from download sites etc.... Your assertion would still hold true for people who only visit the main site from a search engine or whatever.

Originally posted by svero
Maybe, but I'm under the impression that here we're just talking about removing the demo from "your site proper" -- that is... people could still get the demo from download sites etc.... Your assertion would still hold true for people who only visit the main site from a search engine or whatever.

Yeah, I think you'd really have to compare longer term conversion rates to see how many customers you end up missing by not providing a demo on the site. How do the conversion rates on the odd and even pages compare to your overall typical conversion rate?

There may also be people who have now downloaded the demo, from the site and have not yet purchased but will purchase in the future. I guess it depends on the game (and perhaps should be in another thread) but how long does the average "trier" take to become a "buyer"?

Originally posted by ivo
Yeah, I think you'd really have to compare longer term conversion rates to see how many customers you end up missing by not providing a demo on the site. How do the conversion rates on the odd and even pages compare to your overall typical conversion rate?

There may also be people who have now downloaded the demo, from the site and have not yet purchased but will purchase in the future. I guess it depends on the game (and perhaps should be in another thread) but how long does the average "trier" take to become a "buyer"?

Steve was saying that the average trier he sees converts or not within a few days. It's been similar to what I've seen for us, but that doesn't mean there aren't people who come back 2 weeks or even a year later. I've seen that too. I think most people who will convert do so pretty close to when they first try the game. I've seen messages from people who run 30 day trials (usually for utilities though) saying that there's a spike at the end of the trial period, so I suppose it partly depends how your game is locked.

Oh just one other note about the groups of people who come to a site. There's probably a 3rd group of people...people who in the absence of a free trial will risk purchasing right away. I think the money back guarantee is probably a key element of that.

Originally posted by ivo
Perhaps there are two distinct classes of users: those who would buy without trying a demo and those who wouldn't. In that case, removing your demo could effectively reduce sales by up to 43%. This would mean that A and B groups are differentiated according to their preference for demos, which is extremely unlikely since visitors are randomly assigned to either group. It's far more likely that demo and non-demo preferences are more-or-less equally represented in each of the two groups. That is, there are probably as many people who prefer demos in group A as in group B, so the results for group B contradict the notion that Dexterity would lose "up to 43%" of sales for lack of demos.

Originally posted by chronos
This would mean that A and B groups are differentiated according to their preference for demos, which is extremely unlikely since visitors are randomly assigned to either group. It's far more likely that demo and non-demo preferences are more-or-less equally represented in each of the two groups. That is, there are probably as many people who prefer demos in group A as in group B, so the results for group B contradict the notion that Dexterity would lose "up to 43%" of sales for lack of demos.

I agree with you that most likely demo and non-demo preferences are more-or-less equally represented for both the even and odd pages (the 43% was an absolute worst case, I guess I should have said "no more than"). And in fact, there is probably another group of people who would only buy the game if there is no demo (ie. the demo is good enough for them.) This group of people could be significant.

However, the 57/43% statistics are just the percentage of the total sales so they don't show the breakdown of the relative groups. You could be losing some sales by having the demo but gaining others. With a purchase conversion rate of say 2%, if just 1% of the population fell into one group, that could be half your sales. In an ideal world, you'd want to capture all potential buyers.

As other people pointed out, we don't know how many of the non-demo people have actually already tried the demo previously and we don't know how many of the people who downloaded the demo will buy in the future. In fact, we don't even know how many people downloaded the demo relative to the total sales so we can't really even make guesses.

If:
X = # of people who won't buy if they can download the demo
Y = # of people who will only buy if they can download the demo

I guess the real question is: Is X > or < Y ?

and also, does making the demo more available generate any indirect sales (ie a purchase by someone other than the downloader)?

You know you're making waves when you've made slashdot. :p
http://games.slashdot.org/games/03/10/26/0730216.shtml?tid=127&tid=186&tid=98&tid=99

It isn't on the main page unfortunately, only on the 'games' section, and as such doesn't have many views or replies compared to what the main page gets. But it is there!

Why so coy about the name of the developer I wonder?

From the comments on that page, it looks like the people there really don't understand the nature of this experiment at all. Many of the comments are based on incorrect assumptions, like return visitors seeing different versions of a page on each visit. This reminds me of the "telephone game."

Many of the slashdot people are missing the point because nowhere in the story is the details of the experiment (I had to look around to find this and the other thread).

But, yeah, there are some factors corrupting the results
- The demo conversion rate is on a slightly different schedule;
since someone buying from playing the demo isn't buying straight off, demo sales will be underrepresented if you include the first few days of sales (as the sales from people looking at the demo won't have started yet).
You could probably remove this factor by tossing out the first few days (ideally, first month or whatever the full length of the demo trial is) of numbers.

- Previous purchasers (even just having heard about the site before) and previous downloaders (as you mentioned, demo downloaders from other sites) could be skewing the results by making 'just buy' people more likely to buy.

You may have mentioned this in the other thread, I'm not through it yet though.

You are correct, Steve. If you arent familiar with slashdot user comments, let my just bring you quickly up to speed by telling you one of the most frequently posted comments there (other than "first post" or
"1) a beowolf cluster of hot grits down Natalie Portman's pants
2)???
3)Profit"
...don't ask :) ) is "Read The F'n Article". 90% of the people on that site go off half cocked trying to give the "expert" scientific analysis based on a short 1 paragraph summary of the article referenced. Basically, most anything posted there is useless. Generally, the only time you see many useful comments is if the article makes the front page and you get a couple hundred comments. Statistically, you are bound to have at least a few intelligent posts that way (either because they have something intelligent to say, or because they actually did read the article and want to set the record strait for all the morons that couldn't be bothered to read it).

I think the point most people are missing is that Steve sharing his results doesn't really help us others decide anything. Without doing the experiment on our own site we don't know how it will affect our specific case. When we have a few tens (or hundreds) of result sets we might generalize that "in most cases ..", but even then each and everyone should measure for themselves.

The great contribution from Steve has been the idea of another interesting experiment to run on your site.

What about the refund rate?
I'd expect the refund rate to be significantly higher for people who purchase without having demo'd the program, thus partially negating the extra sales.

I'd expect the refund rate to be a little higher w/o the demos, but so far it's still negligible.

Hey! I'm a proud registered owner of Aargon Deluxe and Rocknor's Bad Day, and also an indie game developer. The experiment is really interesting and all, but jeebus, I just want to download some demos, and unfortunately, I got put in the no-demos-for-you group.

I don't really want to clear my cookies, (I will have to reenter about 20 passwords on various sites.) but I can if I have to. Is there a download page I can go to directly or something like that?

Also, I think that if you don't allow trial game downloads, you shouldn't have "Downloads" plastered all over the screen. I realize that it is still possible to purchase downloads, but "Downloads" implies "free" to me, and it feels like bait-and-switch. The whole site gives a positive message except for this one thing.

-Erik

Thanks for the feeback. You can download most of our demos from other download sites like www.download.com. Just search on the name of the game.

The way this experiment has been going, most likely we'll stop making the free demos available to all visitors within the next few weeks.

I take you're very happy with the results then :)

One technical question that I haven't seen addressed properly: you mention the issue of people who downloaded a demo before the test started as a 'noise' factor which fades over time. But what about the people who download a demo from e.g. download.com during the test period? How do you know that they already have the demo when they first visit the site? Do you set a cookie when they download the game? If so, how?

It's great if people get a demo from download.com first and then visit this site. That's the reason we have those demos out there. But once they get to this site and see all the other games, I want them to make a buying decision, not just download more demos. As I've mentioned previously, I'll continue to use the shareware model as a marketing method. But once someone reaches this site, a marketing method isn't what's needed -- that's when we need to sell. If the person reaches dexterity.com (via a free demo, a search engine, or otherwise), then our marketing has already worked.

I just updated the results of the experiment today, which has been running for about 2.5 weeks now. It has shifted a little the other direction, as some predicted.

For the total number of sales, the split is now:
47.8% odd (demo available)
52.2% even (no demo)

For the total dollar volume of sales, the split is now:
45.0% odd (demo available)
55.0% even (no demo)

Based on total dollar volume, the even (no demo) group yielded 122.4% of the sales that the odd (demo available) group produced.

What I also found interesting is that the dollar amount of the average sale for the even (no demo) group is 12% higher than for the odd (demo available) group. So when even people buy, they're spending a bit more on average. This amounts to about one extra game purchase for every 6-7 sales.

The results do seem to be converging. Most of the fluctuation happened during the first 10 days.

I'll keep the experiment going at least until it hits the 4-week mark. But I don't expect the results to change much. Overall the experiment has taken a total of less than 2 hours of my time to conduct and evaluate. But the payoff is potentially a permanent 20%+ increase in sales. But what if I never ran this experiment and just assumed trusted "conventional wisdom?" I'd be leaving enormous amounts of potential income forever untapped. This is why Jay Abraham lists "Not Testing" as #1 on his list of the top 10 marketing mistakes. I could buy a new house with the payoff from this one simple test.

I'd be very interested in seeing what are the figures for people coming from a "Buy Now" button in a game demo (supposedly those that have already made the buying decision) versus those coming from search engines (supposedly those coming to get a free demo)

Just to clarify... the purpose of sharing this information isn't to give everyone here a fully detailed insight into all the facets of Dexterity's business but rather to suggest that it would be wise to setup your own experiments to find out what works best for you. Right now I'm just sharing the most general results in the hopes that some here will find them helpful and see some value in conducting their own tests. I've been doing A-B splits for several years now, and this is the only one who's results I've made public in any manner, since it seems to be applicable to a large number of developers.

It all seems encouraging however, you also have a money back guarantee, I don't think you'd fare as well if you didn't.

Keith

Are your returns up?

The return rate is normally so low that it will take a while to tell. Monthly returns are normally in the single digits and even sometimes zero. But so far returns are no higher than normal.

Forgive me if I'm stupid like a monkey, but I'm missing the point here. :confused:

What was the original idea for this experiment...

...was it to have dexterity.com act as a marketplace only

...and let some users to download from site and some users to download demos from somewhere else...?

so... that's it? the idea is that users download from somewhere else and you get more sales? Please enlighten me. I don't see how this has improved your sales... I just don't get it :D

No the idea was to see if offering a free, playable demo actually makes a difference to sales. The test is to send half the visitors to an area that offers the demo and half to an area that does not offer a demo, just a "BUY NOW".

The reasoning being that if you can get the same (or more) sales without a demo you save time and bandwidth costs.

Of course there is one issue that is needs to be measured and that is customer satisfaction/return visit rate. Are customers who buy without trying first more or less likely to come back and buy again. If they buy the game and don't like it will they return or will they go elsewhere in future. Will a customer who buys after playing it (because they know they like it) be more likely to become a long term customer.

Are you risking short term increases in sales/cost savings at the expense of long term customer base?

Originally posted by obscure
The reasoning being that if you can get the same (or more) sales without a demo you save time and bandwidth costs.

From what I understood, the experiment isn't about 'no longer providing demos' but about 'no longer providing demos from the Dexterity site'.

People can still download demos of the Dexterity games, but only from other places (such as download.com). As soon as they visit the Dexterity site, all they can do is buy. The reason for this is to prevent potential customers from getting distracted by all the other free demo downloads.

I think Mike Boeh mentioned something like this in another thread a while ago: when his installers put links to other Retro64 games in the start menu, downloads increased -- people downloading the other demos as well -- but sales went down. After all, why buy a game when you can play all these other demos for free instead?

Did you change or cancel the experiment?
Because suddenly I have download-links on the Dexterity page again.

If you did cancel it, do you want to tell us why?

Just running some different experiments now.... I've been doing these experiments for a few years now, but most people usually never notice them.

Steve,


Did you finally end your experiment? What happened in the end?
Would you mind sharing some of your findings? It appears that you still have the "download free demo" button together with the "order now" button (on Dweeb Gold page). So things went back to the old way?

The last post to this thread was 25.Nov.2003. Did the thread jump to somewhere else again? :)

>This is why Jay Abraham lists "Not Testing" as #1 on his list of the top 10 marketing mistakes.

Heh, just listened to that on one of his tapes last night and I heartily agree. I ran one test that increased my email subscribers by 244%! I ran another test on sales copy and increased conversion rate for one game by almost 300%. I am sure if I kept testing more and more methods I could get higher results...I have never heard of anyone that tested feel like it has been a waste of time, most say it brings in lots of surprising results and alot more revenue.