Has anyone out there used professional beta testing services (like www.betabreakers.com) to test their games? Do you know of any good affordable services? I'm thinking it probably won't be too cheap, but testing a product systematically on 50 different PC configurations could be very valuable, and help aliviate the ever present "I wonder if it's working for everyone" question. I personally would be willing to pay $100-$200 for a good service, but maybe I'm dreaming.

Any help?

I think it's possible to get 50 people testing without shelling out that kind of money. What sort of product are you releasing?

Not to burst your bubble, but you won't get good testing for $100-$200. That would be a day, day and a half at most worth of work from one dedicated tester. And with the services they're advertising, I bet you'd get a weeks worth for somewhere in the 4 - 5 figure range.

I've thought about doing that kind of thing for indies as side work (since I already do it for a day job - almost done!), but realized that likely no indie would pay what I would have to charge to make it worth my time (imagine $30-$40 an hour * 40, and that's a deal rate).

But, for what it's worth, I've never used those services, so I could be talking out of a different orifice than my mouth. Maybe their are economies of scale or something, but I doubt it. If you can afford it, and if they're good enough, it's probably helpful, especially if you don't have the hardware and your pretty sure you'll make a bundle. Otherwise, go with the free version.

This is something that just popped into my head, and with no research, I wonder if it would be possible to cross-promote somehow to share testers with each other. Some people have a lot of free time on their hands so maintaining a pool of people eager to try out cool stuff (and of course get the product free at the end) might work. I'm thinking something simple like each company has their own testers, but in whatever testing area they have, they mention other beta sign-ups by other companies.

The big potential problem I see is one or two products hogging the tester attention while testers ignore the other stuff. Basically instead of company A and company B both benefiting from Bob, who originally signed up to beta test for company A, Bob would decide company B's product, whose beta he heard about from company A, is much better and would forget his duties testing for company A. If such a problem happened then the idea could very quickly flock hundreds of people to one product, while leaving other products with less testers than they started out with. If this doesn't though, we could have a nice circle of players that regularly sign-up for betas in order to get free stuff. This would probably be particurly popular with the 13-17 year old crowd since these people often don't have easy means to purchase online even if they have the cash. They also have a fair amount of time that they would spend gaming anyway.

Any thoughts?

Actually...

Testing is more than just "playing" a game or "using" a software.
Being the "tester" for the CMS we produced in our firm (yes, I was involved in the programming) I learned a bit about it the hard way.

Testing has to have some system, so you can report back to the programmer what happened exactly and not just "doesn't work".

But, some gamers might be disciplined enough and enjoy the process of working with the authors.

Maybe a forum where everyone sends their beta-testers?

::emp::

I think it's possible to get 50 people testing without shelling out that kind of money. What sort of product are you releasing? Yes, it's quite possible to get 50 people ot sign up for testing... but out of those 50, 10 are dedicated, and only 5 are very active... at least that's been my experience: the problem appears to be getting dedicated testers, and not just some kids off the street who want to get a free game, and that's it. As for the product, it's an arcade-spinoff space game.

Not to burst your bubble, but you won't get good testing for $100-$200. That would be a day, day and a half at most worth of work from one dedicated tester. And with the services they're advertising, I bet you'd get a weeks worth for somewhere in the 4 - 5 figure range.That's what I was figuring, really. I just don't have that kind of money to shell out right now. Sigh...

This is something that just popped into my head, and with no research, I wonder if it would be possible to cross-promote somehow to share testers with each other.I think your idea is pretty great, except for the issue that your brought up. I think it'd be cool to set up a website dedicated to a community of beta testers, and shareware authors who wanted their games tested. Pretty much it'd turn into a "these are the current projects, choose which one(s) you want to test. Here are the requirements for each project (e.g. certain minimum amount of feedback), and here's what each project offers." Ooh--and Beta testers could have rankings: the more they post, give feedback, etc. the higher the ranking... and shareware authors could upgrade certain testers if they had a good experience with them--e.g. give them a "dedicated tester" stamp. Hm... I don't have the time to set up something like that right now... but maybe sometime?

Testing is more than just "playing" a game or "using" a software. I agree and disagree. I hear what you're saying, but I think a large part of compatibility and general beta testing is just seeing how your game performs on tons of different systems.

I could set the beta thing up, and maybe for a very small fee permit usuage of the beta system we have. This is still an idea rolling around in my head though, I'm going to crunch some numbers later and see if it is feasible. In the meantime, how much money would you pay for access to this beta resource, complete with a beta testing area that organizes bug reports, feedback, provides forums, etc. It would be a per-game fee I think, and how many testers you could may or may not have something to do with the price. I would like to keep such a system simple, so maybe for people with more money to spend, they could purchase additional promotion to help get more testers to sign-up. The normal hypothetical fee for this hypothetical service would include:

1) The opportunity to have your beta marketted to a large group of people interested in participating in beta tests.

2) Access to the beta web software, which would allow the beta to be handled through a standardized system. Possibly with an SDK for use in games to allow in-game bug submit.

The big problem I see with this system is bandwidth distribution. A game that is getting 100 testers who are regularly posting bugs will suck up a lot more resources than a game with only 10 devoted testers. Maybe a small fee per tester would be necessary to make things fair and allow smaller products to take advantage of the service. Customers would of course be allowed to place a cap on how many people sign-up and would have the final say on who they allow participating in their beta.

As for motivation, I agree that 50 testers won't be all motivated, but I strongly disagree with your numbers. For the alpha release of CustomBar, we had 7 people. 1 didn't even download the thing, 2 decided they would rather wait for the full version and that the alpha had problems they couldn't live with. The other 4 provided excellent feedback and bug reports. Working with beta testers is at heart a diplomacy issue. You have to treat them with respect and you have to give them reason to provide you this free service. Testers that sign-up for service will probably be more motivated than the average group of people, due to rankings or however such a system would work.

An interesting idea there Lizardsoft. It could end up being a very useful resource. A relatively small fee wouldn't be too prohibitive.

I would add that you should give beta testers the ability to specify what type of games they want to test. When someone has a game that needs testing they can notify those people who fit the profile and restrictions that they can sign up for that specific beta.

Another issue is that if you have multiple beta releases then it's often good to get new people to look at it.

The SDK that you mentioned shouldn't even be considered until a critical mass is attained. I don't really like the idea even then.

Of course I think to get everything going you really need to just set up a free system where every game gets a messageboard and start from there. Once it gets going then think about implementing the rest.

If you've got the time and resources I personally think you should start it off and see where it goes... that's what I'd do if I had either of the above :rolleyes: .

As for paying, I think Nexus has the right idea: get a system rolling, and then depending on who wants to use it, and mainly HOW GOOD THE SYSTEM IS, start charging. The "HOW GOOD" part really is the key as far as I'm concerned... if it's just a message board and some people who signed up to test, I don't know if I'd pay anything--as I've basically got that already. But if it's a good system with a lot of good dedicated testers, and some kind of ranking hiarchy (as I mentioned before)... , and some good PHP backend to allow testers and authors to have easy-use and powerful control panels, and databases to keep track of all the testers' system info, etc. (think garagegames.com, but modified and expanded for our purposes), I'd probably be willing to pay a couple hundred bucks per game for such a service... maybe more.

actually, I'm thinking of doing that myself.

I have a list of over 200 common bugs and have 5 beta testes.
the only problem actually, isnt the costs, but finding more beta-testers.

Okay I've come up with a plan. I have most of the web software already, I just need to make some modifications to support the new concept. I'm going to see how feasible this is and if it's worth the effort it'll be up very soon. The SDK at first would be very basic, just provide people tools to allow testers to submit bugs from within the game, but forgoing the SDK for now might not be a bad idea either. I have a few people interested in this and it looks very possible. I've been working on the private beta system for a while and turning it into something everyone can use shouldn't be too hard. I'm fleshing out the details and once I handle a few business things I'll certainly post more information.

Edit: just to elaborate, I have a whole system where people can have logins, post bugs, keep track of things, send messages, keep todo lists, etc. This is the result of a flexible Backend that has proven to be an effective and profitable project. Getting a site up should only take a couple weeks, I plan on having a lot of testers for my own project so that should kick start the tester pool very nicely. The system would be hosted on our servers as well, so customers do not have to worry about hosting and setting this up. I would appreciate more feedback on what a fair price for this would be.

I'll be waiting...

As for getting testers, I don't think that'll be TOO huge an issue... once we get a few cool game betas up there, and post around some, it'll be the place that game testers want to go... plus, I can refer about 50 testers over.

I was thinking, it'd be cool if you could get some sponsors to offer prizes (e.g. full versions of some cool games) for testers who earned a high rating... the whole idea is to get the testers to WANT to give good feedback, and to be dedicated, and to do a lot of testing.

I was thinking sponsors too but I'm having a bit of trouble figuring how they fit in. I guess the standard give us some prizes and we'll mention your name will be what happens. Companies that make use of this service will be free to reward their testers with more than just a free game, maybe some shirts or something for the top testers. Ultimately the client maintains control of most aspects of the test, we simply provide a meeting point for testers and companies, and the web software to make testing organization easy. I've got a bunch of plans on paper for this now, spent a good portion of last night on it. :)

This is pretty exciting :cool: . I can't believe that nobody's done something like this in the past... like I said, I'd do it if I had the time. I think it'll be a great resource for people who want to test, a very valuable tool for authors, and eventually a profitable endeavor for whoever's running it.

What it really all hinges on, and I have to say this one more time, fogive me, is making the whole system GOOD. It will be the difference between something that takes off and something that flops utterly. If you set it up well, you could make some serious money... offering service that is comparable to what professional beta testing offers, but for, say 1/10 or 1/100 of the price.

Ya I'm having trouble believing it doesn't exist either, and for all we know it does, but that ain't stopping me. Your absolutely right on the quality part, if there's any specific features or concerns people can come up with regarding such a system, that would really help me make sure the design fits the need. The whole concept is exciting since before I was simply going to put all this web software towards only my own beta tests. Bringing the system out into the world is a much cooler idea. Hehe, I better stop posting and get back to the drawing board/paper.

I think this sounds like a great idea too though I do have a few concerns about the pool of testers ignoring the smaller projects which don't necessarily have as nice rewards as others. (only a minor concern but thought I'd mention it)

As a tester, I would find it useful to see the current list of found and known bugs, description, and whether they have been fixed, and if so, in which version/patch/update. Not only is it nice to see which bugs have been fixed etc, but how the project is coming along. It also allows quick reference as to which *supposedly fixed* bugs should be re-tested and checked. Some kind of resolution verification for each bug from programmers point of view (to say its fixed) and then the testers to agree its fixed. A list or something quick and easy to view without scouring the forums for all the details.

As a developer, I love the idea of seeing some kind of profile on the testers - what types of software they have previously tested, what kinds of games they like, their betatesting rating etc. You wouldnt necessarily want someone who dislikes FPS's beta-testing your FPS for instance.

Trust is another issue. People anonymously signing up as testers just to get hold of beta software for whatever reasons. However, I assume that is taken care of by the developers accepting beta-testers requests to join a project, and issuing them with a link to a traceable or self expiring version.

Perhaps as an added incentive to the testers, part of the fees paid by developers could be set on one side and used to purchase software/other stuff which can be issued as extra prizes for the best monthly tester or something. Good testers need to be encouraged to stay.

-Z-

Excellent feedback, thanks. Both testers and projects will have profiles that address things like what a tester is interested in, what sort of rewards are available, etc. The setting aside money for prizes isn't a bad idea if this takes off. Beta security will probably be left up to the developer, but we will have a suggestion page on measures developers can take to help reduce leaked/stolen betas. A tracking system that ties into our own user database is a nice idea in theory but raises privacy issues despite it being a reasonable request. I was thinking of some sort of karma system that helps determine trust, as well as different levels of information provided by the user. For example, if a company is highly concerned about the beta being leaked then it could accept only users that have a certain karma rating or have provided a verified address (maybe photo ID if we ever get a really top-secret project). Some things will simply have to be dealt with on a case-by-case basis, since different companies have different requirements. Companies are also free to require additional things from beta testers, such as signing an NDA.

Here are some things that I invision for the author control panel. Of course, the testers would have a seperate "tester" control panel.

Forums
self explanatory.

Bugs
A list the bugs that have been posted, and a number that indicates how many people have "confirmed" each bug. If you click on the bug name link you get taken to a full description of that bug. If you click on the number (which is a link) a window pops up with the names of the testers that have verified this bug. Their name is of course another link to the tester's profile, where you can see there actual system configurations, etc.

Testers
A list of all the testers that are currently signed up for your project. Maybe a few items in each row showing some system info, etc. Click on the name and you get taken to the full tester profile.

Error Rating
The idea is that every tester has some "error rating" setting that is sort of an overal general rating of how well the game is doing, bug-wise, on their system: 1) no errors, 2) few errors, 3) moderate errors, 4) many errors, 5) won't run. So you would see a list of each "rating" and what percentage of testers were in that category. (e.g. "15% no errors, 35% few errors, 26% many errors..." etc.) Clicking on the link would bring up the actual list of testers who are in that category, and from there you can of course go to their profiles, and see there actual system configurations, etc.

Project Profile
Here you'd change/update/view your basic project profile:
-name of project
-type of game
-awards for testers, top testers, etc.

That's the end of my hypothetical control panel...

I imaging that a tester profile would look something like this:
-system info (everything about the tester's system)
-rating (overal score and ranking, feedback score, number of "dedicated" stamps, number of "slacker" stamps, number of "Best Tester" stamps)
-profile (types of games he/she likes, favorite games, previous games tested)


A little bit more about my rating idea: each tester would have a score for how much feedback they've given (based on forum posts, bug posts, bug confirmations, etc.); number of "dedicated" stamps (for each project the author can "stamp" each tester involved up to one time, either with a "dedicated" stamp, no stamp, or a "slacker" stamp); number of "best tester" stamps (the author can stamp one testor in a project as "best tester"); finally they'd have an overall numeric score that would be based on some kind of complex algorithm that would take ALL of the above into account... the higher the number the better the tester; then they'd also have a ranking based on the overall score out of all the testers in the system (e.g. #17/73).

P.S. It'd be good to be able to look at a bunch of testers at once, and sort them out by different things... eg, find all the testers with processors under 800Mhz who are in the "many errors" error category... stuff like that.

Okay, so I don't know if any of that actually makes sense. Obviously I've just kind of touched bases on some stuff: just to give you an idea of some of the things that would be cool to have. Obviously I'm not expecting that you'd lay out stuff exactly how I've mentioned it or anything... but I thought that my "hypothetical control panel" might be good just to give you some ideas, again--it's by no means supposed to be comprehensive.

Ya most, if not all of this doable. The core system exists, things like rankings are what I'm adding now. I love the stamp idea. I'll add as much tester info and things like that as I can. Ironically forums is probably the hardest part of the coding, I'll likely use a ready-made package for that part. Everything else is great. I don't think I'll do the one score deal, or people would just sort testers by score and the lower guys would have a lot more trouble getting stuff. There can be a score, but it would be based on averages, and having more games under your belt wouldn't give you a higher score (well a bit higher maybe), doing a good job would account for most of your score. I'm going to do everything I can to avoid having cliques of elitist testers. There's a lot of complexity to Internet communities and having a cool environment and a nasty one is a very fine line. That's a whole issue that can't be ignored. Giving people status can turn them into snobs. Thanks a lot, the design spec for this is turning out very well.

This sounds great. I've only scanned over the thread, so forgive me if this has been brought up previously. The only problem I've come up with is a trust issue. Imagine Joe Hacker signing up and downloading a near-finished game beta, just to crack it and release it on his warez website.

Other than this minor problem, I guess there's not much wrong with the idea. It'll be great to see this in action. For what it's worth, I wouldn't mind coding something if needed.

Hey all,

This beta testing thing is a really cool idea. Generally speaking I am usually reluctant to put my name down for beta testing commercial games, the main reason being is the following. I have been a beta tester, mainly for a few accouting / business management systems (using my perspective as IT support) but I have also done a little bit of game beta testing as well. Beta testing is generally hard boring work. You literaly have to sit there and logically play every move to death. That being said I wouldn't mind being a part of this. Beta testing people's shareware games would be good / interesting for me as I am looking at eventually getting into the shareware thing (Got a few learning style games I am doing at the minute :-P) and would like to see peoples approaches to design / implementation. So that being said put me down as the first beta tester :-P

Oh, another thing, speaking for myself only I don't really mind the whole reward / free game thing. When I do beta testing a generally a) like to see cool stuff or b) like to break things :-P

Oh my gosh... I just lost 30 minutes of typing... I'm gonna cry.

Anyway, to say some of what I said in a couple of sentences:

Private messages: There needs to be a good system for PM's. Authors should easily be able to PM any tester on a specific bug or whatnot. They should also be able to PM a bunch of testers at once: say, all the testers that verified a certain bug. The testers will need a nice PM inbox prominently displayed in their control panel. This is important, as an author will often want to get an answer on a specific issue from a specific tester--quickly.

Tester hogging: what about puting a cap on the number of testers allowed to sign up for any given game? Not numerical, but % based on the number of testers in the system--say, no more than 50% of the testers could sign up for any given project.

Anyway, I can't bring myself to repeat any more than that at the moment...

Great feedback, I'll address as best I can:

programmer_ted - I don't think the risk changes from the risk normally present with releasing a beta. Companies will be encourage to put expirey dates and other protection means. Once the full product comes out, I don't think any crackers will care about the beta, so it just has to be secure for that long. If I need any help I'll keep the offer in mind, but right now things seem to be manageable with one dev.

Crispie Critter - these are going to be the type of beta tests where a lot of people download the beta and use it as if they had bought the program, reporting bugs and problems along the way. I haven't heard of any small developers that hire people to do the meticulous try same thing over and over strategy that can be necessary in very large commercial endeavours. Glad to hear your interested :)

jordan - I'm pleased to say a PM system is in place (I just have to make a few additions to allow multiple folders per user). I haven't thought of putting a % of total testers limit but that might be a good idea. So far I've been coming up with the types of packages companies can sign up for. I'm aiming to provide some guarentees where if x amount of people don't sign up for the beta, the company can choose to not do the beta using this service and have their money refunded. The price of the package would go up based on how many users are guarenteed. It'll basically be a ranged system, where you can sign-up for 5-10, 10-20, 20-40, etc, where the low number is the guarentee, and the high number is the maximum. Price goes up the larger the package of course. I'm also considering some sort of pot luck package where you pay a fee that is around the middle of the price of the othes, and there are no guarentees and no caps. The more people testing, the more server resources are being used, so I think this is fair.


More feedback on what people would be willing to pay to have their beta run would be appreciated.

Instead of "packages" what about having some sort of standard thing like $1 per tester or something? And the author could put the cap on the number in the project profile... so if 0 people sign up he pays $0, and if he only wants to pay, say $20, then he just puts a cap of 20 on the project... also, I think it'd probably be good to have projects with 0 testers disappear from the system after X days, as they're obviously very unpopular, and probably detract from the site.

As far as price goes, I think something like $1-$2 a tester is reasonable. The thing is, I think the system is gonna probably need to be ironed out and changed a bit at the beginning stages--so maybe some kind of discount for developers who want to help "get the system going" by putting their projects up and giving feedback on the system, and refering their current testers over. Also, I think the system should always have a money-back guarantee: if a developer doesn't get good results, I don't think they should have to pay... this shouldn't be an issue if the system is good, and even if some do request the refund you should still be able to *at least* break even, and should be able to come out quite a bit ahead I should think.

As someone who has worked in professional test labs and is married to a tester, I thought i would put my 2 cents worth in. I've outlined some areas you should consider while testing.

Motivation
=======
Most profesional game testing needs some motivation to get people to spend enough time on the game. This can come in the form of bragging rights, credit on the game, money, prizes, swag, etc. I've had many people say, "I'd love to beta test your game." But in the end you don't get anything out of them. This is the hardest part of the process.

Organization
=========
You need to have this down to a fine process. For example, in some tests I have participated in in the seattle area, you have designated time to play certain levels, which are broken up by surveys.(more about surveys below). This organization means no wasted time for the participants in the process. This also means you get more for your investment. Always be prepared with printouts, survey tools, machines, etc.

Clear Goals
========
Testers should have clear goals of what is expected of them. With organization, this is easier to achieve. If you want the testers testing the UI or the AI, then be specific. If you are looking for general feedback, then be specific. Some people make assumptions about what you are looking for, so be clear.

Surveys
======
Surveys should be administered throughout the process. They should be brief and occur on regular intervals. The reason for this is that you need to get fresh feedback from the user. Find out if they are frustrated, happy, etc. And where they are in the game(level, character being played, etc.) This helps you find consistant problems in the game play. Also by doing it during the session and not waiting till the end, you get more feedback and the person doesn't have to think about what they did an hour ago. You forget things, so don't give them the opportunity to forget. Sureys should also include things like how they like the key bindings, joystick configurations, graphics, sound, game play. What they would change, etc. One of the best questions I saw was "How are you feeling about the game right now, and what level are you playing?" It's easy for us developers to not notice that a level is too hard, or too easy for the majority of the testers.

Testing Timeliness
=============
Testing has always been an afterthought. Many companies wait until the end and don't end up with a well tested product. This is because the testers are not considered core to the process. If you can get ahold of a tester earlier in the product development, you will be better for it. They can understand how you are designing the game, so they can come up with very unique test cases for the game earlier, and produce great test scenarios from it. The flip side, is that you could end up tainting the tester. Then they might miss stuff since they were involved in the early bugs.

OK. I'll get off my soap box now.
Thanks for listening.

Thanks a lot, I really like the idea of allowing companies to setup online surveys for the betas, that's definitely a feature that has to go in. I'll see about posting some screenshots soon, but there's some things i have to take care of first, and I have to get a good site design going.

The site design is far from finished, and depending on feedback, may look entirely different in the final product, but here is a quickie concept design I made up last night:

http://www.lizardsoft.biz/temp/betasite.png

It's definitely not finsihed for reason that I think are obvious, but that's the general color scheme and style. I'd like to get feedback early on. This is going to be the design for the main site only. The beta tester areas will be done with sub domains and will have a different, company-customizable, design. Please don't be afraid to criticize :)

It looks good. My only comment would be that the text is a bit small and hard to read. My desktop is set to 1920x1440 which is probably unussual for most PC users, but it might not be that unusual for alot of developers.

Do fonts on websites normally appear larger? This size font is used on a lot of sites, are they all too small? I'll see what I can do to improve readability.

The fonts appear fine to me. But I always use relatively low resolutions for my screen size. For my pc at home with a 21" monitor I have it in 1280x1024 resolution.

Anyways, the picture of the site looks good although I assume you haven't filled in the sidebar on the left??? One thing I'm not sure I like is the mixing of the games with other apps. I'd be surprised if you found nearly the amount of people to test the non-games as the games. I haven't really had much experience in that area though so I could be totally wrong.

I also wonder whether it would be good to keep the developer and tester portions more separated. One thing I really dislike about the garage games website is how it mixes developers and customers together with the interface it uses. It makes the site look a lot less professional to me. The website you're developing has different goals though so I can't say where the line should be drawn. I can't really tell much from one screenshot in any case though.

Originally posted by Lizardsoft
Do fonts on websites normally appear larger? This size font is used on a lot of sites, are they all too small? I'll see what I can do to improve readability.

Oh! I see what's happenning! It's an image and if you scale the window down, it scales the image down smaller. I just had it opened in a small window. When I maximized my browser the text became much bigger. And if I scale the window down really small the lines of text just become thin grey lines that eran't readable at all. Weird...

Internet Explorer introduced this horrible "feature" where it auto resizes images to fit the window. You have to find it buried in Internet Options in that long list of checkboxes if you want to turn it off. QuickTime performs similiar mischief.

Yes the left nav bar does not yet exist, and I don't think I'll be using that orange strip pattern for it. I like the point about keeping dev and user sites separate, while that won't exactly be the case, testers won't be assaulted with information that is useful only to developers, and vice versa. As for mixing games and normal apps, we'll see what happens. I agree games will probably be strongest, but since our debut commerical product is not a game, our own line-up of beta testers should help populate the site with people interested in more than just games right away. I don't think having other types of apps hurts the systems marketability but we won't know until we try.

Cooool. So what's the news now? And are you really going to call it LizardBeta? Not a problem if you are (I think), but I'm just curious ;)

That's the idea right now, LizardBeta. If you don't like this, let me know. It doesn't make sense beyond the connection to the company name, but that might help make it memorable.

Hmmm. Well, it's fine, but I was thinking of something more descriptive of the "product," like BetaShare or something. That was just a quick off-the-top-of-my-head suggestion, but you get the idea ;)

Whilst I dont really mind the name LizardBeta, perhaps it gives the impression that its a test area only for LizardSoft software. I have a feeling that many people might interpret it as such anyway b ut I could be wrong.

Just my $0.02

-Z-

BetaShare is under consideration too, I think it sounds pretty good. If anyone has other name suggestions, speak now or forever hold your peace. BetaLizard is also a possibility. Things are going nicely, I have the theme system that will allow each beta's "site" to look different (same overall layout though, since keeping things consistant is important). Hardest part right now is figuring out how to allow non-members to sign-up for betas. I want to present a form to non-members that will allow them to sign-up for the beta and get an account in one step, and there's some little details to sort through with that. I need to present the information in such a way that people understand that filling out this form will make them a member of the whole beta site, allowing them to also sign up for other betas. There's a lot of understandable mistrust on the web, so it needs to be clear that having an account will not result in spam/ads/whatever and the person doesn't have to ever visit the main beta site if they are only interested in this one game beta. The system also has to be friendly and intuitive enough so that signing up for a beta when you are not a member of the system already is not any harder than signing up for any other beta. I hope that makes sense.

If anyone has ideas on how things like contests could be run, or if they even should be, let me know. I also would like to get more feedback on pricing. The price you see in the screenshot is just me hitting keyboard and doesn't necessarily reflect the actual price.

I like the name betashare, although it might have an added connotation that the beta's are for sharing.

I like betalizard next, and don't really like lizardbeta at all.

I guess for contests you could do something like come up with the best idea for an improvement. Otherwise, overall feedback could be used if it was rated correctly.

What do you give out for a prize though? Giving out a full version of the game isn't too great, since you should get a free version if you provide high quality feedback. Perhaps giving out other games might work, but I'm not sure it would be easy to orchestrate unless the games are made by the same company/developer.

BetaShare is good and I think having a name seperate from your company is a lot better for two reasons.

1) If your using / getting beta testers that aren't under your companies financial control this can be misleading and/or bad for you. One of the ways that comes to mind straight off the bat is if a beta tester takes a copy of the software and posts it on a crack site. Seperation from your company = good.

2) It sort of sounds like a professional beta testing company (in this case I mean in a bad way:-P) In that this company requires lots of money to beta test commercial games.

I think names with beta/testing and group would be the way to go. Something like:

Beta League
Game Inspector Group
Beta Party

or what not.

As far as prizes / competitions go, for the beta testers I would think points / rating system per game tested with the highest rating testing for that game to maybe get a free copy, then for other testers to get a "discount". These two would have to be "approved" I suppose by the game owners. Other things, depending on how much your netting, such as software gift certificates or free bits of hardware are also good. Maybe even have a beta tester of the month award for people who do a lot of Quality work ?

As far as developer pricing goes, this is the area that will make or break it. Small games I think should be charge a bit less than larger, more complex games. I would get developers to do this. Estimated length of game, type of game, estimated full version price, months in development etc. That way I little puzzle game isn't going to cost as much as the 12 world RPG. A little puzzler is not going to need the time / resources an RPG would so I think should pay a fair bit less. That being said, the cost of a couple of copies of the game + 1 - 5 free copies for the testers and discounts for the rest I would think would be a resonable thing to ask for, and doesn't take to much money out of the developers pocket. I think many developers would look at this as a marketing exersise as if they have 40 beta testers testing their game not only do they a) get a good range of testers testing their game but also b) get some discounted sales from the beta testers if they enjoy the game and c) word of mouth from the beta testers. If we take the 40 beta testers scenario and say a $15 game. Let's say the company pays 4 x their games price. Thata $60. Let's also say that the company decides to give away 3 copies of their game to the top beta testers and a 20% discount to all beta testers who tested the game and also 10% discount to all beta testers who didn't submit test reports. Let's say that the entire beta testing database consists of 200 people. Let's also say there is a %5 sale rate (5% of the beta testers like the game enough to buy it). We get the following:

Beta Test Cost: -$60
3 Free Copies: -$45
2 out of 40 Beta testers (at 20% discount) ($12) +$24
17 out of 400 Beta testers (at 10% discount) ($13.50) +$220

Total Cost to Developer without Word of Mouth ?
Nothing. Developer makes $139 from sales.

Now if you also have on this site you set up like a shareware games review magazine and get the good beta testers who have played the game to review it that would also increase third party awareness for the developers games. If things really start taking off / getting large then you could do a few things such as:

* Up the cost for the developers due to lots more people beta testing / buying their games.
* Increase traffic for their games via your shareware magazine and even review other peoples games who may not have beta tested with you. (Maybe buy the game as a "reward" for good beta testers and get them to review it for you)
* More "prizes" and competitions for beta testers to increase your test pool.

Anyways enough of that huge rant, keep us informed !!!

I think it looks pretty good so far... obviously the main title page isn't gonna give us a lot of info as far as the developer and tester control pannels, etc. which are the main part of the site, but still, it looks good.

As for keeping the tester and developer cpanels seperate, I agree. Shouldn't really be too hard though... just have a tester section of the site, and a developer section.

As for offering games vs. other apps, I have to say that I am strongly in favor of only offering games, as I think that this would greatly increase the marketability and the focus/coherence of the site. You could then also register a domain name like "betatestgames.com" which would be very effective for drawing in both testers and developers (of games). Of course, I'm coming from one side of the fence, as I'm a game developer, and like you said Lizardsoft, you're the one putting together they system, and your products aren't necessarily games.... so do what you want to do, but I'd have to advise going "games only."

As for a name... I think BetaLizard or something similar is a little too off-the-wall, and too far from an actual description of the "product". BetaShare is OK, but I would go for something even more drastically focused, such as just calling it "BetaTestGames . com" or something in line with a powerful domain as I suggested earlier, that wouldn't just blend in with all the other "e-solutions" out there.


Another $0.02

Okay, looks good. BetaShare is currently the most likely name. I was in the area affected by the big blackout so hence the late response. I'm really not understanding why developers can't give out free copies of the game to beta testers. For people that do business online, this can be done for free. Those with only physical product hopefully can afford shipping, and if not then we'll see what we can work out. Instead of making it mandatory to give that program away, I might let developers choose to give something else. As long as there is some sort of reward for testing, I think that's okay.

Crispy Critter - the review idea isn't bad. The only problem I have with this is:

1) It's a beta that is being reviewed
2) I'm not too hot on the idea of good customers receiving bad reviews on my own site. Maybe some partnership with an existing review site or something could be possible, I jotted it down so we'll see what happens.

I understand the concern about non-games diluting the gaming angle but I think I can pull it off. Also, if this does take off there is no reason I can't create several sections (each with their own domain/design) that are still connected to each other but each aimed to get people interested in different things setup (in practice the split would just be games and non-games, and later maybe games, applications, professional tools (gfx, dev, etc)). I'm sort of rambling in this paragraph because this part hasn't been thought through yet.

The beta system is doing well; there was been a bit of development slowdown on Thursday because of the power outtage but in reality I was only without access to a computer for a few hours ;) It actually took me longer to get access to boiled water than it did Internet.

So what's the news?

The design of the system is more or less fleshed out and I'm working hard on coding this thing. It's gotten to be a pretty big project so even with the huge amount of code that already exists, it's going to take a bit to get it up. There will also have to be a beta test to test the beta center :D In two weeks it'll either be up or close to being up, and then it'll probably be a week before it's open to the general public. I'm trying to get it done sooner but this depends on how smooth things go and how much time I can devote to it. The three week estimate errs on the side of caution.

There are some neat features on the developer side already, such as the customization of the beta site (which is the part I'm working on now). A lot of existing Lizardsoft libraries are getting large updates to handle this unique project. The user management system is more advanced now to be able to handle the complication of multiple beta sites with multiple developers (developer A owns beta site A, but is only a tester for beta site B). That's actually the hardest part: having the system share things like the user database, but also act as a self-contained system for each beta site. Administrators, developers, and beta testers all need certain tools and everything has to integrate seamlessly without compromising security. Once this part works, adding all the really cool goodies will be a snap :)

I'm also working on CustomBar so the beta of that can come out at the same time. This way the two products can cross-promote each other nicely. Just so I know what to expect, is there anyone that would be interested in signing up for the system right when it comes out? If there are people that need to get betas out asap and would like to use this service, I can bump the development priority a bit. ;)

Im interested. But I don't need it ASAP, I hope to be ready in 2 weeks. Will it be free for us beta testing beta testers?
:D

I can't offer the service free to the people that first try it out since that would be missing out on too much income, but there will be a promotional deal that the first wave of sign-ups will be able to take advantage of, as well as additional bonuses for beta testers of the system (by beta tester I mean people that will go through all the scripts and play with them, reporting bugs, trying to break the scripts, etc).

I'd be willing to sign up... we've got a product that's almost in Beta. I do think that those who first try out the system should get some kind of nice discount though, just because there will be fewer testers and more bugs/issues to be worked through.

Hm... anything grooving LizardSoft? What's going on here? Just a slow down, or is this thing falling through?

No slowdown, no falling through. :) The site is going very well, and other than being extremely time consuming it's all going according to plan. The name will in fact be BetaShare (thanks programmer_ted), which I feel best describes the system without being too verbose. It's also the first name that really lept off the page, and there's a rule I heard somewhere where the first idea is usually the best. I'm not equipped right now to properly show off what's been done, but expect detailed information/screenshots within 48 hours. I'll also let you know at that time what is happening with sign-ups.

Woohoo! Ok, I'm happy, I contributed :D

Hmmm...I read in Thinking in C++ that you should take the first estimate of time to complete something, double it and add 10% - can't wait, whether it takes 48 hours or a few days ;)

Hehe good point Ted, but simply showing a preview is fortunately not coding, and therefore I don't believe it falls under this rule :D

Soo.... what is going on? BetaShare is well under way and now has a slightly higher priority rating than CustomBar, since I realize people are waiting on this. I'll start by going over how the process of using this system will work, from a developer perspective:

Step 1. Account Creation

The developer has a beta that is ready or will be ready very soon. He or she will create an account on BetaShare. The process will likely involve filling out a form with your information and what package you would like. We then contact the developer through phone or e-mail, make sure the account suits their needs, in some cases accept payment information (eg. CC #), and activate the account. Cheques and Paypal will definitely be accepted from the start. Credit cards are in the plan as well, but I cannot promise that option will be available immediately upon launch. Let me know if this would be an issue for anybody.


Step 2. Site Configuration

Your site is now available at minimum through http://productname.betashare.com, where productname is of course the name of your program (or some short form of it, you get the idea). If you own a domain name for your product, you can temporarily have it point to the server as well at no extra cost. The third and probably most popular choice for those with the capability is to have a subdomain point to our servers. (eg instead of pointing www.productname.com or www.companyname.com to your beta account, you can point http://beta.productname.com to your beta).

In order to maintain a consistant feel across all BetaShare sites, each account will have the same layout (or a choice of several layouts, system does support this). You will however be able to customize colors, logos, etc to your heart's content. This customizability will be provided using the online CSS editor I have been developing (more on this later).


Step 3. Tester Sign-up

Once you are done playing with colors you are free to begin accepting tester sign-ups. At this stage, anyone that goes to your beta site will be presented with a form that will allow them to create a BetaShare account and sign-up for your beta in one step. People who already have a BetaShare account will be presented with a much shorter form since most of the required information is already in their profile. Since the system depends on new BetaShare members to grow and provide a good pool of testers, new members that sign-up through your beta form will count as 3/4 of a person. This means if your package limits you to 50 people, you would be able to go over this limit if your beta is bringing in fresh testers to the system.


Step 4. Starting the Actual Beta

Once a certain period of time has passed (around two weeks) you should have enough testers to close sign-ups and begin testing. It is completely up to you who you accept to be in your beta. If more people have signed-up than your account allows (btw you can always increase the amount for so a certain amount per tester, if you decide you want more testers after all), or you simply don't feel you want certain people in your beta, you are free to decline their application. You can also simply Accept All. Now that the beta has started, your testers will be notified through e-mail and can download your beta and begin testing it. You can host the file on your own hosting, but we will also provide a certain complementary amount of space and bandwidth for files (exes, screenshots, whatever) as part of your account, so for smaller betas you can use that space. More space/bandwidth is also available for a reasonable price.


Step 5. The Beta Cycle

Now that people are trying out your program, they will be reporting bugs, posting feedback, etc through your beta site. Emphasis in development will be placed on providing good bug reporting facilities. During this time you can also post updated files, extras like movie packs, whatever. Testers will also be able to upload certain file types (images plus any custom game formats you explicitely allow that are not executables). Once the beta is done, it is hoped that you will provide your testers with a free copy of the program, or otherwise give them something to compensate them for all the free time they devoted to helping you create a better product. You are encouraged to also provide something special to your top x testers, but this is up to you. Understandably, not every company has the same resources. This is a part of the system that still needs some thought. You will also be given x stamps to award to testers you feel have done an exceptional job and deserve recognition. Every tester who participates in the test, except flake outs, will also receive a stamp that says they participated successfully in your beta.

I am considering adding in a system where if a beta tester doesn't fullfill certain quotas the system automatically gives you permission to exclude the tester from the reward and give them a negative stamp. In order to be fair to both developers and testers, most disputes will be moderated by Lizardsoft. This means that while developers do have the power to make it known that a tester is poor, they must go through us and therefore cannot abuse this power (not that this problem is anticipated).


Prizes and Sponsorship

The stamp system will qualify testers for additional prizes each month (or whatever period of the time) in order to reward effort. There may also be contests. Prizes are ideally supplied mostly through sponsorship, in order to keep the cost of running BetaShare as low as possible (and consequently keep prices lower). If you are looking for additional publicity for your upcoming beta or full release or 3rd born son you can sponsor BetaShare and provide some prizes. In exchange, we will be sure to throw your name around on the site, as well as provide you with a significant discount on BetaShare services that you can use for yourself or pass along to someone else. For more significant donations we are willing to negotiate further compensation (think free account or free copies of our own product once its out).


When Do I Get to Use All This?

We expect developer account creation and beta sign-ups will be available within a week (give or take a few days). This means that you will be able to create an account for your beta and have people start signing up for your beta very soon! The beta testing facilities will open roughly two weeks after this (I promise!), just in time for the first wave of developers to begin step 4 of the beta process (the actual betaing). The first wave of developers will receive a 25% discount on their package as a thank you for helping the site get started. :) Discount coupons will also be given at the end, which can be used for future tests or passed along to developer friends.

At this point in time we have a system that allows all the beta sites to neatly share common libraries, access the central database, etc. User accounts work, many of the convienances like private messaging and profile editing work, site customization works to a degree, and other stuff that I'm forgetting right now works as well. To provide some visuals, I have taken four screenshots of various parts of the system. Please note that this system is in development, and as a result these shots may not entirely represent the quality of the finished product. The color scheme for the forms is temporarily that of the administration area so don't panic if you don't like pink buttons ;)

Main Page - http://www.betashare.com/preview/frontpage.png

This is how the main page will look like. The layout will probably undergo only minor changes so you are looking more or less into the near future. I lied and kept the orange navigation stripe navbar borders, I would like some opinions on how that looks.


Private Messaging - http://www.betashare.com/preview/sendmessage.png

Yes, you get PM functionality :) The test message I was typing in that screenshot says it all.


CSS Editor - http://www.betashare.com/preview/csseditor.png

This is the editor that will allow developers to customize the look of the site. There are a lot of CSS elements displayed since this is a generic editor I intend to use for many projects, however for the purposes of BetaShare, available properties will mostly be colors and borders. The color selection box as you can see is in mid-development.


Administration Area - http://www.betashare.com/preview/adminarea.png

Just a little peek behind the scenes. This is a screenshot from the custom administration system I wrote to manage site content. It's ultimately what lets all this work, so I figured I might as well show it off a bit ;)


If the above links don't work it means the domain hasn't resolved yet for your ISP, in which case replace the betashare.com/ part with lizardsoft.biz/beta/.

Any comments, questions, etc about any aspect of BetaShare are welcome.

I just wanted to say it looks like excelent work, when working at microsoft we used a simmilar concept at www.betaplace.com and it was very useful. I'll definatly be keeping my eye on this as it develops.

Thanks Dan :)

Here is one more screenshot, this is a layout I've been toying around with (not yet HTMLed) for the beta sites:

http://www.betashare.com/preview/beta_project.png

It's the first page the beta tester would see when they log in. Developers would see the same thing, but with appropriate additional control panel links for administrative stuff. Colors are customizable, along with the logos (eg. instead of seeing the Lizardsoft logo people would see your company or product logo). The color scheme on the Quick Stats box doesn't match because this is a combination of two potential layouts I was working with (just pretend the colors match). I'm really not sure if Main Lab, Downloads, Forums are the best three links to have at the top, or if they should be there at all, I'll leave that up to you guys to tell me. The Lizardsoft Laboratories stuff is actually the layout that I was originally using for the private beta center I had been setting up for LS products.

It is also possible that developers will be able to choose between multiple layouts. I haven't completely decided this yet. The system has supported multiple layouts from the ground up, but as mentioned somewhere in this monster thread, keeping a consistant feel between beta sites is also important. Jury still out on this one.

Anyway, let me know what you think. This layout definitely needs some tuning at the very least, so please don't be afraid to criticize it. I have no problems changing or even scrapping it if need be. :D

Looks good so far :) .

Looks great!

how about adding category to test the demo version too?

gilzu: I'm assuming you mean allowing developer to also upload a beta of the demo so people can test that out and post bugs about it. I like this idea. It'll probably be accomplished simply by putting it in the downloads area and when people report bugs they can specify if its specific to only the demo.

Originally posted by Lizardsoft
gilzu: I'm assuming you mean allowing developer to also upload a beta of the demo so people can test that out and post bugs about it. I like this idea. It'll probably be accomplished simply by putting it in the downloads area and when people report bugs they can specify if its specific to only the demo.

exactly :D

-= Bump =-

Any Updates ?

If you need a hand with anything send me a PM. Not to sure about PHP but I can do graphics / testing :-P

Looking at releasing Friday (or so www.betashare.com claims :p). It might be delayed a couple days if I end up having to go to Toronto though so please don't me hurt me if you have to wait until Monday. The code for the first part is basically there and working, with a few details regarding user profiles and the developer control panel being finished up.

Regarding testing, there will definitely be a need for that as the rest of BetaShare goes up (and it will be in the promised two week time during which developers are getting tester sign-ups). I already have had a small group of people test out the sign-up and if I'm feeling lucky I might post a live CustomBar beta tester sign-up tomorrow night so people can see how the system will work. I'll let you know when I'm ready for some more testers to tear up the system.

I would appreciate it if anyone that has a beta ready to submit right away and is considering this service would let me know, either in this thread or through PM. This is so I have some idea of what to expect come the release date. Since it takes time to get testers to sign-up for your beta (two weeks minimum, but developer can keep sign-up open longer than this) a beta that is almost done qualifies. Getting a few betas up in a reasonable amount of time will probably decide if this service takes off.

On a similiar note, is someone interested in signing up within the month who is unable to pay through Paypal or cheque? Credit card processing might have to wait a bit and I need to know if this is a serious issue for anyone. As this thing gets going I'll be adding every secure payment method that I can, but with this time-frame Paypal and cheque are most sensible at the moment.

This has been a very interesting project to code and I'm happy to say the result to this point is very good. All the major foundational things like user logins, page system, layout system, navigation system (yes, you read that correctly :p), etc are functioning and complete. A good part of the stuff that isn't needed until stage 2 (such as bug reporting) is functional to some degree as well. Thank you everyone for your ideas and support in what is probably the most spontaneous project I have ever done. :D


PS. I'll try to post a couple more screenshots tonight!

Okay here's a screenshot which showcases the developer control panel and the latest incarnation of the betashare.com layout. You'll probably never actually see the dev panel with this layout since you will be using it from your customized beta site (or lab as I like to call them). The layout system I wrote allows me to switch between layouts just by changing one line of text (yes I must brag about this :p). I'm using the main layout here so I can show two birds off with one screenshot ;)

http://www.betashare.com/preview/beta_devpanel.png

The navigation bar is not fully done so excuse the lack of some vital links. For some reason it's one of the last things to be completed. The links in the dev panel may also change as I organize and move things around.

I would also appreciate if people who are using a more exotic browser or a version of IE older than 6 could check out this page:

http://www.betashare.com/preview/lab1.html

and let me know if it looks different from:

http://www.betashare.com/preview/lab_proper.png

This is the original layout template I created for http://www.betashare.com/preview/beta_project.png (this image is a rough idea of how the labs will look). I'm looking for any layout bugs or problems.

I tried lab1.html in Netscape 7.0 and Opera 7.11 and it looked identical to the picture in both cases (and the one in IE 6.0 :P)

Nice job!

Just a suggestion. I found the dashes around the menus on the left navbar irritating. It hurt my eyes.

Thanks ted. If someone has IE 5 I would appreciate if they checked with that, since that browser has some CSS sizing bugs.

Cartman, I'll see what I can do about the navbar. I might release the site with the dashes, and then hope someone suggests a better solution. I tried simply orange lines with a black outline, which don't look *bad* but they lack that extra something. Design making eyes hurt is definitely a bad thing, so it will get fixed sooner than later.

Here's the first thing the developer will see, minus spelling mistakes, when visiting beta site for first time:

www.betashare.com/preview/beta_dev.png

Colors/logos/etc are customizable. I tried choosing a default color scheme which isn't completely horrible while still not something anyone in their right mind would keep (thus encouraging customization of the colors). I gave the whole top for the logo graphic and any decoration developer may want to put up there. The top navbar sits on top of the graphic. The layout is fixed at 755 pixels in width. This is the site that will, after the beta sign-up is opened, advertise the beta to people and provide a means of signing up. Once sign-ups are done and the actual beta starts the system replaces this layout with the lab one posted earlier. I'm working on having them share the same set of style information so as to avoid forcing the developer to re-customize the colors. Feel free to comment and make suggestions, as there is still time thanks to the situation below.

It looks like I will indeed be away for a good part of the weekend so it's probably for the best if the opening is delayed until Monday.

Launch will be soon, a day or two. Just working out some business details, clarifying information use, and checking over everything. :D

Woohoo! Can't wait to see the end result :D

I hope it's really successful and you make millions :D

Hehe millions would be nice but I'll settle for it making enough to have been worth the time :D I have a good mind to write a dev diary or something of that sort once this is complete. Trying to write a system that can effectively run multiple customizable sites has raised some very interesting problems. Making sure this thing can scale up as it gets popular and be easily extended while maintaining a certain level of customizability also took some creativity. The delay has been because of many small details. Forgetting one little design issue like screenshot formats can cost hours (as I proved to myself yesterday). Almost everything is of release quality now though, so if we can pretend that when I said Monday I meant this monday and not last Monday we should be cool.

I'm having a lot of fun right now preparing the CustomBar beta sign-up with this :)

Edit: screenshot - http://www.betashare.com/preview/cbbeta.png

Very cool! I'm preparing a beta today so count us in (Jordan and I :P)

This is it! BetaShare (http://www.betashare.com) is now officially open :D

As of 11:22 EST the site has gone live and is now ready for developers and testers to sign-up. Feel free to try it out and let me know of any problems. The only restriction I make regarding that is that should you find a security problem, that you report it to me privately.

Tomorrow I will post the CustomBar beta sign-up and then report the opening of BetaShare in the Announcements forum. I am going to follow the advice of not letting many people know about version 1.0 and am henceforth declaring this Dexterity week. You are free to let your friends know about the site, post about the release on your own sites, etc, but I will not be doing any submissions to news site or marketting for one week. This should be sufficient time to tidy things up and incorporate user suggestions.

Thank you everyone that has helped keep this idea going, given all the great suggestions, and allowed me to get the site this far.

Over the course of this week I will be extensively tweaking the site, adding more information about the system (including some screenshots of the dev tools, which the info page is currently missing). The main page design will be altered a bit as well. This is the best time to let me know if something is appearing wrong in your browser or any other bugs/suggestions for that matter. This is your chance to really have a say in how BetaShare shapes up.

If you have a beta that is due for release in a month or sooner, now is the best time to sign-up for BetaShare. As a big huge thank you to people that agree to be the first users of this system, I am offering a 25% discount coupon code. This magic code is FJ43JL Feel free to tell others about this code, I have no problem with anyone that signs up in the next few weeks getting this discount. The coupon code is valid for all of October 2003. Coupon codes may not be combined. As an additional incentive, all BetaShare clients will receive an additional single-use 25% coupon upon completion of their beta test, which they may use for future betas or pass it on to a friend.

My own project of a year and a half, CustomBar, will have its beta sign-up posted tomorrow so there will be at least one beta up there soon :)

If you have any questions regarding this service, don't hesistate to ask! I am going to catch up on sleep in an hour or so but I'll be sure to address any questions tomorrow.

Some Screenshots:

http://www.betashare.com/preview/cbbeta.png
This is the screenshot edit page in the developer control panel.

http://www.betashare.com/preview/devinitial.png
This is the control panel and the beta as it looks before the developer begins to customize it.

I will have more screenshots of things like bug reporting once those scripts look a bit prettier and are more completely (they will definitely be done before any beta reaches the beta testing stage).

Woohoo! I'm almost done preparing the Mission: Land! beta 1.0, so you'll be seeing us around in the next couple days. *nudges Jordan...*

Whoa!!! 149$. Glad I'm not paying :p

*nudges Jordan again...*

Yeah... we'll have a beta coming over within a couple of days hopefully... though what's this thing with the $150 price tag! I was really budgeting for $100 :p --but I guess we'll give this thing a whirl.

Don't forget the coupon code guys!! :) FJ43JL - these six characters save a decent amount of money. Only time will tell if $149 is affordable as the normal price, but I believe the service will prove to be worth it. ;)

*Goes back to working on getting CustomBar beta sign-up ready*

Hmmm. I don't think the price really needs to be changed, but do you think two months is a little short? Reasoning: most pre-release betas can't take much longer than two months (for indie games), but what about beta tests for updated versions? Following the guidelines of Steve's articles, games will be updated many times before they bring in any real money. I'm thinking three months would be more worth the 150$ pricetag, because those extra thirty days can mean a lot of updates. Just an idea.

Okay officially it's still two until I can see how things go, but hey, you guys can have three no problem.

Well, looks like a great service so far. We have our site customized and everything, but we have one problem. We're looking to release our game fairly soon, if not before you even receive payment (and use the service for beta testing of updates). This would mean that we should also have the beta testing running before the 14 day minimum open-to-testers period. What we're suggesting is that, if not too much trouble, the developer chooses when to close the doors to testers and when to start the beta testing. For instance (in our scenario):

We release the game, and you receive payment.
Beta test site is launched.
Testers sign up for a week.
We start beta testing period with the testers we have (we already have a few from our own alpha testing efforts, and these would be enough to start with).
Testers continue to sign up for 2-3 more weeks.
We close doors to testers and continue beta testing until our two months runs out.

Would it be too difficult to modify the system so that the developer can start the testing period while testers are still signing up? Thanks for reading :D

Okay I can accomadate these wishes to some degree. As mentioned earlier, I have released BetaShare the moment the sign-up part was ready, with the intention of finishing everything else before anyone's two week minimum period was up. This way people wouldn't be waiting for the darn thing to be released when they could already be accepting sign-ups. I understand you are eager to release your game and start testing asap and I can bend the rules and allow you to start testing this weekend, if you are willing to accept that you are essentially using a beta of BetaShare at that point, and that certain scripts simply won't be available immediately. I am also willing to modify the system so that sign-ups are still possible when the actual beta is going on, and to not count your testing period as officially started until the usual two weeks have passed.

Looking at what is done so far and how much work I can get done this week, if you choose to start testing early, you will be operating with file hosting, bug reports, news, and possibly feature requests. Forums, polls, surveys, the SDK, etc, and some advanced features of released scripts would not be done and would become ready sometime during the week after. If this works for you, I'll do my best to let you start testing this weekend.

Right now I'm only making this exception for you. Allowing people to sign-up while the beta is already going on makes things a bit messy with regards to what the official rules are so I can't really let this be an official feature right now.

Oh, I thought everything was already done :D. Well, we'll probably end up waiting the two weeks, but if we really need to start testing before that we'll let you know. Thanks a ton!

Alright, sounds good. I'll have some screenshots of the actual tester tools soon. :D

First of all, I'd like to say that the system looks great so far, and thanks to LizardSoft for being so flexible and accommodating with everything that we've (Ted and myself) suggested so far.

Now, there is one fairly major issue that I'd like to bring up, and that is the tester cap. As I mentioned in in a previous post, 50 isn't a bad number if we assume that all these people are active and participating. However, I have found that a lot of people who sign up to "test" something just put their name in the form for the alure of free software. Not all testers, granted--and the great thing about BetaShare is that a pool of testers should hopefully develop, and due to the ingenious "stamping" system (I wonder who thought of that ;) ) developers will hopefully be able to pick and choose among qualified applicants, and end up with 50 people who are just rearing to go and test their software. Unfortunately at this stage that pool of applicants doesn't exist... Now I know that each new tester we bring into the system only counts as .75 of a person, but if I send all the people who have signed up for my alpha testing over to betashare probably 40 of them will sign up, but only about 10 of them will be active.... so I could easily end up with 40 "dead weights," and only 15 "real testers."

What am I saying in all this? That either the tester cap should be doubled, or that developers should have the ability to boot testers who have signed up but are not being active, thereby freeing up space for another applicant. This later solution is undoubtedly the best, as it will prevent the "clogging" that would otherwise naturally occur. It would also be good if the booted tester in question received a "negative" stamp, that would serve in the opposite function of a positive stamp.

Now people might worry here about descriminating somehow against testers, or developers being unfair in booting them, or at the very least developers having different standards, and therefore making it unfair for the testers. Well, I'd say that this is all a fairly small and overly-hyped evil. Are developers really gonna go around booting testers for the fun of it? No! Developers are going to boot testers when they feel they are getting nothing out of them, and therefore they are not getting their money's worth. I'll tell you right now that there is nothing that would make me more irritated with the system than to get all my 50 slots filled up and discover that only 15% of these people are dedicated... or even only 25%... or even only 50%.... heck! I want ALL my testers to be doing SOMETHING! I mean, this is a $150 service here, and I'm essentially paying for testers (and the system of course). You could have the negative stamps go away after a certain amount of time, so as not to permanently brand people...

That was long. Well if there's one thing I want to say it's this: for $150 my 50 testers better be doing something.

And again, LizardSoft, thanks for the great support so far! :)


P.S. I also think that 50 is a tad on the small side for a $150 service... especially since it doesn't seem like the number of testers severely influences the service cost.... I would be much more inclined to get more testers and less bandwidth/disk space/other stuff, all of which more direcly influence cost anyway... but again, I think 50 DEDICATED testers might do it. Also, .75 for new testers does bring that number into the 60s...

Okay I've thought about this for a bit, and I think we can come to a good solution with a combination of the things you have suggested. From my own experiences, the tester:deadweight ratio isn't as drastic as 1:4, but it is a ratio that will vary from beta to beta. I would prefer to increase this ratio more consistantly for all betas rather than trying to brute force things with a higher tester limit.

The tester limit actually affects a great deal of things. It determines how much file hosting has to be provided, what the potential load on the server will be, and also affects the developer's perception of what is being purchased. If the limit is set artificially higher to compensate for a low percentage of good testers, one of three scenarios will inevitably happen:

1) Developer doesn't get even close to that many applicants. Getting 100+ applicants is difficult and not required for smaller programs. Nevertheless, the developer feels cheated due to being so far from the limit.

2) The developer's beta looks interesting and enough people apply to actually hit the tester limit. The developer is excited at having so many "testers". Unfortunately, the amount of testers that actually do something is significantly lower, maybe 20%. Now the developer is disappointed and feels cheated.

3) The developer is a pro at running betas and manages to fill the limit with testers that participate. This developer is happy but BetaShare doesn't make any money because the package was never intended to acommadate such a large tester amount.

I feel a better solution to the problem is increasing the percentage of useful testers, and the value of each individual tester. As you mentioned, things like stamps will eventually help the developer make more intelligent decisions about applicants. I also like the idea of being able to boot testers and replace them, so this can be done. The BetaShare toolkit for Windows, which will be available shortly after the rest of the services, should also raise the value of the individual tester as well as the service. The feature that I am most excited about is the ability for this toolkit to collect developer-defined usuage statistics. Here's the post by Dan MacDonald that inspired this:

There can be some middle ground. Sitting in an IGC session yesterday listening to one of the presenters. They were mentionling how they have closed betas for their games, and in those beta builds they log just about everything. The testers all know that the information is being logged. Since they are indies they can't afford big glass rooms and usability sessions in their offices... but with very expressive logging they can see that someone playing through the game did great on levels one two and three, but on level 4 it took them 2x as long, and then they quit in level 6 after repeating one part of the level over and over.

When I read this the first thought that came into my head was "Of course! This is perfect!". Princec has already shown us how valuable such statistics can be, so I think this will be a feature of BetaShare that will prove invaluable and will really round this out as a full beta service. Now even testers that stop posting after one or two reports/comments will still be very useful since the game will be sending these statistics. Additionally, you will be able to get an idea of who is playing the game and who isn't. Those who aren't you can find out why (people often won't voluntarily tell you they stopped testing because they didn't like the program).

I raised your tester limit to 75, which I think will work out quite nicely for you. Keep in mind that an applicant is not necessarily a tester. There's nothing stopping you from getting twice as many applicants as your limit allows, and then deciding which people are best suited for the job. By all means get as many applicants as you possibly can. :)


On another note, the bug reporting script allows a bug category to be chosen. These are intended to be very broad and general categories, and will serve mostly as another way of sorting bug reports. For example, if the developer decides to take a break from coding and spend the day fixing grammer/spelling/formatting problems, he/she can sort the bug list according to Language & Formatting issues. Here's the categories I have currently:

Crash/Hang - program crashes/locks up/explodes
Graphical - rendering problems, mostly for 3D games
Gameplay - unbeatable levels, script bugs, etc
Language & Formatting - spelling errors, bad grammar/translation, text spills out of windows
Compatibility - program doesn't play nice with other hardware/software
General Bug - programming glitches that don't fit above.

I would like some feedback on these categories. Once again, the primary goal of these categories is to make bug organization easier. I would also like to rename the Gameplay category to a word that isn't specific to games, since other application types can have data file problems to (for example my program, CustomBar, can have bugs in the various prefab scripts).

I pretty much agree with everything you said LizardSoft. My "I would be much more inclined to get more testers and less bandwidth/disk space/other stuff, all of which more direcly influence cost anyway... " statement was a bit shallow and misspoken. I apologize. I agree with your 3 responding points, and I definitely see how a greater number of testers generally increases maintanance costs... I won't go into the details of what exactly I was trying to say, because all your points make sense.

As for the tester limit, I completely agree that a "better solution to the problem is increasing the percentage of useful testers." I was definitely trying to emphasize this more than increasing the tester limit (I may have come across the wrong way there). I think the "stamping" thing will work out great, and if you're allowed to boot testers as I suggested, then I don't have any concern here at all... my concern was that I'd be STUCK with testers who were doing nothing. Also, as you pointed out, if the testers in question are still playing the game then they're still valuable even if they're not posting necessarily, as long as you can get data from them... your BetaShare toolkit sounds fantastic. Again, the problem usually is that small developers don't have these kinds of tools.

Thanks again for listening and responding to my suggestions. I'm getting more and more excited about using this system! :) (you say you plan to have all the tools implemented within a couple of weeks?)


P.S. I certainly didn't mean to pressure you into increasing our tester limit... as I said above, I'm much more concerned with the quality of tester--but thanks anyway.

Originally posted by jordan1207
Thanks again for listening and responding to my suggestions. I'm getting more and more excited about using this system! :) (you say you plan to have all the tools implemented within a couple of weeks?)

No problem. Yes, the original tools will be ready within two weeks. The Windows toolkit will follow very closely. I can tell you right now the toolkit will be a dll, supported under Win 95 (with IE) or higher, be fairly easy to integrate in any application, and not require any C++ features.

The Windows toolkit will follow very closely. I can tell you right now the toolkit will be a dll, supported under Win 95 (with IE) or higher, be fairly easy to integrate in any application, and not require any C++ features.

Sounds great. Can't wait :D