Plans for Quality Assurance?

[Vrav]

It seems like good quality assurance is too often overlooked in games these days.

An incredible number of popular Steam releases, etc, are completely filled with bugs and crashes and subsequently have to be patched - some of which never get fixed at all. Most developers seem to wait until the end to run full-on QA, and by that point publishers and other such monetary forces are pushing for early release.

Since Wolfire is publishing the Overgrowth on its own, I don't think a publisher is going to be pressuring the team to release early. But we have to assume it's going to be finished and released eventually. At that point, how much QA will have been applied?

I understand the Overgrowth is still in its alpha stages, ie, it's not really a game yet. But it's getting there. Once all of the game systems are in place, and the actual building of the game commences - scenarios, final menu systems, a campaign - will its development be subjected to extensive QA, through and beyond the finish line?

I'm sure there are numerous hardcore fans who'd be more than willing to slog through checklists and the (subjective) tedium of thorough testing to ensure a quality game is released, and they'd more than likely volunteer to do it for free. I'm not doubting David Rosen's programming capabilities here, since he's always struck me as dedicated and thorough - but in my opinion, QA should be applied liberally and extensively throughout development.

Maybe some of Wolfire's team members already perform this level of QA from week to week - in which case, kudos. But if not, I think once the systems David wants for gameplay are in place, some real QA - even if it's on the part of volunteers - needs to take place. Sooner before later.

If there are any fans out there who have done QA for a game before, you know how important it is. There are always more bugs to find. Maybe some of you are at this already with the current system of reporting bugs? If so, commendations! I'm only creating this thread because I'm not sure what degree of QA Wolfire has planned, or what procedures are ongoing already.

[Garabaldi]

I would imagine that the weekly alphas ensure QA is taking place around the clock, without any staff dedicated to that purpose required - Wolfire has quite a large pool of preorderers to pull data from - people with varying specs, drivers, and software - they've got hundreds of people already playing each alpha and finding bugs and issues and reporting them as they arise - most development studios only have small groups of testers with a very strict and specific schedule and multiple departments that all have to communicate with one another and that are all focused on one single area of development.

In short, I don't think this is an issue that warrants any worry - we're all doing our own QA tresting by playing around with the alphas (and hopefully reporting any issues we run in to).

[Endoperez]

This could cause some problems. The problem isn't in the main features, but in testing that every feature always work. How long did it take before people noticed rigging and animation editors didn't work? Who'd notice if a specific rare weapon's (say, a club tied to the hands of an arena-fighting, imprisoned wolf) range was accidentally changed to almost 0, so that the weapon is almost useless? How long would it take until someone noticed that a single map, which is rarely played, has a bug in specific part that isn't near the enemies, but where a new player might want to go for some reason?

Actual QA testing includes going through the boring stuff too. Fans usually only test the fun stuff.

[Silverfish]

I'd be more than happy to do some real testing for OG once they feel like it's needed. I'm not sure it's really that important to find bugs at this stage since it's so much a work in progress.

But if the guys wants me to find bugs I sure will make a list of things to test, start up OG and test the crap out of those features (SWEDISH STYLE!).

EDIT:
I asked Jeff about it in the IRC and he says that they would appreciate if people where bug testing and reporting bugs even in current alpha builds. So there you go!

[Jacktheawesome]

Eh. I don't really care about it right now. Once the game gets into Beta, I'll expect the bugs to start being exterminated, but until then, I'm quite happy to mess around with sometimes-buggy new features. Even if some bugs do slip through the final release, Wolfire releases a new Alpha with significant improvements every week. They could get out a quick bug fix in 1-2 days.

[tentus]

My experience with bugtesting (from both tester and developer sides) has been that it is slow, communication-heavy work. On the developer side it is lots and lots of "email me this extra-massive log file" followed by "try this exe and then send me the file again." On the tester side it is a lot of trying to duplicate extraordinarily specific circumstances, the most obscure of cornercases imaginable.

Best way, in my experience, to accomplish good QA in a timely manner is to make even the most single-purpose of things scriptable. For example, write scripts to have it literally rain weapons over complex terrain, and then track any weapons that moved more than you expected them to. It produces results a lot faster (and more reliably) than having a horde of players throw things at corners.