PDA

View Full Version : Whatever happened to bug-testing?


Mota
12 Sep 2007, 19:10
Now don't get me wrong here, Worms is one of my favourite game franchises and WOW2 on DS is currently being played non-stop at my house, by various people.

... so hats off to Team17 for that...

... but didn't their games used to be a lot less buggy? I never had any trouble with Worms 1,2,Arma,WP etc...

Worms 3D, on the other hand, was unplayable (on Gamecube). It would usually crash in less than 5 mins of play. I returned + replaced the disc many times, and even tried each copy on friend's cubes too. Always with the same result.

And then WOW2(DS) - There's the 'flak bug'... the single-card play is completely broken (loses sync every time for me, thankfully I bought several copies of the game) and worst of all - that 'shopping' scheme in ranked play where nothing ever drops. I've played that scheme about 20 times now, and seen about 2 weapon drops in all. I've been forced to surrender each time due to sheer skip-go boredom.

I imagine we'll never see a 'fixed' version (Mario64DS and Harvest Moon DS were both re-released with various fixes) as it would probably cause online content mismatches... but I already have my fingers crossed for a bugless WOW3 :)

Spadge
12 Sep 2007, 19:19
I think if you saw the huge check-list that was used for testing, you might reconsider those comments. It's an incredibly complicated game with an absolute raft of features, options and modes. Not to mention online play.

wave
12 Sep 2007, 19:43
i've always found worms games 'unbuggy' its part of the reason i love them. That and the l'il worms being so cute.

Mota
12 Sep 2007, 19:53
I think if you saw the huge check-list that was used for testing, you might reconsider those comments. It's an incredibly complicated game with an absolute raft of features, options and modes. Not to mention online play.

Which comments might I reconsider? Admittedly the topic title is a little provocative (probably a bad idea!) but the post contained nothing but facts. I appreciate that publishers limit the time available for testing and so forth, and I certainly intend no disrespect to the awesome work the Worms team do. I'm also very grateful that you've taken the time to respond - I've been shooting your namesake with bazookas for many many years now - even on the monochrome gameboy!

From my own coding experience I know online features cause all kinds of issues when writing games. Is there any possibility that (in the case of cart-based games) future titles could have a portion of SRAM/Flash reserved for patching content?

No doubt the single-card sync issue stems from timing discrepancies between supposedly identical DS units - something that has hampered my own homebrew dev work from time to time.

I really would love to know what happened with Worms 3D on the cube, though. Was it tested on hardware or just the dev kit? Did the same issue occur with US copies? (mine's PAL).

Please don't mistake this for inflammatory criticism, I'm genuinely interested in the development process.

parsley
12 Sep 2007, 22:14
Which comments might I reconsider? Admittedly the topic title is a little provocative (probably a bad idea!)
Well, you've answered a good part of your own question there...;)

From my own coding experience I know online features cause all kinds of issues when writing games.
Yay! Simply, Yay!

Is there any possibility that (in the case of cart-based games) future titles could have a portion of SRAM/Flash reserved for patching content?
It really ain't up to us. If the system supports patching, then we can make use of it (Worms XBLA was patched), but otherwise, there's nowt we can do.

No doubt the single-card sync issue stems from timing discrepancies between supposedly identical DS units - something that has hampered my own homebrew dev work from time to time.
I don't know about the DS version, but experience makes me feel uncomfortable with this. Usually, there are three time-lines in a game:

Real time: as you'd expect

Draw time: this is when the renderer draws it's view of the logical game state. It tries to balance the discrepancy between real time and game time as best as possible.

Game time: if the *logical* game is written to update at, say, 30Hz, then each game time 'tick' is 1/30th of a second. It doesn't matter how often or seldom (in real time) the game tick executes, each one is exactly 1/30th of a second. To put it another way, it doesn't matter how frequently or seldom you update the game, every 30 updates, 1 second of game time has passed. (That may have taken, say, 0.5 real-time seconds or 10-real time seconds, but it's still 1 game-time second. It leads to peculiar units like seconds per second... seconds of game time per second of real time, that is).

Given this structure, it's unlikely that it's a straightforward timing issue (nothing's ever straightforward on networks... bah).

Was it tested on hardware or just the dev kit? Did the same issue occur with US copies? (mine's PAL).
All games are tested on a variety of hardware and for different purposes, but if a game can be tested on retail hardware for any purpose, then it is.

Here's a fairly generic description of the aim of the different layers of testing: the programmer tests the code; the developer's testers test for completeness, help the programmers identify their cork-ups and pre-empt the publishers and manufacturer's testers; the publisher's testers try to break the game; the console's manufacturer tests for standards compliance.

(By far the in-house testers have the hardest job.)

All layers will report all bugs but, because they're testing for different things, they'll find different categories of bugs.

On some consoles, only the manufacturer can test on real retail hardware and this is where things can slip through: it is not their aim (nor their responsibility) to try and break the game. However, we all test as close as we can to the retail hardware.

(As an aside, at least one publisher will provide the developer with videotape of the tester in action: their determination to find breaks is that extreme that developers simply couldn't accept that the test department would do that...)

All of which brings us back to your original paragraph: no-one in the whole chain of things wants to see a single bug appear on a single user's machine and we expend vast amounts of money, time and personal investment in trying to achieve that infeasible goal. Your thread title was, perhaps, ill-chosen.

Mota
13 Sep 2007, 01:36
Parsley, many thanks for your in-depth reply :)

Re: Patching - Picross DS features downloadable content which is stored on the cart, I would have thought it would be possible to store patch information in the same way. No doubt someone better informed is going to tell me otherwise, though ;)

Re: Timing - Supposedly, the Arm processors in some DS units talk to each other on a slightly different timing to others. It's been blamed for bodging homebrew netcode before, though admittedly I'm not sure how relevant it is to the game in question - it was just a suggestion.

Your description of the testing process was really interesting, and not how I'd imagined it to be honest! How long, typically, does a publisher have to test a product - between receiving it from the developer and sending it to press?

Thanks again for taking my questions seriously, and for the cracking work you do. Clearly I underestimated the logistics of the modern release process... I bet things were easier back in the Amiga days, eh? :P

Wormetti
13 Sep 2007, 04:49
It would be great if you could release a downloadable beta demo or two, purely designed to get feedback/bug reports from a group of testers larger than you could ever pay for.

enigma_0Z
13 Sep 2007, 14:33
It would be great if you could release a downloadable beta demo or two, purely designed to get feedback/bug reports from a group of testers larger than you could ever pay for.

They'd have to be RSA-signed, and people who want them would have to either have the WMB software and a Nintendo WFC USB thing (or compatible wifi card... :D) or get it from a store...

Wormetti
13 Sep 2007, 14:46
They'd have to be RSA-signed, and people who want them would have to either have the WMB software and a Nintendo WFC USB thing (or compatible wifi card... :D) or get it from a store...

True but the PSP version could be downloaded from the publishers site just like the PSP demo was. I've heard rumours of Wii eventually being able to send demos to the DS but don't know if that will happen.

Moiz575
14 Sep 2007, 12:21
ive also heard they're gonna use the wii to semd demos to the ds but when is that gonna happen? that would be pretty cool.

Mota
14 Sep 2007, 16:38
ive also heard they're gonna use the wii to semd demos to the ds but when is that gonna happen? that would be pretty cool.

I heard this too - I would imagine it'll be very similiar to the DS Demo Kiosks (Which you don't usually see in the UK) i.e. the demo stays on your DS until you turn it off. I'd love to see devs using this for open betas etc.