I found this over on Darth Hater and thought it was an interesting way to handle beta feedback.
The use of heat maps, chat logs (to catch issues people talk about but don’t bother to /bug) and other various feedback items tagged for more immediate consumption makes good sense. Use of behavior analysis is neat along with the fog of war as an indicator for areas players generally miss.
I’m really stunned by the analytic techniques they’re using. I guess something good can come from all the targeted marketing algorithms and practices.
On the other hand… big brother is watching.
Q: How do you stop or catch leaks?
A: With all the data tracking we have we can see a single ability used at a single location for every player. If a leaked video shows a player wearing a certain piece of armor or doing donuts on a speeder in a certain zone we can find them very easily.
EDIT: Link to Gamasutra coverage of the same information.
This sounds a lot better and as a tester, I have to go clean my pants now.
How does user feedback influence content creation for The Old Republic? How do you gather this data?
Testing and the use of data generated from testing has been an integral part of our workflow for more than a year now and has been critical for us in validating the game design, rooting out problems and improving the overall game.
Data is gathered via a broad set of methods, including automation, very high detail metrics about user interaction with the game, professional focus testing, in-game player feedback systems, private testing forums and direct contact with individual testers or entire groups via chat.
It’s possible for us to drill down into the game interactions of every single tester and correlate their feedback directly with issues encountered in-game. By using a several different data sources, we can eliminate a lot of the usual bias encountered in direct user feedback.
High detail user interaction metrics also help us analyze complex content issues, develop fixes and most importantly, validate the success of those fixes a few builds down the road.
I’ve been begging my developers (I has developers) to put something like this in for awhile now. When given the ‘not enough time’ excuse I suggested it was already there (mostly) in the Undo stack – just allow us to see it or get a dump some how. This would greatly increase the value of less experienced testers supplying feedback (like interns or contractors new to the product).
They still didn’t do it… bastids.
I got to putter around in Lego Universe some last night and I admit I did see some subtle changes and a few bigger ones.
The zones are mostly the same though there are a lot more quests and daily type quests available now. All pretty simple, but still more than I remember there being which is a good thing because I found it pretty sparse.
The faction tokens drop more frequently now and they’ve added a new choice per faction. The Sentinel group has Knight, Samurai and Space Ranger while Paradox has Marauder (?), Sorcerer and Assassin options. I didn’t look at the other two factions.
I know they added a couple of zones with one being sort of like a capital city and the other(s?) being adventure areas which I haven’t seen yet. I did venture into the Nexus Tower which hosts the advanced sets for the factions and headquarters for each of the faction and the bank! A much needed mechanism.
With the added quests the XP rewards are better and there is a smoother progression to the game since quests and achievements are the only ways to get XP.
I don’t remember the social system too much but I think the one they have in place is improved though it still lacks an easy method of getting to your friend (at least I haven’t found one) that games like Wizard 101 have.
Both kids seem much happier with it.
I have a few reservations about SW:TOR.
First of all, the animations continue to get mentioned as being less than fluid. It’s something you can sort of see in some of the videos. This is also mentioned in a hands-on impressions post from Tap Repeatedly which also didn’t have a lot of nice things to say.
I thought that might be just raised expectations due to all the hype Bioware is throwing out there.
Which brings me to the second point, there isn’t really a whole lot of hype out there from Bioware is there? Most of the hype is fan driven. Digging through the forums and the official SW:TOR site I’m hard pressed to find a lot of specific information.
This sort of scares the crap out of me. It’s about two months from launch and I’ve no idea what races are allowed to be what classes? What do the races actually give you? How do I join friends that might have started on a whole other planet?
There are a lot of basic questions not answered yet and all Bioware is releasing are detail descriptions of planets you may visit without any solid information on how it impacts your game play. Looking through the information the site offers leaves me without a lot of real information.
Latest video released was an opinion piece about who’d win between a Jedi Knight and a Bounty Hunter which supposedly details some of their special abilities. I didn’t see a whole lot other than the force leap (which we already knew about), lightsaber throw (again, knew about it) and the need for the Jedi Knight to close on the Bounty Hunter to win. On the Bounty Hunter side we saw a few items that appeared in other videos already; effectively a root, shoot from the distance and maybe one more gimmick that left such an impression on me that I can’t remember what it was.
Ouch! Too tight!
Maybe that’s not a bad thing that I don’t know every inner mechanic about the game before it releases?
Still, two months before release and we’re seeing posts like this (in response to someone suggesting they should be in the ‘polishing stage’):
Keep in mind that our development process is highly iterative. Feedback is critical to us, and we use it to make changes.
There are several systems that are still undergoing changes meant to balance them and make them fun, and that will be a continuing process. Do we have an idea of where we’d like to be with a given system? Yes. Do we know exactly what changes to make to get there? We feel like testing (the feedback, telemetry, and other data we get from it) is an excellent way to help us make informed decisions and get feedback on a variety of changes so we can determine what works best. So, it isn’t that we don’t have a plan or any idea what the system should be, it’s more that we feel like testing is a critical component in making decisions about those systems and using feedback to tune them.
Sometimes we test big changes, sometimes just minor adjustments – whatever we think the system needs based on what we see in the test.
Okay iterative development. Generally speaking you plan what you’re going to develop, work on it, get feedback and improve it, get feedback and improve it to a point where it’s done. That’s the general overview of it, how it actually works for larger projects is you tend to break the larger project down to smaller portions and work on those. There are varying ways they could be doing this but that’s effectively what it entails.
The intent is to produce high quality software that could (potentially) be released at the end of each Dev cycle (if they’re using sprints) and you can easily push features out for future releases if they’re not ready yet because there is a lot more transparency to the overall project deliverables.
There has been some success with this in the software product market and it works well for development services. But… I’m really curious to see how it works for a MMO.
Really this could be the reason there isn’t a whole lot of ‘crunch’ material available yet because nothing is solid yet. This comment (regarding the “Item Modification system”) sort of emphasizes that:
It’s clear many of you are very interested in the Item Modification system (and understandably so!) and the various changes it has undergone. However, we’d like to remind you that it’s very difficult to get a good impression of an article’s meaning via Google translate – subtlety and context are often easily lost by automatic translations. Translating in general can be very difficult and can lead to confusion or misinterpretations.
It’s very important to keep in mind that we actually use our Game Testing Program to test the game. This means that sometimes we make radical changes to systems in order to gather the feedback we need. This testing allows us to find what works the best and is most fun for players. The Item Modification system is a system that has required some changes and testing as we feel out what players enjoy the most. In one build, items that can be modified may be rare. In another, they may be ubiquitous. Through testing, we’ll try several implementations, gather feedback, and make changes accordingly.
When Daniel spoke about Item Modifications, it is likely that what he spoke about reflected the implementation that was currently in testing – not necessarily the final intended state of the system (and we say “likely” because after talking with Daniel, he doesn’t have an exact recollection of what he said; he does do a LOT of interviews!). We haven’t delved deeply into discussing Item Modifications officially, and one reason for that is exactly what you’re seeing here: talking about things while they’re still in testing (and thus in flux) can be confusing to the community, so we do avoid it when we can while still trying to share information about the game.
We know you’re all interested in the specifics of how the system works, but please do keep in mind that quite a few things in the game are still subject to change as we continue with the test. Don’t panic! Though we’re close to launch, we aren’t done testing!
Which still scares the crap out of me with two months left to go – most of the foundation systems should be complete and solid enough to talk about. Things like character creation for one – I’m not seeing much on it though.
Another part of the quote above also scares the crap out of me is the bold part. I really hope they’re not relying solely on beta testers to provide them with this information. I understand a big part of iterative development is getting customer feedback, but the emphasis on “test” in that first sentence scares me. Most of the people signing up for beta have no idea how to test anything and more still just want early and free access to the game for a time to determine whether they want to buy it or not.
I played a little, probably for about one hour last night after the kids were in bed. I spent the whole time on one map which was quite tricky.
You have four platforms that are tied together sort of like a trident. At the tip of each spear point was the alien spawn area and at the bottom of the shaft *giggle* is the core that you have to protect from the aliens. Sort of like this:
S S S X X X ------- X C
The lines being a wall between the platforms which had tunnels and a blocking point so you could prevent all three from flooding through. The platforms are all connected with ramps that do not have building points on them.
I ended up creating blocking points to flush the aliens through the right side. The right side spawn had a little maze to get through to get to the middle which then had a short maze to get to the left which had a winding maze through most of the left side with a winding way back through the middle, to the right and then finally through to the back where there was another maze.
What did me in was a flier wave with lots of spores and darting bugs. I had adjusted the front three mazes to use Violators (drones that shoot air and ground targets) instead of mostly Anti-Aircraft seeking missile launchers which explode and do damage to multiple targets which would have made short work of the drones.
I was also upgrading a lot of ground targeting defenses and put down damage amps on the ground tiles of the maze which I don’t think impacts the fliers at all.
Ah well, I know what to change and I’ll give it another go.
Looking ahead, I think I have two more maps left to do, maybe three. That doesn’t seem all that long of a game but if you go back and play through it again doing things differently, it could extend the life of the game some.
One worry I have about not having a MMO to play is that I’ll rapidly consume single player games and my costs for video games will increase dramatically.
I’m hard pressed to find any significant changes to Lego Universe short of having to log into the patcher, patch, click play, then log in again…
Hai Lego Universe. One authentication please?
The few changes I have noticed seem to be around some of the puzzles – so far for taming pets. The LBO tamed a turtle which involved picking bricks to build a longcase clock and I think some of the other pets had different puzzles too. I do think they increased the drop rate of ‘class’ tokens, but I’m not sure since I haven’t actually played it – both the boys have been playing it quite a bit though so I’m mostly looking over their shoulder from time to time.
Oh, one difference is that they made subscribing a little easier. In fact, you’re almost better off creating a new account. Previously you had to create a parent account, tie yourself to the child accounts (after you create those too) then manage each account separately and buy game time for it. With a new account you just click one of the plentiful “Upgrade to Member!” buttons and you can create a Lego ID and skip creating a parent account (the option is still available if you’re so inclined). You can still pick between three subscriptions (one month, six months and a year) and you can turn on the recurring billing (which I didn’t). I admit this is much easier than it was previously
The LLO is having an easier time with it now but I don’t believe that’s due to any changes in the game. I think he was a little too inexperienced with PC gaming when I originally picked it up and since then he’s played numerous first person perspective games (Minecraft and Roblox mainly) on the computer so his manual dexterity is better. He’s also reading the quest text to determine what he needs to do which is nice. He’s already created a second character and is piddling around with that one too. He’s always had issues committing to one character, but that is pretty typical of young kids.
LBO is rolling through like a boss. Since it hasn’t changed all that much he’s rolling through the areas he has already done and seems to remember where everything was. As I said previously, it seems the class tokens drop more frequently so it’s not a huge farm fest and he’s working one his second level set for the Paradox dudes. He’s mostly focused on his one character, but will eventually create some more just to cover off each of the four classes available.
I did not create an account for myself or re-enable the previous one and I don’t really intend to though LBO is already asking if I’m going to swipe a character slot on LLO’s account so I can play with him. I might do that – though I might have to swipe one of the LBO’s slots so I can play with the LLO too.
You can find my previous comments about Lego Universe here.
I was trolling around on the SW:TOR boards and came across this thread where a poster asks:
bioware, will beta participants receive an exclusive reward in live for their beta participation?
He’s getting laughed at in the thread but I think it’s a decent question or rather raises a question of, “Who benefits most from beta testing?”
As a professional tester I’m awarded yearly by a rather large salary. Now the quality of my work isn’t what you’d get from the typical pimple-faced teen or unwashed twenty living at home or even smart people that work in other areas so it’s fair to say beta testers shouldn’t be making big money. You’re going to get a lot of feelings, obvious issues and sometimes impossible to reproduce issues because the beta tester isn’t sure what they did – why? They weren’t following a plan. They don’t have the experience to look into other causes. They might not have debugging skills. They might not even be all that familiar with computers.
The one benefit they get is that they can try out the software before it’s actually ready for release. Sometimes that’s not a benefit if you’re stuck with a lot of issues.
On the other hand, beta testing for MMOs (on the MMO side of things) is much like crowd sourcing the testing without really needing to pay much for it. It’s a massive benefit to the MMO creators because they get a lot of different hands using different system specs (can you imagine trying to create a lab with all the different system specs customers might be using?) and with varying experiences so a variable amount of quality in feedback which someone needs to sift through.
So who really benefits?
I’m leaning towards the MMO creator.
How hard is it to add a little something for the beta testers especially given the system for rewards is already in place for pre-orders?
Not that hard really. The main difficulty is that it should be something the testers value but doesn’t impact gameplay or people that didn’t have an opportunity to test.
Name in the credits?
Some in game title?
A color gem for their lightsaber or blaster shot?
Oh wait, I’m getting that third one for pre-ordering! It really was that drew me into pre-ordering – I was on the edge and almost went with not pre-ordering, but that color gem… I had to have it!