Can dev teams make better use of player data?

Across three roundtables at GDC 2024, together with insight leaders from across the industry, I explored how to make player data work for teams through every step of the development lifecycle. Over the course of three hour-long sessions, I had the pleasure of talking to researchers, data scientists and insight directors from across the games industry about the challenges they face when making use of a player feedback, market data and user research to inform ongoing development.

Here are a few of the most potent challenges that were raised and discussed:

How can we ‘humanise the data’ we produce?

A key job for Games User Researchers is transforming raw data to be useful to our stakeholders. Sure, we want to be transparent and forthcoming with everything we find in the process of research, but we must remember that the teams we’re supporting aren’t Researchers or Data Scientists themselves and have a mountain of other priorities to balance. This means it is crucially important for Researchers to not only interpret raw data and draw valuable insight from it, but to re-tell those findings in a concise, empathetic, and digestible way; in a way that makes findings relevant and actionable to busy developer audiences. 

Let’s take a typical playtest of perhaps between 30 and 60 members of the public. We might invite them to play through two-to-five hours of unfinished content. By the end of that test, you have hundreds or maybe even thousands of data points, survey responses, and observations, both quantitative and qualitative. These are steeped in personality and emotion straight from playtesters themselves. Within that data will be some fundamentally transformative feedback, but there will also be reams of noise. How can we summarise that data without removing the stories and nuance that inspires teams?

In my experience, creative people are inherently empathetic and think fundamentally in terms of the emotional experience that a game creates, so presenting playtest findings back to them in the same language tends to accelerate understanding and ultimately manifest positive change in the game design. Of course, there is a balance to strike in not editorialising or manipulating this raw data deceptively and instead staying true to our commitment to rigour. 

How to deliver bad news, without breaking your teams’ hearts?

It is an outcome of playtesting that bad news will arrive with the good: things players didn’t love, didn’t land, or didn’t grok. Players tell you things you don’t want to hear. Maybe a character that was meant to be charming comes off as irritating, maybe a game mechanic that was meant to provide a satisfying challenge presents to the player as infuriatingly convoluted, maybe the UI just isn’t as easy to navigate as you designed it to be. 

It's important as Games User Researchers that we tell the truth. Anything less than that would be a disservice to the team and the product we’re trying to improve. But that doesn’t mean it’s easy. Sometimes the truth hurts. Sometimes it’s necessary to break the team’s hearts. 

For the end-product though, this feedback is essential. And the truth is, it’s this feedback that can often galvanise the most profound improvements to the game. So, the challenge becomes how do insight teams deliver bad news in a maximally constructive way, with minimal impact on team morale?

In our experience there’s two major factors that make communicating this feedback easier. The first we’ve talked about – it’s humanising the data. But the second is cultural. When our partners embrace a culture of testing, difficult feedback shifts from something to commiserate to a reason to celebrate. By redesigning that UI now, we’re avoiding a buildup of frustration — and we can prove it. By dialing up feedback in-game, we’re avoiding players missing a ‘set piece’, and their feedback being tainted. And this is a studio culture shift to recognising that feeling crestfallen is inevitable and essential. Supercell’s ‘game cancellation celebrations’ — where sunseted games are thrown a ‘going away’ party — is a perfect example of this culture in action. 

Games User Researchers, in pursuit of helping deliver the best experience to the player, are constantly trying to balance artistry with raw business reality. It’s a tightrope that must be walked with compassion and confidence.

At my three roundtables during GDC this year, we will dive much deeper into these and other critical topics chosen by you! If you haven’t signed up yet, you can find the full list of sessions here and if you’re unable to join or want to learn more about Player Research, reach out below.