By Brandon Victor, Functionality QA Programme Manager
Not even the world’s brightest hardware manufacturers get their designs right first time around.
Testing and iteration are the keys to success and spotting failures before it’s too late is just part of the process.
If you’re considering hardware testing, you’ve probably noticed that there are tons of resources for software testing in the video games space but not as many for testing the hardware and peripherals that can come with it.
This is completely contrary with how easy it is to rapid prototype circuit boards and microcontrollers.
New consoles, controllers, accessories and technologies are released all the time, fueled by an insatiable appetite for immersive and interactive player experiences. Often this can result in a desire to expedite hardware production and this is completely understandable.
You’ve probably experienced this yourself.
Common misconceptions about hardware testing
The first thought for many manufacturers is to assign the hardware work to your software testing team at the same time. While that could work, you should be testing your hardware separately from your software.
Why is that?
Hardware testing, while it is still testing, works very differently from testing a game.
Not only is a dedicated hardware team thinking about how the device works but also, critically, how it fails. How it fails is completely different when compared to software.
Failures that impact hardware can be so small that they aren’t noticed until consumers start complaining about it after release. However, hardware testing – when planned for and done effectively – can catch these issues before it’s too late, saving you time and money.
Another item to consider is that in the way software testing has many variables, so does hardware testing. And hardware is a variable by itself.
When attempting to identify cause issues, being able to confirm that a software problem isn’t the result of the hardware interaction leaves fewer possibilities for the actual origin
It works both ways, of course.
If there is an issue with the hardware, it’s good to make sure it’s independent of any specific piece of software.
Essentially, compartmentalising the hardware and software allows both to become fault-free without adding any extra work in the respective testing processes.
Passing the ‘Dark Souls Test’
Of course, it’s not always necessary to test hardware separately. There are some cases where it’s perfectly acceptable. For example, the device could be unique.
The Skylanders Portal of Power is a piece of hardware that is essentially a RFID reader. There’s only so much one can do with a device like that as it’s required to run the title it came with.
Outside of those cases, though, it’s usually recommended to test hardware separately from the software.
By separating the testing, non-software dependent tests can be run, such as firmware updating, basic functionality, and statistical and manufacturing variance. Items such as power draw and connectivity can also factor in.
Then there is what I like to call the ‘Dark Souls Test’.
Someone, somewhere, will use your new piece of hardware to try beat Dark Souls. They’ve done it with guitar controllers, DJ controllers and even bongos.
If your hardware allows any sort of interaction with anything outside of the software it was intended for, someone will attempt to beat Dark Souls with it.
Of course, it’s not always possible but the takeaway is that just because you intended your hardware to only work with your title doesn’t mean that’s how the general public will be using it.
Once your new hardware is out of your hands, people will do things with it that aren’t intended. And just testing your hardware with its intended software won’t prepare you for any of those scenarios.
Hardware bugs are completely different when compared with software and use environment is a critical consideration.
What was connected to the device? What was it connected to? What does the RF interference look like?
Does it happen with just this one or with multiples? What were the various status LEDs doing? How do you capture the problem if you can’t use video capture software or take screenshots? How do you determine the frequency of an issue?
Unlike with software, hardware can’t be emailed across the world. There’s shipping and customs to think about.
Embracing common challenges
There are other items to focus on for hardware testing that software testing just doesn’t cover.
The number of hours needed to have confidence in the production version is not just down to a total, but also a total per device. Eight hours on 90 devices doesn’t verify hardware sufficiently like it could software.
Failure of hardware is inevitable, so it’s all about getting it under certain limits – i.e. an acceptable ‘Mean Time Between Incidents’ and ‘Dead on Arrival’ rate is key.
Multiple variations of hardware can be tested at the same time, with the differences being as small as varying wireless chips to determine the cheapest one that can be used without sacrificing quality of the product.
How the hardware interacts with other, already released hardware reduces the amount of firmware updates needed in the future.
Ease of updating and downgrading firmware and ensuring that doing so won’t render a piece of hardware useless can mitigate what could be a massive PR nightmare.
Looking for statistical variances ensures that each player’s experience is generally the same (and can be used to create baselines for future versions of the hardware!).
Accurate performance for player satisfaction
It’s great that we’re seeing huge growth in hardware devices in the gaming space but without proper testing, this can lead to a frustrated user base.
Ensuring the accuracy of hardware is a crucial aspect of immersive player experiences.
Having experienced external specialists to test your new hardware can give you an additional layer of analysis, data and feedback.
Keywords Functionality QA has a proven track record in developing comprehensive test plans and detailed test methodologies to help ensure your hardware devices perform to your vision.
Download our latest Hardware Testing solutions brief to learn more about how Keywords can help with your upcoming hardware project.