13th Aug 2021

How to Plan Submission Testing for Your Upcoming Game Release

Michael Holdaway, Keywords Functionality QA Manager

By Michael Holdaway, Certification Manager, Keywords Functionality QA

Certification, compliance or submission testing.

If you are in the video game industry, you probably have heard one of these terms passed around – maybe with a hint of concern or stress from the person speaking it.

Picture this scenario: You have been working on your game for months, or even years, and it’s getting closer to your release date.

The stress starts to build; everyone starts wondering if you will release on time or how much crunch time will be needed.

Now, with only a few short months to go, you need to get ready for something called “submission” and it’s the last critical milestone before the game can be released.

But, for those of you who are less familiar with the subject, you might be thinking ‘What is submission?’, ‘What can I do to prepare?’ and ‘Does submission really need to be stressful and scary?’.

To dig into those questions, let’s start from the beginning … welcome to the world of certification, compliance or submission testing.

All of these words usually come to a similar meaning and that is to start getting ready for the final test of your game. If you are releasing a game on a console, but not only consoles, then you will need to know what the submission or certification process is and why it exists.

Understanding submission testing

So, what is submission?

Video game submission is the process that every game creator, developer or publisher needs to go through in order to successfully release their game on the hardware they are targeting.

In almost every case, submission means the hardware manufacturer will need to test your game against requirements that they have created and your game needs to meet all of these requirements in order to pass.

Each of the major consoles have their own version of submission criteria and processes that need to be followed. You will not be able to have your game released on any console until the game passes submission.

But where does the term ‘submission’ come from?

Every developer has to ‘submit’ their game to the first-party console manufacturer such as Sony, Microsoft, Nintendo, Oculus, Android and iOS. Each first-party manufacturer ensures these games meet their own requirements before being released to the market.

If your game does not meet their requirements, it won’t be cleared for release.

It’s that simple.

Asking the right questions

Why does submission exist? Why do manufacturers make us go through this grueling and stressful process? Why can we not just release our games on to their platforms without them needing their approval?

To answer these questions, we need to go back to 1983.

Many industries or markets have had a ‘crash’ at some point. For video games, that was the crash of 1983 to 1985.

The video game crash by itself deserves its own article but the short story is that video games were coming out with such bad quality that consumers were asking for refunds in large numbers.

This is partly due to the fact that there were no established quality standards for these games. As a result, video games manufacturers lost revenue, some game console manufacturers went bankrupt and consumers became hesitant of buying video games.

In 1985, one prominent newcomer to the market had learnt from these failings and made it so that only quality video games would be released for their console. Because of the crash, the certification process as we know it was created and has been expanded upon ever since.

Submission exists for a few good reasons, including:

  1. Consumer protection
  2. Brand protection (quality)
  3. Protection of the hardware

The submission requirements created by hardware manufactures centre primarily around these three topics.

As players, if we experience crashes or hangs, we sometimes instinctively blame the console we are playing on, potentially tarnishing the brand in our mind.

That’s why video games should meet a certain quality and the consumer experience should be similar across all games on the platform.

Also, we would not want games to have the ability to format our hard drives or damage the hardware we paid large amounts of money to own.

Now that we have an idea of what submission is and why it exists, let us ask ourselves again ‘does submission testing need to be stressful and scary?’.

The answer I want you to take away from this article is ‘no’, but with some caveats.

In order to avoid the stress and scare, the submission process for your game needs to be approached with a carefully thought-out plan that is executed on.

The way to manage this risk is to prepare. Keep submission testing in mind between different project development cycles and plan accordingly.

Manage the risk by giving yourself time to properly make use of specialised certification testing – quality assurance (QA) testing that checks all the requirements against your game.

To paraphrase one of the manufacturer’s submission documents: “The certification requirements should be consulted before development begins, during development, at development milestones and leading up to release.”

How many pages of documentation are there for the certification requirements?

That depends on a number of factors, including:

  • How many major features your game has
  • How many consoles you are targeting
  • How many system software features your game uses

The way you answer these questions can mean the difference between 100 pages or more than 1,000 pages of documentation of certification requirements alone.

That’s on top of all the development guides you already need to read to implement code into your game.

Having the expertise for submission success

This is where having certification testing expertise on your side can help make this a smoother ride and avoid the potholes along the way.

From planning to release, everyone involved should be aware of the different certification or compliance requirements that impact them – teams such as art, audio, localization, monetization, programing and, of course, QA.

This is because any one of those teams can potentially impact the risk of not passing your submission. As a QA professional, this is one of the most common misunderstandings I encounter with different games and developers.

Certification is not just a QA issue and the release process for a game is much smoother and less risky when everyone is onboard the same train from the beginning.

Plan for the worst, aim for the best

Let’s talk about a way to prepare for submission. Starting from one of the most important dates – the release date of your video game – we are going to work backwards to see when Certification QA should be involved.

From the release date, move back two months for a physical released game or one month for a digital only released game. This is your ‘must-have’ submission approval deadline.

It takes time to print and distribute hard copies of a game and also takes time to set up all the digital store fronts after passing submission.

Once you have established your approval deadline, move back one more month. This is when you should be planning to have your first submission. However, potentially that might not be enough time. When you plan, always have a contingency.

It’s standard practice to plan for a second submission, traditionally two weeks after the first.

This leaves you with enough time to go through the process, make fixes to issues found by first party manufacturers, submit again and get your results.

As much as QA vendors or publishing certification teams would love to say they can pass first submission 100% of the time, the truth is you should not be dependent on that. Numbers provided by the hardware manufactures place submission approvals in the range of two to three submission attempts.

That being said, it’s possible to pass on your first attempt if you prepare and, if you do, then you have ample time to work on patches, day-one or post-release DLC content.

Who doesn’t love to be ahead in their development schedule?

Now, let’s continue to go back in time along the development life-cycle of your product.

From your submission date, let’s go back two more months. This is the absolute latest stage that QA should start being involved in testing your game against the certification requirements.

Throughout these two months, there is enough time for multiple (three or more) rounds of testing.

Allow one week of testing for each round, with one week of development time to fix issues between rounds. This also leaves a final optional round open if the game requires additional fixes for certification issues just before submission.

Going back further, if you have the ability to do so, have QA do a round of testing after reaching beta and alpha milestones (or content lock and feature lock if you use those terms instead).

These rounds help you make sure that, as you near these earlier milestones, your game’s features already address upcoming certification requirements and challenges from the get-go.

Starting this early gives you assurances that time spent developing a problematic feature doesn’t go to waste and, instead, everything points to go.

Failure to do this can mean having to spend hours or days on programming a feature only to find out later the whole thing needs to change because it does not comply with a certification requirement.

Finally, going back even further, work with your QA partner at the planning phase on the features you want your game to have. In this case, a partner would inform and guide you to understand what requirements to keep in mind for each feature and help keep development on the right path.

Early planning for video game submission testing helps ensure your title does not need major changes with only two months to release. This way your game is not in jeopardy because of a feature or mechanic that breaks critical first-party requirements for submission.

How can we help you succeed?

At Keywords Studios, we also go a bit further than just planning for submission. We have dedicated teams that live and breathe certification testing.

Starting near the top, you have a Certification Manager who works as project management to ensure all teams have the resources they need and that any clients can comfortably work with us in certification.

At Keywords Studios in Montreal, we also have a Certification Lead for each of the main hardware manufactures. They are dedicated to learning and keeping up to date with anything to do with that manufacturer.

These Certification Leads work as a single point of escalation and knowledge, helping to ensure our decisions are as educated and consistent as possible. They are also important for making sure all their knowledge is distributed amongst all the testers they work with.

New consoles just launched … what now?

Luckily, this does not change much for your planning. Unlike the previous generation jump, the submission process has remained mostly the same for Sony and Microsoft.

There are some small changes to a requirement or a small feature that exists on one console but not the other. However, the timing from testing to release has stayed mostly the same.

You may want to take into consideration that you are now developing for two additional platforms if you plan to release across both generations. This means your experiences are still very much valid for the new generation – and so are ours.

We are always prepared

At Keywords Studios, we place a large emphasis on training, coaching and mentoring our testers so that they have full knowledge and comprehension of the certification requirements. Teams are encouraged to discuss possible requirement failures with each other as they find them.

We make sure everyone who joins our team can work well in a collaborative environment. At the end of the day, it is not the number of issues one single individual tester has found but how successful the team worked together to pass submission.

Our approach is based on almost two decades of experience, with close to 100 individually, homegrown experts currently working on a never-ending cycle of incoming certification work.

Since 2018, we have worked on more than 500 submission types across all platforms.

Preparation and partnerships are key for success

As you can see, preparing for game submission does not need to be scary or stressful. There will be a lot of work but if we make sure to plan together for this critical gate then it’s possible to plan for the worst and achieve the best!

At Keywords Studios, our leads can help your teams understand the requirements – whether they are a tester, manager, artist or programmer.

The process can carry much less risk if everyone works together, and if you work with an experienced partner who can guide you along the way.

Discover more about how Keywords Functionality QA services can deliver proven, quality results for your game development project.

Share This