Cost Analysis

Your company is going to want to undertake its own cost analysis before embarking on the introduction of any significant automated testing. Too often managers will see the introduction of automated testing as some kind of "silver bullet" that should cure all their cost overruns and late product releases. While significant success stories are starting to appear in the game industry following various companies' deployment of automated test systems, there are far more tales of woe where companies tried to automate and got badly burned. The reality is that there are many costs to consider before taking the step to automate.

Setting clear expectations to upper management is crucial in presenting the concept of test automation. Clearly the need is to keep clear in your mind where you feel test automation will definitely impact your company positively and where it may have either a minimal or insufficient impact. And then when presenting to management, be sure to make clear what the realistic benefits of automation are likely to be and when those benefits are likely to be realized. If introducing large-scale test scripts is not likely to significantly benefit the current title in development, or even the ensuing two, but is expected to start showing benefits from the third project onward, then be sure to present this picture clearly rather than promote the frequent misconception that automation will yield immediate and tangible financial benefits. Here are some of the factors your management team will need to consider:

• No hands-on testing. It's easy to factor in how much an additional human tester will cost and what his/her likely value will be, but how do you factor computer control and testing?

• Assumed increase in productivity. The mere fact you have decided to automate will give the expectation of increased efficiency and lower costs.

• Reduction of perceived productivity. Ironically, your department will seem less productive due to the time it takes to implement and use automatic tools.

• Deeper levels of test issues. Because automation brings new opportunities to test, it will often introduce deeper levels of testing, which can be a good or bad thing.

• Fundamental changes in how software is developed. The ramifications of introducing automated testing can reach company-wide.

• Test cycles performed. Because automatic testing runs so much faster than human testing, it can lead to greater churning, which in turn may encourage productivity or lead to laziness .

• Improved professionalism. Automating testing can often motivate the team if it is done well and lead to improved morale and professionalism .

In computing the likely ROI on the company's proposed investment in automating its testing program, you have to consider many more variables than management is expecting to have to look at. You'll also have to argue that all the benefit ultimately gained will not be realized for some time, and this is in itself going to be a hard sell. Not least because you need to be clear the extent to which automating the testing will impact company-wide operations and change the way that the development team codes its games, the way game engines are written, and so forth. However, you have some convincing weapons in your arsenal to persuade that automating at least part of your testing program will make a lot of sense: If you produce massively multiplayer online games, for instance, your "pitch" can be substantially assisted by the well-publicized reports that automation has greatly benefited many developers when it comes to regression and load/stress tests on massive online environments. But even if online MMP games are not your company's main output, you will still be on solid logical ground arguing that highly repetitive testing is well suited to being replaced with automated testing. You will also be on firm ground arguing that to some degree, at least, automated "sniff test" testing should be introduced, which ensures that new code does not contain obvious flaws before a commit is permitted.

Some of the predictable fixed costs when converting to automation are as follows :

• Additional hardware (or at the least upgrades to existing hardware)

• Middleware and tool licenses (you are likely to need at least some)

• Scripting tool creation

• New game code to add hooks, framework, and so on for automation

• Tool training

• Management software and support

• Ramp-up costs

• Additional team members who are devoted to automation

You will face some variable costs, too:

• Test case implementation

• Test case designs specific to automation

• Maintenance

• Results analysis

• Defect reporting

• Night-time system runtime

It should be straightforward to calculate the costs of these factors. They are based on well-known variables and cost bases that already exist in some (at least similar) form within your company, either in relation to existing manual testing or to game code development.