Results

Reduced Test Execution Time

We have been able to run test suites of up to 260 tests in under a minute when running against an in-memory database. Those same tests run orders of magnitude slower when running against a relational database. On a project we are currently involved with, 53 tests are executing in memory in a time of around ten seconds, while the same 53 tests running against an Oracle database have an execution time of about ten minutes. This is not surprising given the relative cost of memory access (measured in nanoseconds), compared with the cost of disk access (milliseconds.)

Reduced Testing Code

By continually improving our testing infrastructure, we have reduced the average size of our tests by 60%. We estimate that this translates into an effort reduction of 70%. We also suspect that the quality of testing has improved, but this is hard to measure directly.