Evaluating Regression Tests

After you successfully execute a regression test, you can analyze the results. To analyze a regression test, it is necessary to know how PostgreSQL finds out whether a test has been successful or not: The result of the SQL statements are stored in files and can be compared with the "correct results" distributed with the test suite. The results are compared by using a simple diff command.

Several things can cause a test to fail; you learn about these in the following sections.

Error Messages

Some SQL commands cause the system to raise an error. This behavior is expected, because some SQL commands in the test suite contain errors.

Not all error messages are raised by PostgreSQL, but come instead from the underlying software. The result of a test suite and the expected results are compared with a simple diff command. If the two are not 100% equal, the regression test for a specific component fails. In most cases, this does not mean that your system is not able to run PostgreSQL.

Local Differences

The tests should be run in plain C locale. If you are running the tests against a completed installation, this might lead to problems if something different than C locale is used. If you run the test against a temporary installation, this won't be a problem, because the test suite makes sure that the correct settings are used.

Date and Time Differences

The expected results have been generated for time zone PST8PDT (Berkeley, California). Some regression tests are very time zone “dependent, so the results might vary depending on the time zone your system is in.

To set the time zone to the correct value for performing the regression test, you can use this command:

 [hs@athlon regress]$  export PGTZ='PST8PDT7,M04.01.0,M10.05.03';  

Sometimes the regression test for timestamp can also fail because of underlying libraries.

Floating-Point Differences

Comparing floating-point numbers is always a crucial thing. Depending on the type of microprocessor you have got, the result of a floating-point operation might vary for many reasons.

On one hand, differences can occur because of the way the floating-point unit of your processor works. On the other hand, some differences might occur because of compiler settings, which influence the way floating-point operations are treated. Most differences occur because of differences in the way numbers are rounded.

On AIX-based systems, for instance, it is possible to tell the system how to round floating numbers (check out the RS/6000 docs and the docs of XLC).

In most cases, differences emerging by the rounding algorithm can be found only with the human eye, because the "errors" in the calculations are comparatively small.

Polygon Differences

Geometric datatypes often contain non-integer components . This leads to the same problems discussed in the previous section about floating-point numbers. The errors are comparatively small, so take a closer look at the results when the regression test on your system fails.

Ordering Differences

It might appear that the same data is returned in different order than expected. If no ORDER BY clause is used, this can occur easily. Most regression tests do not use ORDER BY clauses, so failures might happen.

On some platforms, though, the way data is ordered might vary. Some differences might occur because of a non “C locale setting; other differences might happen because of differences in the underlying function.

In most cases, the results will be the same on all machines, but you have to keep in mind that there might be slight differences. Generally you won't have to worry about the order of your records, because these differences usually won't be significant.

"Random" Testing

In most cases, generating random values is not random. In many real-world applications, deterministic random generators are needed. This might sound strange , but imagine an algorithm that produces huge amounts of data. If the program is used on multiple machines (for scientific purposes, for example), the result of an operation is the same on every machine. Therefore, random generators are, in most cases, deterministic (see man pages of lrand48).

At least one case in the random script is intended to produce random results, and this can cause the regression test to fail because a stochastic value differs from the expected result. If the regression test for "random" never fails even when you execute it many times, you might have to worry.