Automating Customer Tests

The way to interpret this table (let s ignore the first two rows for a moment) is to read each row as a separate test. The first two columns are input to the software, and the third column is the expected result, in this case. The first row verifies the equation (1000/10 = 100.0000). So, even without understanding how FIT does what it does, you can read and understand the tests. You also do not need to know how FIT runs the test. When FIT runs the script, you get the following result:

As part of the running of the script, the code is exercised and the verification is performed. In this case, all the tests pass, so all the cells in the quotient() column are shaded, indicating success. If there is a failure, the table cell is colored red and the actual value is present along with the expected value, which allows the customer to look through the tests at a glance and see the tests that pass and the ones that don t. (If this were a four- color book, you d see the green shade of a cell for success and red for failure in the tables on this page.) Also, because this is HTML you can add additional content that explains what is being tested around the tests (which can be text or graphics). This content turns these HTML documents into the form of an executable specification.

Connecting FIT to the Implementation

To be able to specify your tests in this manner, you need a bridge between the FIT framework and your software. In the previous table, the first row specifies a class, eg.Division , which is used as the bridge between FIT and the actual implementation. The second row specifies methods that are called on the eg.Division class that FIT calls during the execution of the test. In the next section, we describe in detail how you can take the manual script from the previous section and convert it into an automated script run by FIT using our Web service.

Automation with FIT

The first step in the script is to verify that we can retrieve a recording given its id. Following is the translation of the manual step into the format of the table to be used in FIT.

This test uses a class contained in the FIT framework named ActionFixture , which is a simple controller that parses each row and passes the values off to the appropriate handler. The header of the table defines the controller used to run the test. Each subsequent row in the table defines a step in the script. This test script uses the start , enter , and check commands that are part of the ActionFixture .

Start Command

The start step has a special meaning: it is used to initialize a class that is used to adapt the application classes to run inside the FIT framework. In this example, the name of this special class is CustomerTests.CatalogAdapter , which has to be written to support running the FIT tests and should not be considered a part of the application itself. Because it is strictly for FIT integration, it should not contain any application-specific logic beyond that needed to test.

Enter Action

The next row in the table begins with the enter action . This step is a simple action to be taken against the application via the adapter. In this case, the script specifies that the application must call the FindByRecordingId method on the CatalogAdapter to retrieve a recording with the recording id equal to 4. This method will change the state of the CatalogAdapter so that the retrieved recording is available for further steps.

Check Action

The last row in the table begins with the check action , which is the verification of the correctness of the application s response to previous user input. There are two parts to each check operation:

• What to check This is the name of the method defined on the adapter to use for application response verification. In our example, we want to verify that the recording is correctly retrieved; our CatalogAdapter class defines a Found method that can be called to verify that the recording is successfully retrieved.

• What value the user expects to receive from the application In our example, we expect the Found method to return True to indicate that the recording with the given recording id is found.

CatalogAdapter

The test script shown previously specified a class named CatalogAdapter . Its job is to adapt the calls that FIT will make into the actual calls to the Web service.

namespace CustomerTests
{
   public class CatalogAdapter : Fixture 
   {
      private CatalogGateway gateway = new CatalogGateway();
      private static RecordingDto recording;

      public void FindByRecordingId(long id) 
      {
         recording = gateway.FindByRecordingId(id);
      }

      public bool Found()
      {
         return recording != null; 
      }
   }
}

This class seems to fit the requirement in that it simply adapts our existing software to the FIT framework. Notice that the names of the methods have to match the names specified in the test script. Otherwise, the test results will show exceptions for these rows.

Running the Script

We use the FileRunner class in FIT to run the test script. Figure 7-2 demonstrates the sequence of messages exchanged between FIT, the CatalogAdapter , and the Web service during the execution of the test script:

Figure 7-2: Message exchange

Running this test yields the following result:

In addition, FIT provides a test summary section that gives statistics related to the execution of the test script, as you see at the top of the next page:

Continuing with the Automation

Now that we can retrieve recordings from the database using FIT scripts, we need to augment the script to also check the content of the recording.

This script is very much like the previous one. To get this script to work, we have added the following methods to the CatalogAdapter : ArtistName , ReleaseDate , LabelName , and Duration . These methods simply return the appropriate value from the RecordingDto object that is retrieved by the enter action . The modifications result in the following changes to the CatalogAdapter :

namespace CustomerTests
{
   public class CatalogAdapter : Fixture 
   {
      private CatalogGateway gateway = new CatalogGateway();
      private static RecordingDto recording;

      public void FindByRecordingId(long id) 
      {
         recording = gateway.FindByRecordingId(id);
      }

      public bool Found()
      {
         return recording != null; 
      }

      public string Title()
      {
         return recording.title;
      }

      public string ArtistName()
      {
         return recording.artistName; 
      }
      
      public string ReleaseDate()
      {
         return recording.releaseDate; 
      }
      
      public string LabelName()
      {
         return recording.labelName;
      }

      public string Duration()
      {
         return recording.totalRunTime.ToString();
      }
   }
}

The adapter maintains its rather mundane existence. The only thing of interest or concern is that the type of the Duration implies a data conversion from an integer to a string. This situation seems odd, and in fact when we run the test we get the following result:

There is a failure. The Duration specified in the test script was 72:51, but the number we get back from the adapter is 4371. This is clearly a problem that will need to be addressed. For now, though, let s continue with the rest of the conversion process, and we promise to come back later and correct this problem. There is no way we can forget because every time we run the test it fails.

Verifying Track Information

The next part in the manual script verifies track information. It turns out that FIT has a different type of fixture, a RowFixture, that works best for this type of test. From the FIT website: A RowFixture compares rows in the test data to objects in the system under test. Methods are invoked on the objects and returned values compared to those in the table. An algorithm matches rows with objects based on one or more keys. Objects may be missing or in surplus and are so noted. This type of fixture fits very well with looking for track information associated with a particular recording. The Track tests from the manual test script expressed in a RowFixture looks like this:

This takes a bit of explanation. We need to implement another adapter that inherits from RowFixture to retrieve track information from the recording. Unfortunately, this script does not stand alone; it requires that a recording was retrieved in a previous step. In this case, the script will be run after the previous one. The TrackDisplay class is shown as follows :

public class TrackDisplay : RowFixture 
   {
      protected override Type getTargetClass() 
      {
         return typeof(CustomerTests.TrackDisplayAdapter);
      }

      public override object[] query() 
      {
         TrackDto[] dtoTracks = CatalogAdapter.Tracks();
         
         TrackDisplayAdapter[] adapters = 
            new TrackDisplayAdapter[dtoTracks.Length];

         for(int index = 0; index < dtoTracks.Length; index++)
         {
            adapters[index] = 
               new TrackDisplayAdapter(dtoTracks[index]);
         }

         return adapters;
      }
   }

This class returns an array of objects. The query method takes no parameters, but must somehow return the track information associated with a recording. In this case, it makes a call to the CatalogAdapter (which retrieved the recording in the previous test) to get the tracks associated with the recording. The query method also converts the TrackDto objects into TrackDisplayAdapter objects just like the CatalogAdapter adapts the RecordingDto .

public class TrackDisplayAdapter
   {
      private TrackDto dto;
      
      public TrackDisplayAdapter(TrackDto trackDto)
      {
         dto = trackDto;
      }

      public string Title()
      {
         return dto.title;
      }

      public string Duration()
      {
         return dto.duration.ToString();
      }

      public string GenreName()
      {
         return dto.genreName;
      }

      public string ArtistName()
      {
         return dto.artistName;
      }
   }

When we run the tests, we get the following result:

All the duration checks failed, which seems similar to the previous test (in which the check of the duration of the recording failed). Just as we did in the previous test, we make a note of it and we will return to it as soon as the script is fully automated.

Verifying Review Information

The review information is similar to the track information. In fact, we did create a new RowFixture called ReviewDisplay and an associated adapter named ReviewDisplayAdapter . (Their implementations are left as an exercise for the reader.) The problem the test exposes is related to the variability of the data. In the previous test, the Recording and Track information does not change, which is not the case with the review information ”which is updated by reviewers all the time. Clearly, we do not want to write customer tests that need to be updated at random times. So what should we do? One option is to create a recording in the database as part of the script and subsequently remove it after the test is complete. Another option is to use a separate database that is a snapshot of the production database but not constantly modified by the users. The last option is to create a fake recording in the production database. This recording has well-known content and can t be changed by the users.

We choose to insert a well-known record in the database. It is the simplest alternative at the moment, and it allows us to use the actual deployment environment to run our tests. This solution is not immune from risk, however; someone can delete or change the fake recording in some way. And even though we believe the id will never be assigned, it is possible that someday it could overlap with a real recording, which would make our tests fail. There does not seem to be a perfect answer to this problem. If we choose a separate database, we have to employ some process to ensure that the schemas would stay synchronized, and so on. Given our solution, the script for verifying review information is as follows:

When we run the tests for reviews, they all pass.

Invalid ID Script

The last part of the script that needs to be automated is the one that attempts to retrieve a recording that is not present in the database. Just as in the review script, the id chosen does not exist in the database and should never be assigned to a valid recording. If this assumption remains true, our test never has to change. The following is the test script:

This script does not require any changes to the CatalogAdapter because it uses existing methods. When we run the test, it passes as well.