Conformance Testing

Team-Fly    

SyncML®: Synchronizing and Managing Your Mobile Data
By Uwe Hansmann, Riku Mettälä, Apratim Purakayastha, Peter Thompson, Phillipe Kahn
Table of Contents
Chapter 13.  Interoperability Verification


The first step to obtain a SyncML Logo is to successfully complete the Conformance Testing process. It ensures that the Client or Server conforms to the appropriate SyncML DTD. For Device Management applications, the conformance process ensures that the Client or Server conforms to SyncML DM. In addition it ensures conformance to the supporting DTDs, such as the Device Information DTD [SDI02] and the Meta Information DTD [SMI02], as well as the supported transport binding specifications. The Conformance Testing also covers the Protocol either the SyncML Synchronization Protocol [SSP02] or the SyncML Device Management Protocol [SDM02].

Each manufacturer can apply to use the term "SyncML Conformant" by going through a self-certification exercise and submitting the documents to the SIC. The SIC reviews the documents and lets the submitter know if his implementation is SyncML Conformant. The manufacturer is then free to use the term "SyncML Conformant" in relation to that one product and should continue with the Interoperability Testing process. The only valid reason not to continue is that there are no other devices to test against. In this case it is not possible to pass Interoperability Testing, because there are no other devices to test with.

SIC publishes the submitted documents on the SyncML Web site, but only after the device passes Conformance Testing.

This self-certification process consists of:

  • Completing and submitting the SyncML Implementation Conformance Statement Proforma [SICSP02]

  • Testing the implementation with the SyncML Conformance Test Suite

  • Executing the manual test cases as described in the SyncML Manual Test Cases Document [SMTC02]

  • Submitting all test results to SIC for review

An implementation that has successfully passed SyncML Conformance Testing, as shown in Figure 13-2, can use the term "SyncML conformant", but not the SyncML Logo. The SyncML Logo is reserved for products that successfully pass the SyncML Interoperability Testing process.

Figure 13-2. Conformance Testing process

graphics/13fig02.gif

SyncML Implementation Conformance Statement (SICS)

A SICS proforma is available for download on the SyncML Web site. The proforma is a template that can be used by a manufacturer to check if an implementation is conformant. The completed SICS needs to be submitted to the SyncML Interoperability Committee. First, the manufacturer has to fill out the device information: Is the implementation a SyncML Server or Client, which are the supported content formats, and which are the transport bindings used (HTTP, WSP, or OBEX)? Contact information is also needed. Depending on whether the product is a Client or Server, the appropriate sections in the SICS need to be completed.

The SICS lists all the commands of the SyncML DTDs (both DS and DM), the supporting DTDs, the possible protocol element types, and the allowed MIME headers. Each feature is indicated as mandatory to support (MUST) or optional (MAY, SHOULD).

To be SyncML conformant, all MUST features in the appropriate sections have to be implemented. Even if only one MUST feature is not implemented, the SIC has to return the application to the submitter and notify them that their implementation is not conformant. From a conformance process point of view, the manufacturer is free to support or not support a feature that is marked with SHOULD or MAY. The SyncML Initiative strongly encourages manufacturers to implement the SHOULD features, whereas MAY features really are optional.

SyncML Conformance Test Suite (SCTS)

Based on the fact that interoperability is key for the success of SyncML in the market and with customers, the Initiative decided in the Fall of 2000 to develop a SyncML Conformance Test Suite (SCTS). It serves two purposes: First, it helps the developers of SyncML software to verify their implementations; second the SCTS helps during the SyncML Conformance Testing to easily verify if an implementation is conformant or not.

In November 2000, a team of software engineers started to work on the SCTS for the SyncML DTDs and protocols. First, they created a list of test cases that the Test Suite should execute, which was extensively reviewed by the SyncML Initiative before the actual implementation started. From February to September 2001, the tool went through extensive testing by all the companies that applied for SyncML conformance. Starting in November 2001, it was a requirement to pass the SCTS to get the SyncML Conformance label.

In September 2001, the Device Management Expert Group started to work on the test cases for a version of SCTS that tests conformance to the SyncML DM DTD and Protocol.

The development expenses for the SCTS are shared between the companies using it. A nonmember company has to pay the full license, Supporter-level members get a discount (equivalent to their membership fee), Promoter-level members get an even higher discount, and for Sponsor-level members, the license fee for SCTS is included in their membership fee. Currently, a SCTS license is bound to the major version of SyncML DS or SyncML DM that the purchased SCTS version tests. This license entitles company-wide usage, as well as updates. The up-to-date terms and conditions for SCTS are available at the SyncML Web site.

The SyncML Conformance Test Suite contains the following:

  • SyncML Conformance Test Tool

  • Test Cases

  • Online Help

  • List of changes

How to use the SCTS 2.x

SCTS version 2.x is a Windows® application. SCTS can act as a SyncML Server or as a SyncML Client. The SCTS contains the following six pages:

  • General

  • Objects

  • Test

  • Commands

  • Logs

  • Status

The first step is to configure SCTS to work with your Client or Server. SCTS comes with two sample configurations, one as a Client and one as a Server. These samples can also help one to configure the tool.

The best way is to start is with the General page (Figure 13-3) and select whether SCTS should act as a Client or a Server. Usually the default settings (HTTP, two-way sync, WBXML) can be used.

Figure 13-3. SyncML Conformance Test Tool General page

graphics/13fig03.jpg

SCTS supports communication over HTTP, as well as over OBEX. You will need to set the Address Configuration group. If SCTS is to act as a Client using HTTP, you will need to set the SCTS Address to the address of the local machine (e.g. localhost) plus some identifying string (in the example, SCTS). The Test Object Address is the full URI of the device to be tested. If a proxy is being used, then one must fill in the full URI of the proxy as well. If SCTS is to act as a Server using HTTP, only the SCTS Address needs to be filled in.

The entry fields in the product information group are added to the report that SCTS creates and are used by the SIC.

In case the SCTS is a Client, the next step is to enter the Server username and password on the Objects page (Figure 13-4) in the Source Details group. The username and password are used to access the SyncML Server.

Figure 13-4. SyncML Conformance Test Tool Objects page

graphics/13fig04.jpg

Using the Auto Configure function on the Test page (Figure 13-5), SCTS can exchange the Device Information with the Server and use the data received to configure itself. The Auto Configure function is an easy way to configure SCTS, but it is still possible to enter all the require data manually on the Objects page.

Figure 13-5. SyncML Conformance Test Tool Test page

graphics/13fig05.jpg

The Objects page now lists the different content types that the Server supports, in this case, contacts, events, and notes. Note that an object called "devinf10" is shown, which represents the device information document that was exchanged. By clicking on one of these data types, the SCTS displays the respective Source and Target URIs, and the MIME type used to exchange data (for example text/x-vcard or text/calendar). After executing the different test cases, the SCTS also displays the stored source and target anchor on this page.

To pass the SCTS, it is necessary to go through one Test Group at a time in the indicated Test Group order. Each Test Group requires the previous Test Group to be executed it is not possible to just execute Test Group 6, for example. This is because the SCTS test cases need to set up some data on the Client or Server. SCTS also needs to be in a defined state before a Test Group can be executed.

For more general testing purposes, it is also possible to execute a random test case and start it without executing the previous ones. In some cases, this requires the SCTS to initiate a slow sync with the other device.

As shown in Figure 13-5, the SCTS marks the Test Groups that have been successfully passed and stops at the Test Group that failed. To each Test Group, SCTS describes the state that the device under test and the tool need to be in. For example, in Test Group #5 the datastore needs to be empty and for Test Group #6 five items need to be added to the device's datastore. Table 13-1 lists the various functions tested.

Table 13-1. Functions tested by SCTS

Test Group

Tested behavior

#1

Authentication with no credentials.

#2

Authentication with wrong credentials.

#3

Authentication with both Basic and MD5.

#4

Authentication with correct credentials and exchange of device information.

#5

Alert with slow sync and normal sync.

#6

Alert with Add of 5 objects and 5 corresponding Maps.

#7

Replace, Delete, Replace without target.

#8

Slow sync to check consistency of databases.

#9

Init and Sync in the same package, as well as maps to be sent in the next session.

#10

Maps sent from previous session. Add, Delete, Replace, and Status with multiple items.

#11

Slow sync to check consistency of databases.

#12

Slow sync with data only coming from the SCTS.

#13

Support of multiple messages.

The Command page (Figure 13-6) lists each step the SCTS has performed. The page lists, for example, the commands that were created. In addition it displays what was sent to and what was received from the test object. Each exchanged command is logged (including Session ID, Cmd ID, Msg ID, Command, Status, Source and Target Ref), as well as the result of the command.

Figure 13-6. SyncML Conformance Test Tool Command page

graphics/13fig06.jpg

The Logs page gives easy access to all log files generated by SCTS during test case execution.

Finally, every action and state is not only in the log files, but is also displayed on the status page (which is not shown here).


    Team-Fly    
    Top
     



    SyncML. Synchronizing and Managing Your Mobile Data
    SyncML: Synchronizing and Managing Your Mobile Data
    ISBN: 0130093696
    EAN: 2147483647
    Year: 2001
    Pages: 124

    flylib.com © 2008-2017.
    If you may any questions please contact us: flylib@qtcs.net