EVALUATION IN AN INSTRUCTIONAL DESIGN SCENARIO

An important contribution to the e-learning arena is happening in the way of designing, locating and delivering educational contents in the Internet. Learning objects lead the set of theories devoted to the design and development of learning contents, also known as instructional design (Merrill, 1994). A concrete case of a system for the shared creation of knowledge is one dedicated to the development of learning objects. During the process, a number of instructional designers may wish to contribute. They could, for example, make some modification to the structure of a course or add some learning resource to the course contents. Interaction between authors should be coordinated to extend or modify the educational material. Authors try to meet a protocol that reflects their different interaction styles.

Under a constructivist instructional design approach (Koper, 1997), in the creation of educational content, not only must teachers take part, but also the receivers of the training (i.e., students). Therefore, consumers and designers of learning objects have a participative relationship, where the subordinated agents can also participate in the instructional design process, although the final decision lies with the higher-level agents .

Evaluation Scenario

The evaluation scenario consists of a knowledge mart where three agents are producing knowledge in the representation of a docent coordinator ( C ₁ ) and two instructors ( I ₁ and I ₂ ). The goal of agents in the mart is the development of a IMS/SCORM (IMS, 2001) learning object ” a course named ˜XML Programming ” that fulfills a set of educational objectives. Although there is room in IMS standards for describing each part of the learning object ” e.g., organizations, resources, metadata, etc. ” we will restrict the discussion to the ToC structure, devoting the interaction process n to it.

When authors submit proposals, they will include the differences between both ToCs and will refer to the same interaction. The interaction protocol is executed by every receiving author until the proposal is eventually accepted or replaced with a further elaborated proposal. This process continues until some proposal wins all evaluations, an agreement is reached, or until some degree of consensus is achieved (depending on the kind of interaction, i.e., competitive, negotiating or cooperative). Although the authors' behavior is an asynchronous process, the agents' interaction protocol helps to synchronize their operations.

Objectives

These are the educational objectives that define the preference relation used to evaluate proposals about the ToC of the course:

1. Ability to program XHTML (i.e., XML-generated HTML) web applications

2. Ability to program server-side web applications

3. Ability to program XML data exchange applications

The degree of fulfillment of educational objectives is modeled as a threecomponent vector x = ( x ₁ , x ₂ ,x ₃ ), with x _k ˆˆ I=[0,1] for k =1,2,3. Let f : I ³ ’ I be a numerical measure of how well a proposal meets the objectives.

Evaluation Criteria

The relevance of a proposal is graded by the fulfillment of the educational objectives described above. All objectives being equally satisfied, the rank of the agent will decide (coordinator is higher that instructor). If ranks are the same, the time when the proposal was issued will decide. To determine the instant of generation, every proposal will include a time-stamp.

Each proposal p is described by a three-component vector ( p ₁ , p ₂ , p ₃ ), where:

• p ₁ = f ( x ) measures the degree of fulfillment of educational objectives.

• p ₂ is the numerical rank held by the submitter agent.

• p ₃ is a time-stamp.

Notation for Proposals

To simplify the notation, proposals are represented by x _ij , where x _i is an identification of the author, and j is a sequence number ordered by the instant of generation of the proposal. Using this notation, the following proposals will be elaborated by agents in the mart:

• i ₁₁ : Create a unique chapter for "XML script programming".

• i ₁₂ : Divide "XML script programming" into two new chapters: "Clientside XML script programming" and "Server-side XML script programming".

• i ₂₁ : Add a chapter about "Document Type Definitions (DTD)".

• i ₂₂ : Add a chapter about "DTD and XML schemas".

• c ₁₁ : Add a chapter about "Using XML as data".

Preference Relationship

A preference relationship > is defined between any two proposals p = ( p ₁ , p ₂ , p ₃ ) and q = ( q ₁ , q ₂ , q ₃ ):

The preference relation given in (1) defines a partial order, where i ₁₁ < i ₁₂ < i ₂₁ < i ₂₂ < c ₁₁ , in accordance with the evaluation criteria.

Sequence of Events

The sequence of events generated by agents in the mart is made up of three acts , which are depicted in Figure 2 and traced as follows :

1. I ₁ starts by sending a proposal i ₁₁ . When i ₁₁ is about to be consolidated, I ₂ agent will issue a better evaluated proposal i ₂₁ . C ₁ does nothing and silently accepts every proposal that comes to it.

2. I ₁ and I ₂ elaborate two respective proposals i ₁₂ and i ₂₂ , approximately at the same time during the distribution phase. The proposal from I ₂ has a better evaluation than I ₁ 's, and both are better than those in the first act.

3. C ₁ builds and sends the best-evaluated proposal of this scene, which will eventually win the evaluation.

Figure 2: Sequence of Events for the Learning Object Evaluation Scenario

The series of messages exchanged during act (2) is depicted in more detail in Figure 3 and is described as follows:

1. Both I ₁ and I ₂ receive each other's proposal and begin the distribution phase, therefore starting timeout t . Proposals i ₁₂ and i ₂₂ also arrive at C ₁ , which is not participating in the process and silently receives them.

2. I ₁ compares i ₂₂ to i ₁₂ , turning out that its proposal has a worse evaluation. It is reasonable that an evaluation of proposal i ₁₂ obtains a higher value than i ₂₂ , as for the second objective described above. Concerning the first and third objectives, any relevance function should result in similar values for both proposals, so they would not be decisive . Then, I ₁ starts timeout t ₁ , giving i ₂₂ a chance to be consolidated. On the other hand, I ₂ also compares both proposals and reminds I ₁ of the results by again sending i ₂₂ , then extending timeout t in order to give a chance for other agents' proposals to come.

3. When timeout t expires , I ₂ sends a consolidation message for i ₂₂ that arrives to every agent in the mart. At the reception , I ₁ finishes the protocol because it is expecting the consolidation for i ₂₂ . C ₁ simply accepts the notification.

4. Finally, at the expiration of t ₁ , I ₂ is notified about the end of the consolidation phase for i ₂₂ , and its execution of the protocol finishes successfully. Therefore, every agent in the mart will eventually know about the consolidation of the proposal.

Figure 3: Execution Example of the Interaction Protocol

These tests have been carried out to examine how far the multi-agent architecture facilitates the coordination of a group of actors that are producing learning objects. In this educational scenario, quantitative measurements of performance and throughput were taken concerning observable actions and behaviors about a number of aspects, such as conflict-solving facility, effectiveness in coordinating the production process, fair participation of members from other groups, overall quality of the final results, and speed and quality of the generated knowledge.

In this context, the agent-mediated solution has been found to facilitate the following aspects of the distributed creation of learning objects during the instructional design process:

• Bring together instructional designers' different paces of creation.

• Take advantage of designers' different skills in the overall domain and tools that are managed.

• Reduce the number of conflicts provoked by interdependencies between different parts of the learning objects.

• In a general sense, avoid duplication of effort.