Section 23.4. EXPERIMENTAL CRITIQUE

23.4. EXPERIMENTAL CRITIQUE

Our goal in undertaking an early assessment of aspect-oriented programming was to gather information that could be helpful in directing the evolution of the technology. In support of this goal, we wanted to gather data about both the performance and the experience of programmers using the approach. We chose an exploratory study format in which we compared the results of a small number of programmers using an aspect-oriented approach with others using a control language for several reasons:

• The control groups provided a basis on which to assess the performance of the AspectJ groups,

• The cost of running and analyzing a trial was high so we wanted to balance cost with the maturity of the technology,

• The pool of potential participants and the amount of time available from each participant was limited, and

• We chose to forfeit some precision in measurement in favor of realism [15].

Within these constraints, we took several steps to ensure our results had some validity. To achieve internal validity, we provided the different groupsaspect-oriented versus non-aspect-orientedwith as similar support as possible, limiting variances, as much as possible, to the features of interest. For instance, the lock constructs we provided for use in Java and Emerald provided a mechanism similar to that available in AspectJ.

To address construct validity, we gathered data from multiple sources. One source was the qualitative statements made by the participants during the taped interviews; the other sources were the data analyzed from the tapes. Sometimes the data from the multiple sources was corroborative, other times it was contradictory. Corroborative data strengthened the result under discussion: Contradictory data weakened the result.

The reliability of our experiments was high with respect to the procedures we followed in conducting the experiments and analyzing the data. However, as expected, the skills of the participants varied greatly as finding participants with Java (or Emerald) experience, concurrent programming, and, in some cases, distributed programming was difficult.

The variability in the skills of participants and the modest number of participants limits the generalizability of our results. We chose to make this tradeoff because, in these exploratory studies, we were interested in how the participants worked with the approach; the quantitative data supported the analysis of the qualitative data.

The external validity of the experiments is also affected by the problems we chose as a basis for the experiment and the limited training provided to the participants. The faults seeded into the system for the debugging experiment, for instance, were all synchronization problems that could be solved by altering Cool code. We informed the participants that synchronization faults had been seeded into the program; AspectJ participants may thus have been pointed towards the Cool code. The performance of all of the participants may also have been affected by being asked to work with either new languages (the AspectJ participants), or with particular constructs (Java and Emerald) introduced to provide a basis of similarity between the languages. Scholtz and Wiedenbeck have shown that programmers experience a drop in performance and their solution process is disrupted when using an unfamiliar programming language [20]. All of our participants likely experienced this effect in differing degrees; our experimental method did not allow us to explicitly quantify or qualify the differences.

The limitations in our studies could be overcome by refining and expanding the experimental method. However, the cost entailed in conducting more controlled experiments must be weighed against the development curve of the technology being studied. Our exploratory study format has provided insights useful for researchers building aspect-oriented programming environments at this early stage of the technology.