14.2 Implementing a Research Plan

Most of what we need to know about software development methodology is simply not known. We choose language systems such as Java and C++ strictly, but strictly based on current trends and fads. The choice of these programming languages has resulted in some incredible performance and reliability problems for those who have unwisely chosen to use them in inappropriate contexts. The problem is that we really do not know what are good applications, if any, to apply these programming languages. We do know that it will be literally impossible to construct reliable and precisely defined systems based on programming languages fraught with syntactic and semantic ambiguities such as C++. It might well be that C++ is really the language of choice for some applications. The disturbing fact is that we do not know what that context might be. Even more disturbing is the fact that there seems to be little real interest in discovering what that context might be.

14.2.1 Software Research Training Program

It is clear that there will be very little expertise to draw on to develop a software research program within today's corporate computer development organizations. We are confronted with the problem that there is very little academic preparation in the disciplines of software engineering, computer science or informatics in the conduct of empirical inquiry. There will be very limited opportunities to hire a research staff with empirical research credentials from individuals with this academic preparation. There are other disciplines, such as chemistry and physics, where training in empirical research is an integral part of academic preparation. Unfortunately, the individuals with the requisite background in empirical research have little formal knowledge of software engineering.

Given this dilemma, the obvious strategy is to institute a software research training program. It will be the objective of this training program to take individuals with cognate preparation in software engineering or computer science and bring them up to speed in the conduct of empirical research. Not every software developer needs to participate in this training program to the same degree. There should probably be three levels of training programs in a typical software development organization. These three levels would probably correspond directly to the individuals having the bachelor's degree, the master's degree, and the doctoral degree.

The first level of training in empirical research should be at the recognition level. Individuals trained to this level will be equipped to recognize sound empirical research. What little empirical research is currently available in the published literature is astonishingly weak. The experimenters reporting the research have failed to meet the simplest criterion for published science. That is, the reported research must be reproducible. Any outside observer should be able to read the published research and replicate the experiment precisely. The essence of science is that reported research results should be reproducible. The research training at the recognition level should equip the software developer to at least recognize science.

The second level of training takes the research process one step forward. Training at this level will involve the ability to conduct simple experiments to answer immediate questions in the software development process. This research process is simply scaled so that it can be performed by a small group of developers within a limited budget.

The third level of research training will equip an individual with the ability to design experiments in basic software research. Perhaps the most effective means to deliver this type of training is the mentoring system. Each novice researcher would be assigned to understudy a person, a mentor, who has recognized research experience.

14.2.2 Research and the Decision-Making Process

Many of the decisions governing today's software development processes are capricious and ill-considered. The decision to implement applications in different programming language metaphors is a very good example of this. For example, we really do not know whether it is better to use C, C++, Java, or even FORTRAN for certain applications. It is clear that there is no panacea as far as programming languages are concerned. No one language is clearly the best language for all applications. It might well be that C is to be preferred to C++ for certain classes of applications. Unfortunately, we do not know for which applications C might be the language of choice. Again, there are multiple concomitant criteria that must all be investigated simultaneously. We might wish to look at performance, reliability, maintainability, and security as criteria in evaluating a language. When we turn to the computer science or software engineering literature to seek guidance in the decision-making process to select a language for a new project, we find little or no help. There are plenty of anecdotal stories about languages in given applications, but no real science.

If we make the wrong decision in the choice of a language, we will possibly pay horrible consequences for this decision. What is worse about the initial bad decision is the fact that there are no processes in place to monitor the effect of having made the wrong decision. Any manager who has made a development decision will go to great lengths to guarantee that everyone clearly understands that the proper choice was made. A manager may have decided, for example, to use Java for a new application. The case might have been made that using Java would reduce the total system development time. After the system is up and running, it is then discovered that the new system is incredibly slow. To run it effectively will require investment in new hardware with sufficient processing speed to overcome the inherent performance hit taken by Java. The cost of this new hardware is the hidden cost of the manager having made a very bad decision in choosing a language. Meanwhile, the manager is basking in glory for having delivered a system on time and under budget.

In essence, to make sound engineering decisions about the software development process is going to mean that the first step in the process will be to invest in the necessary basic research to provide the information necessary to make informed and cost-effective decisions. Scientific inquiry should be an integral part of the software development process. Much of what we need to know to make effective decisions about software development is simply not known. What is worse is that the nature of the applications that we are developing is also rapidly evolving.

14.2.3 Institutional Research Reporting Mechanism

It is one thing to conduct a simple experiment that results in valuable insight into the mechanics of the software development process. It is quite another to be able to share this result across the corporate software development structure. Without appropriate reporting mechanisms, most of the results of simple experiments conducted on the software development floor will either be lost to history or replicated again and again by other software development groups. With today's level of connectivity through the Internet and on intranets, it is possible to share research results very widely and rapidly. This is an excellent mechanism for recording and sharing the results of corporate research projects.

14.2.4 Institutionalize the Research Review Process

If research results of homegrown microexperiments are to be widely distributed, there must be some type of peer review process instituted to filter this dissemination of information. The consequences of making software development decisions based on poorly conducted or ill-conceived experiments are too great. The entire nature of the peer review process has changed dramatically. The traditional peer review process is much too slow and labor intensive to provide the timely feedback necessary for a corporate research program. Again, the connectivity provided by local communication networks can provide the solution to the peer review process.

As the results of each research effort are made accessible through the local communication network, provisions should be made for online commentary. Each person who reads an article should have the ability to make a rebuttal to the article. These commentaries are then appended to the original article. Criticism of a critical and constructive nature should be invited. One of the major drawbacks to yesterday's research reporting was that the reader never had the opportunity to review the results of the peer review process. Once an editorial decision was made to publish an article, the contents of the article then became fact, regardless of the objections that any one of the reviewers might have had.

What immediately becomes obvious is the distinction between bad research and controversial research. Whenever new ideas are first introduced, there is a real reluctance on the part of the research community to embrace these new ideas. Hubble's work on the idea that the universe was not a static place but was rapidly expanding brought him the scorn of many of his colleagues. This new idea represented too great a threat to the established view of the astronomical community. In the long run, however, the notion of an expanding universe was empirically validated. Science prevailed. Franz Joseph Gall, on the other hand, developed the science of phrenology. In the world of phrenology, it was believed that by examining the shape and unevenness of a head or skull, one could discover the development of the particular cerebral organs responsible for different intellectual aptitudes and character traits. This theory could not be empirically validated. In the past, it took a very long time for the technical review process. It took even longer for the validation process. With today's communication capabilities, this whole process has been accelerated many orders of magnitude.

So, not only should a company invest in the research process, but it should also provide the infrastructure for reporting the research results and a forum for the review and analysis of research results.

14.2.5 Provide Kudos for Research Investigations

Unless some level of corporate recognition is built into the research process at the grass-roots level, research activities will either not get off the ground in the first place or will rapidly diminish. If research is seen as an increment to the daily grind, it simply cannot succeed. A nurturing environment must be created for a research program to succeed. In a research-friendly environment, research activities will be part of a job assignment. Employees can propose a simple research project, submit the project for peer and management review, and perform the research activity, all as part of their assignment.

It is clear that employees who are providing a research contribution in addition to their work commitment should be singled out for reward. Not everyone will be able to do research. Those who are capable should be encouraged to participate. There are two incentives that can be used very effectively here. First, there are financial rewards. Bonuses can be given for each published research result. A personal and privileged parking space might well be at the top of the list of motivators. Second, the researcher can be recognized in department-level meetings. Peer recognition is a very important process in the social structure of any organization.