Conduct a Make-or-Buy Analysis

Conduct a Make-or-Buy Analysis

The make-or-buy analysis determines whether a course should be made in house or outsourced and how much to spend (see Figure 4-3 on page 103). Our criteria were the size and location of the audience, availability of subject-matter experts, whether the content was proprietary to Rockwell Collins, and value to the organization.

click to expand
Figure 4-2a: Learning Methods .

click to expand
Figure 4-2b: Learning Methods.

click to expand
Figure 4-2c: Learning Methods.

click to expand
Figure 4-2d: Learning Methods.

click to expand
Figure 4-2e: Learning Methods.

click to expand
Figure 4-2f: Learning Methods.

click to expand
Figure 4-3: Make-or-Buy Flowchart.

For example, 2,500 employees needed training on electromagnetic interference (EMI) ”a problem that was costing Rockwell Collins more than a million dollars per year because of inadequate application of EMI fundamentals in the design phase. This easily justified the $400,000 the learning and development department spent on a custom course that helped to eliminate the problem in a matter of months.

On the other hand, if a core group of plant workers need to capture the knowledge of one retiring expert, it makes far more economic sense to use a methodology other than an expensive CBT. A key aspect of selecting the appropriate design methodology is obviously the cost per student hour of the training. Other factors are the location of the learning population and flexibility of the delivery method.

If our team decides to buy a course, we go to one of our existing vendors and make a choice. If our team makes a course in house, we use further steps to validate methodologies, style, core competencies, and effectiveness.

We contracted with an instructional-systems-design expert to craft a comprehensive curriculum-design and development standards guide that would provide focus, direction, and required "rules of engagement" for Rockwell Collins. The goal of the guide is to facilitate development of effective education and training programs.

It establishes the roles, responsibilities, and requirements for all those involved in curriculum development, management, and implementation at Rockwell Collins, including course managers, coordinators, instructors, subject-matter experts, support personnel, and others involved with instructional development and execution. It identifies the primary sources of instructional media used at Rockwell Collins, including detailed style guidance for use of type, font, style, color , and formats for media-specific presentations, along with design requirements, design templates, lesson file organization, and delivery methods and techniques.

For content that is developed in house, developers and SMEs are expected to adhere to these standards. Every vendor our team worked with from that point on was required to use our standards guide as a model for our content and to work with the Rockwell Collins team to ensure that the highest-quality design standards were applied.

Flowcharting these processes and defining standards are simple activities that clearly define for the entire organization how and why the learning and development department chooses particular content and delivery methods. They give our team guidance and verification that the choices we make have sound instructional principles behind them. These processes were also very important to the development of our learning consultants .

Define Roles and Responsibilities of Learning-Team Members

Each work flow was a piece of a formal process we were creating to ensure that the training our team offered was consistent, had defined goals, used sound instructional-design theory, was of the highest quality, and tied training to the goals of the company. Many of the processes had subsets that were also mapped out. For example, whether we built training in house or worked with an outside vendor, a team of Rockwell Collins people would be involved. We weren't just going to assign training development to a vendor and see what the vendor came up with six months later; we were going to manage the process from the front end. That meant that every custom course required a subject-matter expert and a learning consultant to work on the vendor's team, and each member had a defined set of roles and responsibilities (see Figure 4-4).

start figure

Custom Courseware Development Roles and Responsibilities



Rockwell Collins

Learning Director

Approves project funding, direction, and scope changes.

Project Manager

Oversees the project; directs flow of tasks ; coordinates development schedule; keeps team informed of changes; stays focused on the "big picture"; ensures application of company standards and style guidelines; ensures vendor-proposed architecture is consistent with internal technical infrastructure; tests final product to meet functional, technical, and instructional requirements of the project; interacts with all team members.

Subject-Matter Expert

Intellectual resource for project content; responsible for providing the tasks and performance steps to the instructional designer (ID); the final authority on what something means or how it is to be performed; concerned with content accuracy, not instructional design; primarily interacts with ID and Project Manager; provides content capture for instructional designers; reviews scripts and lesson betas, specifically to read script, make edits, organize and send comments to developer, and participate in conference calls to clarify comments; must confirm commitment to and availability on the dates indicated on the schedule; understands and agrees to communication format and frequency during project.

Content Approval Authority

Approves the conversion of subject matter to electronic form; delegates content authority to subject-matter expert; interacts with SME, Learning Director, Project Manager.

Learning Technologies Manager

Provides Pinnacle Learning Manager software and infrastructure support as needed.

e-Business Project Manager

Coordinates required e-business services to advise , approve, and provide support for the project.

Course Developer/Vendor

Account Manager

Initiates project with Project Manager; writes sales proposal based on customer requirements; interacts with Project Manager and Course Manager.

Course Manager

Acts as the "voice" of the customer to the vendor-development team, ensuring that the course meets the customer's requirements; manages course-development team for vendor; interacts with Project Manager to keep project on schedule; all communication to Project Manager directed through Course Manager.

Instructional Designer

Ensures instructional integrity of learning system through application of instructional design and clear writing; responsible for how material will be presented; acts as primary contributor to scripts and storyboards by writing with CBT learner, audio narrator, content accuracy, and sound instructional design in mind; does not interpret content; ensures that implementation of graphics and programming is consistent with the intended instructional design; interacts with Course Manager, SME, Project Manager.

Lead Developer

Uses the agreed-upon authoring language or package to implement the ID's storyboards; writes the code to create the final product; incorporates visual recommendations of the graphic designer; designs overall architecture and functionality of product; manages additional programmers if needed; interacts with Course Manager.

Graphic Designer

Designs attractive and intuitive interface; creates the visual content, graphics, and animation of the course; assesses what style will best fit needs of customer; interacts with Course Manager.

Multimedia Developer

Responsible for audio, video, and animation elements of the course; interacts with Course Manager.

Quality Assurance

Responsible for testing applications to ensure that product execution matches design; interacts with Course Manager.

Clearly defined roles and responsibilities add clarity and consistency to the course-development process.

end figure

Figure 4-4: Roles and Responsibilities Chart.

Further, we discovered that previous development delays had often been caused by SMEs who were not the sole expert on a subject. Their missing knowledge resulted in significant gaps in the final version of the course, or the SMEs dropped out or were pulled off of projects prior to their completion. In both scenarios, developers were left with partially completed courses, which required them to return to the design phases, which in turn added time and costs to the project.

To reduce this waste, we developed a set of criteria to choose SMEs that helped the development staff identify the true subject-matter expert and whether more than one SME was required to round out the necessary knowledge. The criteria look at tangibles and intangibles, such as:

  • Their mastery of the content

  • Whether they are obvious representatives of the targeted audience

  • Whether they have clear knowledge of the audience's roles, responsibilities, and learning needs

  • Whether they understand the project context

  • Whether they are enthusiastic about the project and content

  • Their availability for the span of the project

  • Their credibility

  • Whether they empathize with the end users

This simple checklist gives developers a road map for choosing SMEs and improves the completion rate for courses.

To be sure that the SMEs they chose would be committed to and free to work on the project, we created a contract for SMEs and their supervisors to sign (see Figure 4-5). The contract defines the number of hours the SME will be expected to work on the project over a set period of time so that everyone's expectations are set ”on the front end. If necessary, the learning and development staff also agreed to pay part of the SMEs' salaries to compensate for their lost productivity on the job.

click to expand
Figure 4-5: SME Contract.

By retooling the roles of trainers and linking them to the business units, then supporting them with a structure of standards, guidelines, and process maps to follow, we made it possible for Rockwell Collins's learning and development team to consistently link training solutions to the business goals. We established defined criteria that led to consistently high-quality courses and reduced the costs necessary for development and creation of measurable performance changes within the organization.

Without these tools and relationships, trainers had no idea how to make the connection between courses and competencies. We created a strategy that bridged the gap between supporting the back end to creating an environment in which training achieved specific, well-defined objectives.