Section 3.3. Open Source Skills Inventory

3.3. Open Source Skills Inventory

This section provides a precise definition of what is meant by each category of skills in each level, so an IT department can understand exactly the sort of skills it will need to succeed using open source. (In Chapter 5, a program on skill improvement will be proposed.)

3.3.1. Open Source Development Tools

Open source started as primitive fundamental elements, languages, and operating systems that were combined and recombined to create the incredible trove of software available today. Appendix A examines the different sorts of platforms that can be constructed with open source.

One of the earliest and still most active areas of open source is the creation of tools for developers. The keyboard mappings of Emacs and vi editors are deeply embedded in the brains of millions of developers. Other tools such as Ant are used widely to compile and assemble programs. Open source is bundled together with its own set of tools, such as tar.

The fact is, when you start digging into an open source project, you might encounter 5, 10, or 15 open source or Linux development tools or commands that are crucial to understanding how to compile and construct the project on your computer. The first few times you wade through this it can be slow going. But eventually you become one of the informed, and a process that started out taking 6 to 8 hours now takes 15 minutes.

The most important tools to understand are:

Utilities for dealing with distribution archives (tar, gzip, bzip, zip, unzip, etc.)

Almost all open source projects are distributed as downloadable "tarballs."

The GNU configure and build system (autoconf, automake, libtool, gettext, m4, and perl)

This system is the most popular among open source projects written in C, C++, and other low-level languages. It uses the three-step process of configure, make, and make install to install software from source by determining your system's setup (configure step), compiling the source and linking it to your existing libraries (make step), and then installing the source according to your system's conventions (make install step).

Perl-centric systems

These follow a high-level process that is built on top of the GNU configure and build system. It involves generating a system-specific makefile based upon your Perl installation parameters. For most Perl module installations, Perl provides a very simple interface to the Comprehensive Perl Archive Network (CPAN) that takes care of all the low-level details.

Java-centric systems

These have started to unite in their use of Apache Ant. More comprehensive build and configure systems such as Apache Maven are also ready for prime time.

PHP systems

These are more informal in their packaging. Since most PHP systems are accessed through a web server, installation normally involves unpacking the source in the web server's document space. Larger PHP systems use the GNU configure and build system.

Python-based systems

These are a mixed bag when it comes to build and configure methods. The more popular and mature systems come with sophisticated installers for each major platform, since Python is portable and works on Windows, Mac OS X, and Unix systems equally well. Less mature systems fall back to GNU autoconf or tarballs.

3.3.2. Hosting

Open source software has to run somewhere, and the hosting environment must be friendly to the installation, operation, and support of open source if it is to ever go into production. Here we describe the difference between beginner/intermediate hosting, which is focused on running open source reliably as a black box, and advanced/expert hosting, which is focused on optimization for the highest possible performance:

• Beginner/intermediate hosting

  • Manage domain registration information (setting up domains).

  • Manage bandwidth and disk-space quota issues (control space used by installed open source software and log files).

• Advanced/expert hosting

  • Manage web server configuration (set up virtual hosts).

  • Manage runtime configurations (tune PHP or Java runtime settings to help the open source software run better).

  • Manage DNS records (redirect traffic to a test site, for example).

3.3.3. System Administration and Operations

System administration and operations skills can be a significant problem when open source is introduced into departments that lack Unix skills. Many of the tools used in open source, and the way the programs interact with the operating system, are Unix oriented, and if an IT department already has Unix skills, adoption is easier. Certain skills are needed for running open source as a black box, and other skills are needed for running it in a more high-performance mode for optimization. System administration and operations work for open source projects is largely similar to that performed for commercial software. There is just more of it because of lack of productization. These are the skills required at each level:

• Beginner/intermediate system administration and operations skills

  • File management: if you are trying out a few open source projects (or one large one) you might need to manage space issues by recompressing tar files. You might also need to tweak the install process by setting up symbolic links and deleting unwanted files. From time to time you might also need to distribute files securely, as well as manage file permissions and ownership.

  • Execute commands and scripts: often the open source project is available as a tarball, and the README or INSTALL file describes the series of steps needed to get it running.

  • Starting, stopping, and monitoring applications: on Unix systems, the init scripts are not usually provided by the installation process and have to be configured manually.

  • Testing: open source projects typically have a wide installed base, and the installation and configuration process does not anticipate all the environments it is going to be installed in. So, the process of installing and configuring an OSS project is filled with incremental testing steps. Upon encountering an error, one often has to check the project's community documentation (FAQs, mailing lists, etc.) to find similar situations and possible solutions.

• Advanced/expert system administration and operations

  • Patches: apply security-related patches and associated corrections.

  • Monitoring: perform monitoring and intrusion detection.

  • Manage process resource usage: control the resource usage of long-running processes.

  • Manage configurations: sometimes the configuration for your system might not match the assumptions made by the install process of the open source project you are trying to install, and you might need to edit system configuration files.

  • Network interface configuration: in general, when dealing with web-based systems you might need to tweak the configuration of your network interface.

  • Change control: since living with open source software means changing the source or configuration, it is vital that you manage these changes via change control.

3.3.4. Open Source Infrastructure

An understanding of the infrastructure components that form the open source stack is crucial to rapid evaluation and successful deployment of open source. Most of the time, open source projects are constructed from top to bottom using open source components. Linux is the most common operating system, although it is not a requirement. Most mature open source runs well on Windows operating systems and other flavors of Unix. MySQL or Postgres can be used as the database, and Apache is usually the default web server. Other common components can also be included, depending on the programming language used to create the project.

Each component is a world unto itself that usually must be configured in a way that allows the open source program to work correctly. It is not uncommon for the configuration of one or more of these components to have to be adjusted to make an open source project work in a particular environment. For example, making open source work with single sign-on systems can be challenging. If an IT team is not familiar with a particular component, it can take a while to understand how to adjust the component to the required configuration. Familiarity with components can reduce this time period to a matter of minutes. Here is a list of the sort of knowledge that is helpful to have about each of the most prominent components of the open source stack:

• Linux

  • Knowing which configuration settings are commonly in need of adjustment

  • Associated package management functions

    Red Hat Package Manager (RPM)

    BSD has ports

• Apache

  • Virtual hosts

  • Apache module management

  • Access control issues

• MySQL

  • Basic database administration

  • Backup/restore

3.3.5. Programming Languages

To really understand an open source project and be able to analyze and fix bugs, it is important to be able to read and understand the programming language in which the project is written.

Understanding the programming language of all but the most mature open source software should be a requirement for mission-critical use of all but the most mature category of open source projects. Using Linux, Apache, MySQL, and other programs seldom requires knowledge of source code.

But imagine how hard it would be to track down a problem in a less mature project if you had to learn the language and try to understand the program at the same time. Of course, if the use of the software is not mission critical, an IT department has more time. But when faced with a severe operational crisis, all the skills required for diagnosis and debugging should be available.

This is not to say that you must be an expert in every language, or that it is impossible to use an open source project without knowing the language. But the more deeply you become involved with, and dependent on, an open source product, the more likely it is that you will need to interact with some level of the source code.

The source code can be useful for several different reasons:

• During evaluation of an open source project, the source code could be inspected as a way to evaluate the quality and maturity of a project.

• If the project must be extended to add new functionality or to support integration with other software, the source code must be consulted.

• If a bug is encountered that nobody else has discovered, the source code is required.

• For security audits, the source code is required.

• For performance tuning, it is often necessary to identify and improve a particular area of a project's functionality. This is impossible without source code.

In all of these situations, skills in the programming language can greatly reduce the amount of time required and the associated risks.

But knowledge of a programming language itself is really only the beginning. Open source software, and commercial software as well, is not constructed out of the programming language, but rather, out of all the reusable libraries of utilities and other special-purpose code. The real understanding of a language for open source purposes must include knowledge of these libraries, or at least skill in quickly understanding them. The Perl language has one of the most comprehensive collections of libraries at CPAN.org. As noted earlier, CPAN stands for the Comprehensive Perl Archive Network.

The most popular languages for open source development are:

C

The C programming language is a low-level, standardized programming language developed in the early 1970s by Ken Thompson and Dennis Ritchie, for use on the Unix operating system.

C++

Bjarne Stroustrup began C++ in 1978 as a "simulator for described computer systems." He wrote it for his Ph.D., at the Computing Laboratory at Cambridge University in England.

Perl

Perl is a practical language for extracting information from text files and generating reports from that information. It was released by Larry Wall in 1987.

Python

Python is a high-level scripting language invented by Guido van Rossum in 1989 to be powerful and easy to understand.

Java

The Java language was created at Sun Microsystems in a project initiated by Patrick Naughton, Mike Sheridan, and James Gosling in 1991.

Ruby

Ruby was created in 1993 by Yukihiro Matsumoto to be an object-oriented scripting language.

PHP

PHP is a simple scripting language for personal use, created by Rasmus Lerdorf in 1995.

In Chapter 5, we analyze the issue of choosing a preference among these languages. The important issue for this chapter is determining what sort of skills are required at the different levels.

Use of source code by beginner and intermediate-level users is confined mostly to use of templates or small snippets of source code for configuration. At the advanced and expert levels, source code is read to understand the larger structure of the program for debugging or extending the program.

3.3.6. Open Source Community Skills

The storehouse of knowledge in open source projects is other people. To make the community work for you, you must know the community's rules, ethics, and style. Remember, open source projects are not staffed by people who are paid to help you. No amount of yelling or abuse can get the community's attention. If you come to the community with a last-minute request, it is not likely that anyone will drop everything to help you, unless implementing that request is somehow interesting to them. Losing "one throat to choke"--a vendor that is on the hook to helpis one of the greatest fears of those who are suspicious of open source.

This warning aside, open source communities generally are extremely friendly and are populated by people who are more than willing to help others who are making a sincere effort and are not asking lazy questions.

3.3.6.1 Evaluating the maturity of open source

As we discovered when writing this book, evaluating open source is an art, not a science. One of the key skills for anyone who seeks to use open source is the ability to evaluate an open source project and determine what it will take to use and support it. Beginner/intermediate-level users are mostly interested in what it does out of the box, while advanced/expert-level users will perform a deeper evaluation.

The key is to develop your own efficient style of evaluation. In Chapter 2, we presented one style based on the evaluation of hundreds of open source projects we had researched for this book. Your style will depend on your skills and the skills you want to develop. Most of the time, a beginner/intermediate-level evaluation tries to determine what the project claims to do, how well it does what it claims, what new skills will be required, and how it can be supported. The advanced/expert analysis dives more deeply into the project's architecture, the size and nature of the community surrounding the project, the quality and management style of the project's leadership, and the project's long-term prospects.

3.3.6.2 Networking with open source developers

Open source users at all skill levels must be able to unlock the value of the community. The forums and resources at open source project web sites are just the beginning of the trail to tracking down information. The people who post and answer questions are potentially valuable resources who can be tapped to help out, accelerate learning, and provide consulting services. The key skill is engaging such people in a respectful way. The approach can mean everything. "Hello, I have a problem. Please do my work for me," is seldom a winning line of inquiry. Rather, if you have done research and you show the person that she might learn from you, it is much more likely that you will be able to engage her in a productive conversation.

Key social skills are not sales oriented, but rather, depend on polite directness, respect for the time of others, and a demonstrated willingness to work to research problems and share the results. In the article "How to ask questions the smart way," Eric Raymond distills the conventional wisdom: do your homework first (i.e., search the web or forum archives before asking), ask politely (i.e., provide all the relevant details and only the relevant details), and follow up when you find the solution.