Tools

awk

Awk is a scripting language that is best suited for bulk modification and formatting of text files. Generally speaking, awk scripts move through a text file looking for patterns and perform some sort of modification (or insertion or deletion) on the pattern when found.

Awk comes in several flavors including:

awk:	The original ca. 1977 scripting language named after its creators, Alfred Aho, Brian Kernighan, and Peter Weinberger.
nawk:	New awk, a 1985 improvement of the original programming language.
gawk:	GNU awk, the GNU implementation of the awk programming language.

Awk programs are always invoked from a command line starting with the word awk. Next on the command line comes either a pattern/action sequence enclosed in quotes or a filename in which the pattern action sequence is stored. The third argument is the text sequence or file against which the pattern is to be matched and upon which the action is to be performed.

Processing of awk scripts is on a per-line basis. That is, each line is considered a separate entity against which patterns may be matched and upon which actions are performed.

The following are some examples of simple awk scripts.

Example: The following script prints out the first element of each line in the /etc/passwd file. Note that although there is no pattern specified, the action (print $1) prints the first token of each line. The -F option is used to specify a field separator of ':' in place of the default field separator of white space. The action to be performed is encased in curly braces.

awk -F : '{print $1}' /etc/passwd

Example: The following script prints out each line of the file data.txt containing the pattern 'val1'. The pattern to be matched is delimited by forward slashes.

awk '/val1/' data.txt

Usage

Awk is a good general-purpose tool for manipulating text files. In recent years it has been somewhat overshadowed by Perl, which does in fact do pretty much everything awk does and more besides.

Further Reading

sed & awk, (2nd ed.) by Dale Dougherty and Arnold Robbins Linux Shells by Example by Ellie Quigley

bison

Bison is the GNU adaptation of yacc (Yet Another Compiler Compiler). It is a tool that takes a formal grammar specification as input and gives a C program that parses input files (e.g., computer programs) that conform to the rules of that formal language as output. Bison is usually used in conjunction with flex (a.k.a. lex) to generate toy parsers that fulfill the homework assignments for graduate-level computer science courses.

To use bison effectively you need to have at least a nodding acquaintance with regular expressions, context-free and context-sensitive grammars, and the theory of computation. For this reason bison is not commonly seen in the wild. Once bison is added to your mental toolchest, however, you'd be surprised how often it comes in handy.^[1]

^[1] No, really. For example, just after I got done taking compiler construction, I somehow became involved in a project whose goal was to parse BibTex text files into records and store them in a database. Yacc suited this purpose nicely.

Further Reading

lex and yacc by John R. Levine, Tony Mason, and Doug Brown Compilers: Principles, Techniques and Tools by Alfred Aho, Ravi Sethi, and Jeffrey Ullman

Introduction to Compiling Techniques: A First Course Using ANSI C, Lex and Yacc

Ah, yes. C. The C programming language is the backbone of the UNIX operating system and, by extension, the Internet. Most UNIX system programming is done in C.

C suffers somewhat from its age in that it is not explicitly object oriented nor originally designed to support GUI applications. It is also relatively easy to write totally unreadable C code. However, its achingly terse syntax and unrivalled power continues to thrill cognoscenti and junior geeks alike, even 30 years after its inception. To a large degree, C continues to be the default language of computer science in general and UNIX in particular. If you learn only one programming language it should be C.^[2]

^[2] Or maybe Java.

Usage

UNIX system programming, network programming, character- based application development, and (with the right library, perhaps glibc/GTK+) GUI application development are not particularly well suited to Web applications, though they can be used profitably for creating extensions to the Apache Web server.

Further Reading

The C Programming Language by Brian W. Kernighan and Dennis M. Ritchie

C++

C++ is an object-oriented extension of the C programming language. Though it is currently in the process of being supplanted by Java, C++ went through several glory years as the industry standard object-oriented programming language. For this reason it has an enormous base of installed code. Consequently, even if all C++ development were to cease immediately the industry would still need C++ programmers for the foreseeable future to maintain legacy systems.

C++ is almost (but not quite) backward compatible with C. The main difference between the two is that C++ requires detailed function prototyping.

Further Reading

The C++ Programming Language (Special Edition) by Bjarne Stroustrup

C++ Primer (3rd ed.) by Lippman and Lajoie

ColdFusion

ColdFusion is a commercial product designed to facilitate the interaction between client browsers and any number of database servers, email servers, and programming tools.

ColdFusion is sold alongside a visual development environment, ColdFusion studio, which Web designers seem quite taken with. Depending on your needs, this alone may justify the not-inconsiderable purchase price. It enables you to develop applications using a drag-and-drop interface similar to Visual Basic.

ColdFusion has the strong advantage of being able to facilitate complex interactions between client, server, and database without the developer doing much typing. Also, the end product usually looks pretty sharp.

Also on the down side, ColdFusion was designed for Windows platforms and it shows. The only officially supported open source platform for ColdFusion is Red Hat Linux. The company documentation contains a lot of implicit assumptions about the platform that just don't hold true for UNIX. However, it is certainly not impossible to get ColdFusion applications to run under Linux, particularly Red Hat. Evaluation copies and details about how to purchase ColdFusion are available from:

http://www.allaire.com

CORBA

CORBA is an acronym for Common Object Request Broker Architecture. As the name implies, it is a mechanism for constructing platform-independent applications. The advantage of using CORBA software is portability. CORBA programs require that data structures and methods that are to be used by the application first be defined using the Interface Definition Language (IDL). To access these resources, individual components of the application pass their requests through the Object Request Broker (ORB).

Resource requests are made via a component called the ORB, which routes the requests as appropriate and relays the results to clients.

The CORBA design uses aspects of the a client server model in that there are clients that want services. However the client server analogy is imperfect in that the process taking a client role in one situation may take the server role in another. Indeed, much of the communication is peer to peer.

CORBA uses the OMG IDL to define the interfaces to object and the data types used to communicate with them. The IDL is a language devoted exclusively to declarations no other types of statements are possible.

Though the use of CORBA does add to the initial development time of an application, it makes later modifications dramatically simpler to implement. Individual services can be replaced atomically, without modifying the mechanisms whereby the clients access those services.

Usage

Due to the initial overhead that accompanies its use, CORBA is best suited for large commercial software projects that anticipate existing in a few years. CORBA should also be studied by C++ programmers who like money.

flex

The lexical analyzer known as flex (a.k.a lex, in UNIX-speak) is a tool for recognizing the smallest units of a formal language. Flex is traditionally used in conjunction with yacc (bison) to create compilers. Flex sifts through an input file (e.g., C source code) and divides it into grammatical units known as tokens or lexemes. These tokens are generally passed on as input to a program such as bison that attempts to recognize the patterns of a formal grammar within them.

Usage

Flex is typically used in compiler design. However, it can also be applied to any situation where you need to use formal language specification to create a parser.

Further Reading

HTML: The Complete Reference by Powell

HTML & XHTML: The Definitive Guide by Chuck Musciano and Bill Kennedy

Java

Java is an object-oriented programming language developed by Sun Microsystems. Java originated at Sun but has since gone on to more or less devour the programming world. Java was first released in 1993, about the same time the popularity of the World Wide Web was exploding. This was a fortuitous coincidence to which Java owes at least some of its success. As it turns out, Java programs are largely platform independent, a property that made them ideal for generating dynamic content in the aggressively heterogeneous environment of the World Wide Web.

Java applications may be written either as traditional standalone programs or as applets designed to be run under applet viewers such as those found in modern Web browsers. The functionality of applets is somewhat more limited than that of regular Java programs for security reasons. Error handling is built into Java.

Both applets and applications are compiled into platform-independent classes. Let me say that again: Java executables are platform independent the same one will run on UNIX, Windows, Apple, whatever. How is this possible? Unlike traditional executables that contain platform-specific machine code, Java compiles into an intermediate language that is machine independent. This intermediate language is run on the various platforms using a virtual machine that is itself platform specific but knows how to translate the Java class files into the language of the local CPU.

As mentioned earlier, Java programs are composed of classes. Classes are in turn composed of methods, which are sort of analogous to traditional functions in that they do stuff and return results when they're done.

Perhaps Java's biggest strength is its vast collection of publicly available class libraries, which provide building blocks you can tie together to perform almost any task. A big part of mastering Java is knowing where to go to get the class somebody else wrote that does what you need to do.

Further Research

Java: How to Program by H.M. Deitel & P.J. Deitel

http://java.sun.com

JavaScript

JavaScript is not a subset of Java, or even really much like Java. The name is a marketing ploy. Don't let that prejudice you against it, however, as it has much to recommend it.

Not least, JavaScript is supported by both Netscape 4 and Internet Explorer 4. This means that Web developers can include executable content in Web pages without worrying much about whether their client browsers will support it or, as in the case of PHP, whether they can find a server enabled for it.

JavaScript is an interpreted language. Typically, said interpretation occurs on the client side in the Web browser, though server-side extensions to the basic package do exist. As with most modern languages, JavaScript is fully object oriented. It is notably different from Java in its use of weak typing new variables can occur anywhere within the script.

Obviously much of what you do with JavaScript is going to involve dynamic generation of HTML code. However, you can also use it to access databases, do mouseovers, interact with Java applets, and manipulate cookies.

Usage

Broadly supported extension of basic HTML functionality.

Further Reading

JavaScript: The Definitive Guide by David Flanagan

lex

See flex.

Make

Make is a utility for compiling executable files from a collection of source code files. Make's single most sterling virtue is the ability to detect when a piece of the source has become outdated and needs to be recompiled.

To use make, you must first create a Makefile. Makefiles specify the components of the executable, the dependencies of the components (e.g., object files are dependent on source code files, source code files are dependent on libraries) and the command lines that are used to compile the various pieces.

The syntax of the Makefile can be a bit dense, but it's well worth wading through. Once your Makefile is written, all you need to do to compile a program is type "make" and the executable will be assembled. Frequently, this is just a simple matter of rebuilding an object file or two and relinking the results. However, makefiles can be written that check for new revisions of source code in your revision control software, install the finished executable, or clean out your directory when things get untidy.

Usage

All software developers should know how to use make.

Further Reading

Managing Projects with Make by Andrew Oram

MySQL

MySQL is a database management system, not unlike Oracle, Informix, and so forth except that it doesn't cost anything. I've used it fairly extensively and I have nothing but good things to say about it. It can be easily integrated into application software using almost any of the major programming or scripting languages (including C, Perl, Python, Java, PHP, and many more). The documentation for MySQL is world class, including MySQL, by Paul Dubois and Michael Widenius, which is perhaps the best technical book ever written on any subject.

Though MySQL recently went Open Source, commercial support packages are available. About the only two disadvantages to using MySQL are the lack of transaction support and the fact that it isn't Oracle. (I'm not necessarily a huge fan of Oracle, but you have to admit it's become the standard).

Usage

Anyone developing software on a budget should investigate MySQL. MySQL is also a perfect tool on which to learn database management.

nawk

See awk.

Perl

Perl, by Larry Wall, is the standard by which other scripting languages are measured. Perl is an extraordinarily powerful language, which frequently manages to remain readable. Though applicable to almost any task, Perl's strong text processing and network interface components make it particularly suitable for scripting system administration tasks.

The not inconsiderable basic functionality of Perl is further extended through the use of Perl modules. Perl modules are software components that provide specialized capabilities such as database interaction, CGI utilities, and graphics. Vast libraries of Perl modules are available on the Web and may be downloaded free of charge. Perl modules may be loaded into your executable at run time; no re-compiling is necessary.

Usage

Perl is good for just about everything. Some of the highlights are network programming, database interaction, string manipulation, and Web applications.

Further Reading

http://www.cpan.org

Perl by Example by Ellie Quigley

Learning Perl by Randal Schwartz

Programming Perl by Larry Wall

PHP

PHP stands for Personal Hypertext Preprocessor. It is a scripting language designed to create programs that will be embedded in HTML source files. PHP vastly extends the functionality of HTML. Some of the more flagrantly useful features are database connectivity, image manipulation, and data structure support, to name a few.

It would not be entirely unfair to describe PHP as a poor man's ColdFusion.

To use PHP, your Web server must be PHP-enabled. PHP- enabled Web servers scan the text of files requested by clients for PHP tags before the pages are served. If a PHP sequence is found within the file, the Web browser executes the PHP commands and serves the results of those commands to the clients. If all goes well, the results are valid HTML code.

Usage

Web application development.

Further Reading

http://www.php.net

Core PHP Programming by Leon Atkinson

Python

Python is an emergingly popular general purpose scripting language. It is interpreted (as opposed to compiled). Python is roughly comparable to Perl in terms of functionality but tends to be much more readable, to the point where Python has been described as executable pseudo-code. Python is freely available for a wide variety of platforms, which of course include Linux.

It is an extremely high-level language intended to enable programmers to create working programs quickly, without worrying about machine-specific trivia. It's a good choice for creating quick and dirty applications where high performance is not a key design requirement but speed of deployment is desired.

Further Reading

Practical Programming in Tcl and Tk by Brent Welch

Tcl and the Tk Toolkit by John Ousterhout

yacc

See bison.