Preface
"And Now for Something Completely Different . . . Again"This book teaches application-level programming with Python. That is, it is about what you can do with the language once you've mastered its fundamentals. By reading this book, you will learn to use Python in some of its most common roles: to build GUIs, web sites, networked tools, scripting interfaces, system administration programs, database and text processing utilities, and more. Along the way, you will also learn how to use the Python language in realistically scaled programsconcepts such as object-oriented programming (OOP) and code reuse are recurring side themes throughout this text. And you will gain enough information to further explore the application domains introduced in the book, as well as to explore others. About This BookNow that I've told you what this book is, I should tell you what it is not. First of all, this book is not a reference manual. Although the index can be used to hunt for information, this text is not a dry collection of facts; it is designed to be read. And while many larger examples are presented along the way, this book is also not just a collection of minimally documented code samples. Rather, this book is a tutorial that teaches the most common Python application domains from the ground up. It covers each of Python's target domains gradually, beginning with in-depth discussions of core concepts in each domain, before progressing toward complete programs. Large examples do appear, but only after you've learned enough to understand their techniques and code. For example, network scripting begins with coverage of network basics and protocols and progresses through sockets, client-side tools, HTML and CGI fundamentals, and web frameworks. GUI programming gets a similarly gentle presentation, with one introductory and two tutorial chapters, before reaching larger, complete programs. And system interfaces are explored carefully before being applied in real and useful scripts. In a sense, this book is to application-level programming what the book Learning Python is to the core Python languagea learning resource that makes no assumptions about your prior experience in the domains it covers. Because of this focus, this book is designed to be a natural follow-up to the core language material in Learning Python and a next step on the way to mastering the many facets of Python programming. In deference to all the topic suggestions I have received over the years, I should also point out that this book is not intended to be an in-depth look at specific systems or tools. With perhaps one million Python users in the world today, it would be impossible to cover in a useful way every Python-related system that is of interest to users. Instead, this book is designed as a tutorial for readers new to the application domains covered. The web chapters, for instance, focus on core web scripting ideas, such as server-side scripts and state retention options, not on specific systems, such as SOAP, Twisted, and Plone. By reading this book, you will gain the groundwork necessary to move on to more specific tools such as these in the domains that interest you. About This EditionTo some extent, this edition's structure is a result of this book's history. The first edition of this book, written in 1995 and 1996, was the first book project to present the Python language. Its focus was broad. It covered the core Python language, and it briefly introduced selected application domains. Over time, the core language and reference material in the first edition evolved into more focused books Learning Python and Python Pocket Reference. Given that evolution, the second edition of this book, written from 1999 to 2000, was an almost completely new book on advanced Python topics. Its content was an expanded and more complete version of the first edition's application domain material, designed to be an application-level follow-up to the core language material in Learning Python, and supplemented by the reference material in Python Pocket Reference. The second edition focused on application libraries and tools rather than on the Python language itself, and it was oriented toward the practical needs of real developers and real tasksGUIs, web sites, databases, text processing, and so on. This third edition, which I wrote in 2005 and 2006, is exactly like the second in its scope and focus, but it has been updated to reflect Python version 2.4, and to be compatible with the upcoming Python 2.5. It is a minor update, and it retains the second edition's design and scope as well as much of its original material. However, its code and descriptions have been updated to incorporate both recent changes in the Python language, as well as current best practices in Python programming. Python ChangesYou'll find that new language features such as string methods, enclosing-function scope references, list comprehensions, and new standard library tools, such as the email package, have been integrated throughout this edition. Smaller code changesfor instance, replacing apply calls and exc_type usage with the newer func(*args) and exc_info( )have been applied globally as well (and show up surprisingly often, because this book is concerned with building general tools). All string-based, user-defined exceptions are now class-based, too; string exceptions appeared half a dozen times in the book's examples, but are documented as deprecated today. This is usually just a matter of changing to class MyExc(Exception): pass, though, in one case, exception constructor arguments must be extracted manually with the instance's args attribute. 'X' also became repr(X) across all examples, and I've replaced some appearances of while 1: with the newer and more mnemonic while True:, though either form works as advertised and C programmers often find the former a natural pattern. Hopefully, these changes will future-proof the examples for as long as possible; be sure to watch the updates page described later for future Python changes. One futurisms note: some purists might notice that I have not made all classes in this book derive from object to turn on new-style class features (e.g., class MyClass(object)). This is partly because the programs here don't employ the new-style model's slightly modified search pattern or advanced extensions. This is also because Python's creator, Guido van Rossum, told me that he believes this derivation will not be required in Python 3.0standalone classes will simply be new-style too, automatically (in fact, the new-style class distinction is really just a temporary regression due to its incompatible search order in particular rare, multiple-inheritance trees). This is impossible to predict with certainty, of course, and Python 3.0 might abandon compatibility in other ways that break some examples in this book. Be sure to both watch for 3.0 release notes and keep an eye on this book's updates page over time. Example ChangesYou'll also notice that many of the second edition's larger examples have been upgraded substantially, especially the two larger GUI and CGI email-based examples (which are arguably the implicit goals of much of the book). For instance:
In addition to the enhanced and new, major examples, you'll also find many other examples that demonstrate new and advanced topics such as thread queues. Topic ChangesIn addition to example changes, new topics have been added throughout. Among these are the following:
Beyond these specific kinds of changes, some material has been reorganized to simplify the overall structure. For example, a few chapters have been split up to make them less challenging; appendixes have been removed to save space (references are available separately); and the PyErrata web site example chapter has been removed (it didn't present many new concepts, so we've made it and its code available in the book's examples distribution as optional reading). You'll also find a new "Sneak Preview" chapter for readers in a hurrya throwback to the first edition. This chapter takes a single example from command line to GUI to web site, and introduces Python and its libraries along the way. Focus UnchangedFundamentally, though, this edition, like the second, is still focused on ways to use Python rather than on the language itself. Python development concepts are explored along the wayin fact, they really become meaningful only in the context of larger examples like those in this edition. Code structure and reuse, for instance, are put into practice by refactoring and reusing examples throughout the book. But in general, this text assumes that you already have at least a passing acquaintance with Python language fundamentals, and it moves on to present the rest of the Python storyits application to real tasks. If you find code in this book confusing, I encourage you to read Learning Python as a prelude to this text. In the remainder of this preface, I'll explain some of the rationales for this design, describe the structure of this edition in more detail, and give a brief overview of how to use the Python programs shipped in the book examples package. This Book's MotivationOver the 10 years since the first edition of this book was written, Python has transitioned from an emerging language that was of interest primarily to pioneers to a widely accepted tool used by programmers for day-to-day development tasks. Along the way, the Python audience has changed as well, and this book has been refocused with this new readership in mind. You will find that it is a nuts-and-bolts text, geared less toward introducing and popularizing the language and more toward showing you how to apply Python for realistically scaled programming tasks.
Since writing the first edition, I have also had the opportunity to teach Python classes in the U.S. and abroad some 170 times as of mid-2006, and this book reflects feedback garnered from these training sessions. The application domain examples, for instance, reflect interests and queries common among the thousands of students I have introduced to Python. Teaching Python to workers in the trenches, many of whom are now compelled to use Python on the job, also inspired a new level of practicality that you will notice in this book's examples and topics. Other book examples are simply the result of me having fun programming Python. Yes, fun. One of the most common remarks I hear from Python newcomers is that Python is actually enjoyable to useit is able to both kindle the excitement of programming among beginners and rekindle that excitement among those who have toiled for years with more demanding tools. When you can code as fast as you can think, programming becomes a very different proposition and feels more like pleasure than work. As you will see in this book, Python makes it easy to play with advanced but practical tools such as threads, sockets, GUIs, web sites, and OOPareas that can be both tedious and daunting in traditional languages such as C and C++. It enables things you may not have considered or attempted with other tools. Frankly, even after 14 years as a bona fide Pythonista, I still find programming most enjoyable when I do it in Python. Python is a remarkably productive and flexible language, and witnessing its application firsthand is an aesthetic pleasure. I hope this edition, as much as the two before it, will both demonstrate how to reap Python's productivity benefits and communicate some of the excitement to be found in this rewarding tool. This Book's StructureAlthough code examples are reused throughout the book and later chapters build upon material in earlier chapters (e.g., GUIs), topics in this book are covered fairly independently and are grouped together in different parts. If a particular domain's part doesn't interest you, you can generally skip ahead to a part that does. As a result, it's not too much of a stretch to consider this edition as akin to four or five books in one. Its top-level structure underscores its application-topics focus (see the Table of Contents for a more fine-grained look at the book's structure):
Two notes about the structure: first of all, don't let these titles fool youalthough most have to do with application topics, Python language features and general design concepts are still explored along the way, in the context of real-world goals. Secondly, readers who use Python as a standalone tool can safely skip the integration part, though I still recommend a quick glance. C programming isn't nearly as fun or as easy as Python programming is. Yet, because integration is central to Python's role as a scripting tool, a cursory understanding can be useful, regardless of whether you do integrating, scripting, or both. This Edition's DesignThe best way to get a feel for any book is to read it, of course. But especially for people who are familiar with the prior edition, this section will clarify regarding what is new this time around. It's Been Updated for Python 2.4 (and 2.5)All of the example code has been upgraded to use the latest features of the Python language and its standard library. Python is still largely compatible with the code in the first two editions, but recent language additions such as nested scopes and list comprehensions simplify many coding tasks. For instance, default arguments are no longer required to pass objects into most lambda expressions, and the new email package greatly simplifies the tasks of parsing and adding email attachments. See the Python changes list earlier in this chapter for more on this subject. Although the GUI examples in this book required almost no code changes, they have been updated to run on Tk 8.4, the library used by Python 2.4 as its standard portable GUI toolkit. Among other things, the latest Tk allows window icons to be set by the program. Although begun under 2.4, this edition is also compatible with the upcoming Python 2.5 release. It's Been ReorganizedA few chapters have been moved to make the flow more logical; for example, the sections on files and directories and the PyMailGUI example are now in chapters of their own. In addition, all appendixes were cut (this book is neither a reference nor a Python changes log), and a new initial preview chapter was added to introduce topics explored throughout the book. As mentioned earlier, in deference to space, one second-edition chapterthat on the PyErrata web sitehas been cut in this edition. Its main, unique topics on state retention have been incorporated into other chapters. The original source code for the PyErrata site still appears on the book's examples package, as supplemental reading.[*]
It Covers New TopicsYou'll find much-expanded coverage of Zope, the ZODB database, threading tools and techniques including the queue module, SQL interfaces, XML parsing, and more. See the example and topic changes lists provided earlier for additional details. Most of the new or expanded topics are a result of the evolution of common practice in the Python world. While this book doesn't address core language evolution directly (the basics of new language tools such as list comprehensions are the domain of the text Learning Python), it does employ it throughout its examples. It's Still Mostly Platform-NeutralExcept for some C integration examples, the majority of the programs in this edition were developed on Windows XP computers, with an eye toward portability to Linux and other platforms. In fact, some of the examples were born of my desire to provide portable Python equivalents of tools missing on Windows (e.g., file splitters). When programs are shown in action, it's usually on Windows; they are demonstrated on the Linux platform only if they exercise Unix-specific interfaces. This is not meant as a political statement; it is mostly a function of the fact that I wrote this book with Microsoft Word. When time is tight, it's more convenient to run scripts on the same platform as your publishing tools than to frequently switch platforms. Luckily, because Python has become so portable, the underlying operating system is largely irrelevant to developers. Python, its libraries, and its Tkinter GUI framework work extremely well on all major platforms today. Where platform issues do come into play, though, I've made the examples as platform-neutral as possible, and I point out platform-specific issues along the way. Generally speaking, most of the scripts should work unchanged on common Python platforms. For instance, all the GUI examples were tested on both Windows (ME, XP) and Linux (KDE, Gnome), and most of the command-line and thread examples were developed on Windows but work on Linux too. Because Python's system interfaces are built to be portable, this is easier than it may sound; it's largely automatic. On the other hand, this book does delve into platform-specific topics where appropriate. For instance, there is coverage of many Windows-specific topicsActive Scripting, COM, program launch options, and so on. Linux and Unix readers will also find material geared toward their platformsforks, pipes, and the like. C integration code platform issuesThe one place where readers may still catch a glimpse of platform biases is in the Python/C integration examples. For simplicity, the C compilation details covered in this text are still somewhat Unix/Linux-oriented. One can make a reasonable case for such a focusnot only does Linux come with C compilers, but the Unix development environment it provides grew up around that language. On standard Windows, the C code shown in this book will work, but you may need to use different build procedures (they vary per Windows compiler, some of which are very similar to Linux compilers). In fact, for this third edition of the book, many of the C integration examples were run on the Cygwin system, not on Linux. Cygwin provides a complete, Unix-like environment and library for Windows. It includes C development tools, command-line utilities, and a version of Python that supports Unix tools not present in the standard Windows Python, including process forks and fifos. Unlike Linux, because it runs on Windows, Cygwin does not require a complete operating system installation (see http://www.cygwin.com). Cygwin has a GPL-style, open source license that requires giving away code (more on this later in the book). If you do not wish to download and install Cygwin, you may have to translate some of the C integration build files for your platform; the standard C development concepts apply. On standard Windows, you'll have to translate for your C compiler. O'Reilly has published an outstanding text, Python Programming on Win32, that covers Windows-specific Python topics like this, and it should help address any disparity you may find here. It's Still Focused for a More Advanced AudienceBecoming proficient in Python involves two distinct tasks: learning the core language itself, and then learning how to apply it in applications. This book addresses the latter (and larger) of these tasks by presenting Python libraries, tools, and programming techniques. Learning Python syntax and datatypes is an important first step, and a prerequisite to this book. Very soon after you've learned how to slice a list, though, you'll find yourself wanting to do real things, like writing scripts to compare file directories, responding to user requests on the Internet, displaying images in a window, reading email, and so on. Most of the day-to-day action is in applying the language, not the language itself. That's what this book is for. It covers libraries and tools beyond the core language, which become paramount when you begin writing real applications. It also addresses larger software design issues such as reusability and OOP, which can be illustrated only in the context of realistically scaled programs. Because it assumes you already know Python, this is a somewhat advanced text; again, if you find yourself lost, you might do well to learn the core language from other resources before returning here. It's Still Example-OrientedAlthough this book teaches concepts before applying them, it still contains many larger working programs that tie together concepts presented earlier in the book and demonstrate how to use Python for realistically scaled tasks. Among them:
See the earlier example changes list for more about how some of these have mutated in this edition. Besides the major examples listed here, there are also mixed-mode C integration examples (e.g., callback registration and class object processing); SWIG examples (with and without "shadow" classes for C++); more Internet examples (FTP upload and download scripts, NNTP and HTTP examples, email tools, and socket and select module examples); many examples of Python threads and thread queues; and coverage of Jython, HTMLgen, Zope, COM, XML parsing, and Python ZODB and MySQL database interfaces. In addition, as mentioned earlier, the second edition's PyErrata web site example appears in the examples distribution.
Using the Book's ExamplesBecause examples are central to the structure of this book, I want to briefly describe how to use them here. In general, though, see the following text files in the examples directory for more details:
Of these, the README-PP3E.txt file is the most informative. In addition, the PP3E\Config directory contains low-level configuration file examples for Windows and Linux, which may or may not be applicable to your usage. I give an overview of some setup details here, but the preceding files give the complete description. The Book Examples TreeIn a sense, the directory containing the book's examples is itself a fairly sophisticated Python software system and the examples within it have been upgraded structurally in a number of important ways:
The Book Examples Distribution PackageYou can find the book examples distribution package on the book's web page at O'Reilly's web site, http://www.oreilly.com/catalog/python3/. The book examples directory is located in the PP3E subdirectory of the topmost Examples directory in the packagethat is, Examples\PP3E on Windows and Examples/PP3E on Linux. If you've copied the examples to your machine, the examples directory is wherever you copied the PP3E root directory. Example titles reflect this tree's structure. For instance, an example title of PP3E\Preview\mod.py refers to the Examples\PP3E\Preview\mod.py file at the top level of the book examples distribution package. You can run most of the examples from within the package directly, but if you obtained them on a CD, you'll want to copy them to a writable medium such as your hard drive to make changes, and to allow Python to save .pyc compiled bytecode files for quicker startups. See the example package's top-level README file for more details, or browse the examples directory in your favorite file explorer for a quick tour. Depending on how often the book's distribution package is maintained, it may also contain extra open source packages such as the latest releases of Python, the SWIG code generator, and Windows extensions, but you can always find up-to-date releases of Python and other packages on the Web (see Python's web site, http://www.python.org, or search the Web). In fact, you shouldmost likely, the Web will very quickly become more current than any extra software included in the book's package. Running Examples: The Short StoryNow the fun stuffif you want to see some Python examples right away, do this:
The Launch_* scripts start Python programs portably[*] and require only that Python be installedyou don't need to set any environment variables first to run them. LaunchBrowser will work if it can find a web browser on your machine even if you don't have an Internet link (though some Internet examples won't work completely without a live link).
The demo launchers also include a number of web-based programs that use a web browser for their interface. When run, these programs launch a locally running web server coded in Python (we'll meet this server script later in this book). Although these programs can run on a remote server too, they still require a local Python installation to be used with a server running on your machine. Running Examples: The DetailsThis section goes into a few additional details about running the book's example programs. If you're in a hurry, feel free to skip this and run the programs yourself now. Demos and gadgetsTo help organize the book's examples, I've provided a demo launcher program GUI, PyDemos2.pyw, in the top-level PP3E directory of the examples distribution. Figure P-1 shows PyDemos in action on Windows after pressing a few buttons. We'll meet in this text all the programs shown in the figure. The launcher bar itself appears on the top right of the screen; with it, you can run most of the major graphical examples in the book with a mouse click, and view their source code in pop-up windows. The demo launcher bar can also be used to start major Internet book examples if a browser can be located on your machine and a Python-coded server can be started. Figure P-1. The PyDemos launcher with gadgets and demos Besides launching demos, the PyDemos source code provides pointers to major examples in the examples tree; see its code for details. You'll also find automated build scripts for the Python/C integration examples in the Integration examples directory, which serve as indexes to major C examples. I've also included a top-level program called PyGadgets.py, and its relative, PyGadgets_bar.pyw, to launch some of the more useful GUI book examples for real use instead of demonstration (mostly, the programs I use; configure as desired). Run PyGadgets_bar to see how it looksit's a simple row of buttons that pop up many of the same programs shown in Figure P-1, but for everyday use, not for demonstrations. All of its programs are presented in this book as well and are included in the examples distribution package. See the end of Chapter 10 for more on PyDemos and PyGadgets. Setup requirementsMost demos and gadgets require a Python with Tkinter GUI support, but that is the default configuration for Python out-of-the-box on Windows. As a result, most of the demos and gadgets should "just work" on Windows. On some other platforms, you may need to install or enable Tkinter for your Python; try it and seeif you get an error about Tkinter not being found, you'll need to configure it. If it's not already present, Tkinter support can be had freely on the Web for all major platforms (more on this in the GUI part of this book, but search the Web with Google for quick pointers). Two external dependency notes: PyPhoto will not run without PIL, and PyMailCGI runs without PyCrypto but uses it if installed. Both PIL and PyCrypto are open source third-party extension packages, but must be installed in addition to Python. Some book examples use additional third-party tools (for instance, ZODB and MySQL in the database chapter), but these are not launched from the demos and gadgets interfaces. To run the files listed in the preceding section directly, you'll also need to set up your Python module search path, typically with your PYTHONPATH environment variable or a .pth file. The book examples tree ships as a simple directory and does not use Python's Distutils scripts to install itself in your Python's site packages directory (this system works well for packed software, but can add extra steps for viewing book examples). If you want to run a collection of Python demos from the book right away, though, and you don't want to bother with setting up your environment first, simply run these self-launching utility scripts in the PP3E directory instead:
These Python-coded launcher scripts assume Python has already been installed, but will automatically find your Python executable and the book examples distribution and set up your Python module and system search paths as needed to run their programs. You can probably run these launcher scripts by simply clicking on their names in a file explorer, and you should be able to run them directly from the book's examples package tree (you can read more about these scripts in Part II of the book). Web-based examplesBeginning with this edition of the book, its browser-based Internet examples are not installed on a remote server. Instead, we'll be using a Python-coded web server running locally to test these examples. If you launch this server, though, you can also test-drive browser-based examples too. You can find more on this in the Internet section of this book. For a quick look, though, PyDemos attempts to launch both a web server and a web browser on your machine automatically for the major example web pages. You start the browser by running the LaunchBrowser.py script in the examples root directory. That script tries to find a usable browser on your machine, with generally good results; see the script for more details if it fails. The server is implemented by a Python script, assuming you have permission to run an HTTP server on your machine (you generally do on Windows). Provided the server starts and LaunchBrowser can find a browser on your machine, some demo buttons will pop up web pages automatically. Figure P-2, for example, shows the web examples index page running under a local server and the Firefox browser. Figure P-2. The PyInternetDemos web page Clicking this page's links runs various server-side Python CGI scripts presented in the book. Of special interest, the getfile.html link on this page allows you to view the source code of any other file in the book's web server directoryHTML code, Python CGI scripts, and so on; see Chapter 16 for details. Top-level programsTo summarize, here is what you'll find in the top-level Examples\PP3E directory of the book's examples package:
You'll also find subdirectories for examples from each major topic area of the book. In addition, the top-level PP3E\PyTools directory contains Python-coded command-line utilities for converting line feeds in all example text files to DOS or Unix format (useful if they look odd in your text editor); making all example files writable (useful if you drag-and-drop off a CD on some platforms); deleting old .pyc bytecode files in the tree; and more. Again, see the example directory's README-PP3E.txt file for more details on all example issues. Conventions Used in This BookThe following font conventions are used in this book:
Safari® Enabled
Safari offers a solution that's better than e-books. It's a virtual library that lets you easily search thousands of top tech books, cut and paste code samples, download chapters, and find quick answers when you need the most accurate, current information. Try it for free at http://safari.oreilly.com. Where to Look for UpdatesAs before, updates, corrections, and supplements for this book will be maintained at the author's web site, http://www.rmi.net/~lutz. Look for the third edition's link on that page for all supplemental information related to this version of the book. As for the first two editions, I will also be maintaining a log on this web site of Python changes over time, which you should consider a supplemental appendix to this text. O'Reilly's web site, http://www.oreilly.com, also has an errata report system, and you should consider the union of these two lists to be the official word on book bugs and updates. Contacting O'ReillyYou can also address comments and questions about this book to the publisher:
O'Reilly has a web page for this book, which lists errata, examples, and any additional information. You can access this page at:
To comment or ask technical questions about this book, send email to:
For more information about books, conferences, software, Resource Centers, and the O'Reilly Network, see the O'Reilly web site at:
Using Code ExamplesThis book is here to help you get your job done. In general, you may use the code in this book in your programs and documentation. You do not need to contact us for permission unless you're reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing a CD-ROM of examples from O'Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a significant amount of example code from this book into your product's documentation does require permission. We appreciate, but do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: "Python Programming, Third Edition, by Mark Lutz. Copyright 2006 O'Reilly Media, Inc., 978-0-596-00925-0." AcknowledgmentsIn closing, I would like to extend appreciation to a few of the people who helped in some way during all the incarnations of this book project:
And finally, a few personal notes of thanks. To all the friends I've met on the training trail, for hospitality. To my mom, for wisdom and courage. To OQO, for toys. To my brothers and sister, for old days. To Guinness, for the beer in Dublin. To Birgit, for inspiration and spleenwurst. And to my children, Michael, Samantha, and Roxanne, for hope.
|
When you see a Safari® Enabled icon on the cover of your favorite technology book, that means the book is available online through the O'Reilly Network Safari Bookshelf.