Section 2.8. Step 6: Adding a Web Interface
2.8. Step 6: Adding a Web InterfaceGUI interfaces are easier to use than command lines and are often all we need to simplify access to data. By making our database available on the Web, we can open it up to even wider use. Anyone with Internet access and a web browser can access the data, regardless of where they are located and which machine they are using. Anything from workstations to cell phones will suffice. Moreover, web-based interfaces require only a web browser; there is no need to install Python to access the data except on the single-server machine. Although web-based approaches may sacrifice some of the utility and speed of in-process GUI toolkits, their portability gain can be compelling. As we'll also see later in this book, there are a variety of ways to go about scripting interactive web pages of the sort we'll need in order to access our data. Basic CGI scripting is more than adequate for simple tasks like ours. For more advanced applications, toolkits and frameworks such as Zope, Plone, Twisted, CherryPy, Webware, Django, TurboGears, mod_python, and Quixote can provide tools that we would otherwise need to code from scratch. Zope, for instance, simplifies many CGI scripting tasks and provides for security, load balancing on the server, and more. For now, let's keep things simple and code a CGI script. 2.8.1. CGI BasicsCGI scripting in Python is easy as long as you already have a handle on things like HTML forms, URLs, and the client/server model of the Web (all topics we'll address in detail later in this book). Whether you're aware of all the underlying details or not, the basic interaction model is probably familiar. In a nutshell, a user visits a web site and receives a form, coded in HTML, to be filled out in her browser. After submitting the form, a script, identified within either the form or the address used to contact the server, is run on the server and produces another HTML page as a reply. Along the way, data typically passes through three programs: from the client browser, to the web server, to the CGI script, and back again to the browser. This is a natural model for the database access interaction we're afterusers can submit a database key to the server and receive the corresponding record as a reply page. We'll go into CGI basics in depth later in this book, but as a first example, let's start out with a simple interactive web page that requests, and then echoes back a user's name in a web browser. The first page in this interaction is just an input form produced by the HTML file shown in Example 2-30. This HTML file is stored on the web server machine and is transferred to the web browser when accessed. Example 2-30. PP3E\Preview\cgi101.html
Notice how this HTML form names the script that will process its input on the server in its action attribute. The input form that this code produces is shown in Figure 2-10 (shown in the open source Firefox web browser running on Windows). Figure 2-10. cgi101.html input form page After the input form is submitted, the script in Example 2-31 is run on the web server machine to handle the inputs and generate a reply to the browser on the client machine. Like the HTML file, this Python script resides on the same machine as the web server; it uses the cgi module to parse the form's input and insert it into the HTML reply stream, properly escaped. The cgi module gives us a dictionary-like interface to form inputs sent by the browser, and the HTML code that this script prints winds up rendering the next page on the client's browser. In the CGI world, the standard output stream is connected to the client through a socket. Example 2-31. PP3E\Preview\cgi-bin\cgi101.py
And if all goes well, we receive the reply page shown in Figure 2-11essentially, just an echo of the data we entered in the input page. The page in this figure is produced by the HTML printed by the Python CGI script running on the server. Along the way, the user's name was transferred from a client to a server and back againpotentially across networks and miles. This isn't much of a web site, of course, but the basic principles here apply, whether you're echoing inputs or doing full-blown e-whatever. Figure 2-11. cgi101.py script reply page for input form If you have trouble getting this interaction to run on Unix-like systems, you may need to modify the path to your Python in the #! line at the top of the script file and make it executable with a chmod command, but this is dependent on your web server (more on servers in the next section). Also note that the CGI script in Example 2-31 isn't printing complete HTML: the <html> and <body> tags of the static HTML file in Example 2-30 are missing. Strictly speaking, such tags should be printed, but web browsers don't mind the omissions, and this book's goal is not to teach legalistic HTML; see other resources for more on HTML. Before moving on, it's worth taking a moment to compare this basic CGI example with the simple GUI of Example 2-28 and Figure 2-6. Here, we're running scripts on a server to generate HTML that is rendered in a web browser. In the GUI, we make calls to build the display and respond to events within a single process and on a single machine. The GUI runs multiple layers of software, but not multiple programs. By contrast, the CGI approach is much more distributedthe server, the browser, and possibly the CGI script itself run as separate programs that usually communicate over a network. Because of such differences, the GUI model may be simpler and more direct: there is no intermediate server, replies do not require invoking a new program, no HTML needs to be generated, and the full power of a GUI toolkit is at our disposal. On the other hand, a web-based interface can be viewed in any browser on any computer and only requires Python on the server machine. And just to muddle the waters further, a GUI can also employ Python's standard library networking tools to fetch and display data from a remote server (that's how web browsers do their work). We'll revisit the trade-offs of the GUI and CGI schemes later in this book. First, let's preview a handful of pragmatic issues related to CGI work before we apply it to our people database. 2.8.2. Running a Web ServerTo run CGI scripts at all, we need a web server that will serve up our HTML and launch our Python scripts on request. The server is a required mediator between the browser and the CGI script. If you don't have an account on a machine that has such a server available, you'll want to run one of your own. We could configure and run a full-blown web server such as the open source Apache system (which, by the way, can be tailored with Python-specific support by the mod_python extension). For this chapter, however, I instead wrote a simple web server in Python using the code in Example 2-32. We'll revisit the tools used in this example later in this book. In short, because Python provides precoded support for various types of network servers, we can build a CGI-capable and portable HTTP web server in roughly 20 lines of code (including comments, whitespace, and a workaround added to force the CGI script to run in-process because of a Windows problem I ran into on two of my test machinesmore on this later). As we'll see later in this book, it's also easy to build proprietary network servers with low-level socket calls in Python, but the standard library provides canned implementations for many common server types, web based or otherwise. The SocketServer module, for instance, provides threaded and forking versions of TCP and UDP servers. Third-party systems such as Twisted provide even more implementations. For serving up web content, the standard library modules used in Example 2-32 provide what we need. Example 2-32. PP3E\Preview\webserver.py
The classes this script uses assume that the HTML files to be served up reside in the current working directory and that the CGI scripts to be run live in a /cgi-bin or /htbin subdirectory there. We're using a /cgi-bin subdirectory for scripts, as suggested by the filename of Example 2-31. Some web servers look at filename extensions to detect CGI scripts; our script uses this subdirectory-based scheme instead. To launch the server, simply run this script (in a console window, by an icon click, or otherwise); it runs perpetually, waiting for requests to be submitted from browsers and other clients. The server listens for requests on the machine on which it runs and on the standard HTTP port number 80. To use this script to serve up other web sites, either launch it from the directory that contains your HTML files and a cgi-bin subdirectory that contains your CGI scripts, or change its webdir variable to reflect the site's root directory (it will automatically change to that directory and serve files located there). But where in cyberspace do you actually run the server script? If you look closely enough, you'll notice that the server name in the addresses of the prior section's examples (near the top right of the browser after the "http://") is always localhost. To keep this simple, I am running the web server on the same machine as the web browser; that's what the server name "localhost" (and the equivalent IP address "127.0.0.1") means. That is, the client and server machines are the same: the client (web browser) and server (web server) are just different processes running at the same time on the same computer. This turns out to be a great way to test CGI scriptsyou can develop them on the same machine without having to transfer code back to a remote server machine after each change. Simply run this script from the directory that contains both your HTML files and a cgi-bin subdirectory for scripts and then use "http://localhost/..." in your browser to access your HTML and script files. Here is the trace output the web server script produces in a Windows console window that is running on the same machine as the web browser and launched from the directory where the HTML files reside: ...\PP3E\Preview> python webserver.py localhost - - [17/Jan/2005 14:30:44] "GET /cgi101.html HTTP/1.1" 200 - localhost - - [17/Jan/2005 14:30:45] code 404, message File not found localhost - - [17/Jan/2005 14:30:45] "GET /favicon.ico HTTP/1.1" 404 - localhost - - [17/Jan/2005 14:31:30] "POST /cgi-bin/cgi101.py HTTP/1.1" 200 - localhost - - [17/Jan/2005 14:31:30] CGI script exited OK localhost - - [17/Jan/2005 14:31:31] code 404, message File not found localhost - - [17/Jan/2005 14:31:31] "GET /favicon.ico HTTP/1.1" 404 - localhost - - [17/Jan/2005 14:32:31] "GET /cgi-bin/cgi101.py?name=Sue+Smith HTTP /1.1" 200 - localhost - - [17/Jan/2005 14:32:31] CGI script exited OK To run this server on a different port, change the port number in the script and name it explicitly in the URL (e.g., "http://localhost:8888/"). To run this server on a remote computer, upload the HTML files and CGI scripts' subdirectory to the remote computer, launch the server script on that machine, and replace "localhost" in the URLs with the domain name or IP address of your server machine (e.g., "http://www.myserver.com/"). When running the server remotely, all the interaction will be as shown here, but inputs and replies will be automatically shipped across network connections, not routed between programs running in the same computer. On systems that don't require custom code like the Windows workaround in our code, you can also start a CGI-capable web server by simply running the file CGIHTTPServer.py in the Python standard library (this script is located in the C:\Python24\Lib directory on Windows, for instance, under Python 2.4). This file's test code is similar to our script, but it defaults to port number 8000 unless a port number is given on the command line as an argument. In Chapter 16, we'll expand Example 2-32 to allow the directory name and port numbers to be passed in on the command line, and we'll augment the module search path for platforms where the server runs the script in-process.[*]
2.8.3. Using Query Strings and urllibIn the basic CGI example shown earlier, we ran the Python script by filling out and submitting a form that contained the name of the script. Really, CGI scripts can be invoked in a variety of wayseither by submitting an input form as shown so far, or by sending the server an explicit URL (Internet address) string that contains inputs at the end. Such an explicit URL can be sent to a server either in or outside of a browser; in a sense, it bypasses the traditional input form page. For instance, Figure 2-12 shows the reply generated by the server after typing a URL of the following form in the address field at the top of the web browser (+ means a space here): http://localhost/cgi-bin/cgi101.py?user=Sue+Smith Figure 2-12. cgi101.py reply to GET-style query parameters The inputs here, known as query parameters, show up at the end of the URL after the ?; they are not entered into a form's input fields. Adding inputs to URLs is sometimes called a GET request. Our original input form uses the POST method, which instead ships inputs in a separate step. Luckily, Python CGI scripts don't have to distinguish between the two; the cgi module's input parser handles any data submission method differences for us. It's even possible, and often useful, to submit URLs with inputs appended as query parameters completely outside any web browser. The Python urllib module, for instance, allows us to read the reply generated by a server for any valid URL. In effect, it allows us to visit a web page or invoke a CGI script from within another script; your Python code acts as the web client. Here is this module in action, run from the interactive command line: >>> from urllib import urlopen >>> conn = urlopen('http://localhost/cgi-bin/cgi101.py?user=Sue+Smith') >>> reply = conn.read( ) >>> reply '<title>Reply Page</title>\n<h1>Hello <i>Sue Smith</i>!</h1>\n' >>> urlopen('http://localhost/cgi-bin/cgi101.py').read( ) '<title>Reply Page</title>\n<h1>Who are you?</h1>\n' >>> urlopen('http://localhost/cgi-bin/cgi101.py?user=Bob').read( ) '<title>Reply Page</title>\n<h1>Hello <i>Bob</i>!</h1>\n' The urllib module gives us a file-like interface to the server's reply for a URL. Notice that the output we read from the server is raw HTML code (normally rendered by a browser). We can process this text with any of Python's text-processing tools, including string methods to search and split, the re regular expression pattern-matching module, or the full-blown HTML parsing support in the standard library. When combined with such tools, the urllib module is a natural for interactive testing and custom client-side GUIs, as well as implementing automated tools such as regression testing systems for remote server-side CGI scripts. 2.8.4. Formatting Reply TextOne last fine point: because CGI scripts use text to communicate with clients, they need to format their replies according to a set of rules. For instance, notice how Example 2-31 adds a blank line between the reply's header and its HTML by printing an explicit newline (\n) in addition to the one print adds automatically; this is a required separator. Also note how the text inserted into the HTML reply is run through the cgi.escape call, just in case the input includes a character that is special in HTML. For example, Figure 2-13 shows the reply we receive on another machine for form input Bob </i> Smiththe </i> in the middle becomes </i> in the reply, and so doesn't interfere with real HTML code (if not escaped, the rest of the name would not be italicized). Figure 2-13. Escaping HTML characters Escaping text like this isn't always required, but it is a good rule of thumb when its content isn't known; scripts that generate HTML have to respect its rules. As we'll see later in this book, a related call, urllib.quote, applies URL escaping rules to text. As we'll also see, larger frameworks such as Zope often handle text formatting tasks for us. 2.8.5. A Web-Based Shelve InterfaceNow, to use the CGI techniques of the prior sections for our database application, we basically just need a bigger input and reply form. Figure 2-14 shows the form we'll implement for accessing our database in a web browser. Figure 2-14. peoplecgi.html input page 2.8.5.1. Coding the web siteTo implement the interaction, we'll code an initial HTML input form, as well as a Python CGI script for displaying fetch results and processing update requests. Example 2-33 shows the input form's HTML code that builds the page in Figure 2-14. Example 2-33. PP3E\Preview\peoplecgi.html
To handle form (and other) requests, Example 2-34 implements a Python CGI script that fetches and updates our shelve's records. It echoes back a page similar to that produced by Example 2-33, but with the form fields filled in from the attributes of actual class objects in the shelve database. As in the GUI, the same web page is used for both displaying results and inputting updates. Unlike the GUI, this script is run anew for each step of user interaction, and it reopens the database each time (the reply page's action field is a link back to the script). The basic CGI model provides no automatic memory from page to page. Example 2-34. PP3E\Preview\cgi-bin\peoplecgi.py
This is a fairly large script, because it has to handle user inputs, interface with the database, and generate HTML for the reply page. Its behavior is fairly straightforward, though, and similar to the GUI of the prior section. The only feat of semimagic it relies on is using a record's attribute dictionary (_ _dict_ _) as the source of values when applying string formatting to the HTML reply template string in the last line of the script. Recall that a %(key)code replacement target fetches a value by key from a dictionary: >>> D = {'say': 5, 'get': 'shrubbery'} >>> D['say'] 5 >>> S = '%(say)s => %(get)s' % D >>> S '5 => shrubbery' By using an object's attribute dictionary, we can refer to attributes by name in the format string. In fact, part of the reply template is generated by code. If its structure is confusing, simply insert statements to print replyhtml and to call sys.exit, and run from a simple command line. This is how the table's HTML in the middle of the reply is generated (slightly formatted here for readability): <table> <tr><th>key<td><input type=text name=key value="%(key)s"> <tr><th>name<td><input type=text name=name value="%(name)s"> <tr><th>age<td><input type=text name=age value="%(age)s"> <tr><th>job<td><input type=text name=job value="%(job)s"> <tr><th>pay<td><input type=text name=pay value="%(pay)s"> </table> This text is then filled in with key values from the record's attribute dictionary by string formatting at the end of the script. This is done after running the dictionary through a utility to convert its values to code text with repr and escape that text per HTML conventions with cgi.escape (again, the last step isn't always required, but it's generally a good practice). These HTML reply lines could have been hardcoded in the script, but generating them from a tuple of field names is a more general approachwe can add new fields in the future without having to update the HTML template each time. Python's string processing tools make this a snap. 2.8.5.2. Using the web siteUsing the web interface is as simple as using the GUI. To fetch a record, fill in the Key field and click Fetch; the script populates the page with field data grabbed from the corresponding class instance in the shelve, as illustrated in Figure 2-15 for the key bob. Figure 2-15. peoplecgi.py reply page Figure 2-15 shows what happens when the key comes from the posted form. As usual, you can also invoke the CGI script by instead passing inputs on a query string at the end of the URL; Figure 2-16 shows the reply we get when accessing a URL of the following form: http://localhost/cgi-bin/peoplecgi.py?action=Fetch&key=sue Figure 2-16. peoplecgi.py reply for query parameters As we've seen, such a URL can be submitted either within your browser, or by scripts that use tools such as the urllib module. Again, replace "localhost" with your server's domain name if you are running the script on a remote machine. To update a record, fetch it by key, enter new values in the field inputs, and click Update; the script will take the input fields and store them in the attributes of the class instance in the shelve. Figure 2-17 shows the reply we get after updating sue. Figure 2-17. peoplecgi.py update reply Finally, adding a record works the same as in the GUI: fill in a new key and field values and click Update; the CGI script creates a new class instance, fills out its attributes, and stores it in the shelve under the new key. There really is a class object behind the web page here, but we don't have to deal with the logic used to generate it. Figure 2-18 shows a record added to the database in this way. Figure 2-18. peoplecgi.py after adding a new record In principle, we could also update and add records by submitting a URLeither from a browser or from a scriptsuch as: http://localhost/cgi-bin/ peoplecgi.py?action=Update&key=sue&pay=50000&name=Sue+Smith& ...more... Except for automated tools, though, typing such a long URL will be noticeably more difficult than filling out the input page. Here is part of the reply page generated for the "guido" record's display of Figure 2-18 (use your browser's "view page source" option to see this for yourself). Note how the < and > characters are translated to HTML escapes with cgi.escape before being inserted into the reply: <tr><th>key<td><input type=text name=key value="guido"> <tr><th>name<td><input type=text name=name value="'GvR'"> <tr><th>age<td><input type=text name=age value="None"> <tr><th>job<td><input type=text name=job value="'BDFL'"> <tr><th>pay<td><input type=text name=pay value="'<?>'"> As usual, the standard library urllib module comes in handy for testing our CGI script; the output we get back is raw HTML, but we can parse it with other standard library tools and use it as the basis of a server-side script regression testing system run on any Internet-capable machine. We might even parse the server's reply fetched this way and display its data in a client-side GUI coded with Tkinter; GUIs and web pages are not mutually exclusive techniques. The last test in the following interaction shows a portion of the error message page's HTML that is produced when the action is missing or invalid in the inputs, with line breaks added for readability: >>> from urllib import urlopen >>> url = 'http://localhost/cgi-bin/peoplecgi.py?action=Fetch&key=sue' >>> urlopen(url).read( ) '<html>\n<title>People Input Form</title>\n<body>\n <form method=POST action="peoplecgi.py">\n <table>\n <tr><th>key<td><input type=text name=key value="sue">\n <tr><th>name<td><input type=text name=name value="\'Sue Smith\'">\n <tr><t ...more deleted... >>> urlopen('http://localhost/cgi-bin/peoplecgi.py').read( ) '<html>\n<title>People Input Form</title>\n<body>\n <form method=POST action="peoplecgi.py">\n <table>\n <tr><th>key<td><input type=text name=key value="Missing or invalid action!">\n <tr><th>name<td><input type=text name=name value="\'?\'">\n <tr><th>age<td><input type=text name=age value="\'?\'">\n<tr><th>job ...more deleted... In fact, if you're running this CGI script on "localhost," you can use both the last section's GUI and this section's web interface to view the same physical shelve filethese are just alternative interfaces to the same persistent Python objects. For comparison, Figure 2-19 shows what the record we saw in Figure 2-18 looks like in the GUI; it's the same object, but we are not contacting an intermediate server, starting other scripts, or generating HTML to view it. Figure 2-19. Same object displayed in the GUI And as before, we can always check our work on the server machine either interactively or by running scripts. We may be viewing a database through web browsers and GUIs, but, ultimately, it is just Python objects in a Python shelve file: >>> import shelve >>> db = shelve.open('class-shelve') >>> db['sue'].name 'Sue Smith' >>> db['guido'].job 'BDFL' >>> list(db['guido'].name) ['G', 'v', 'R'] 2.8.5.3. Future directionsNaturally, there are plenty of improvements we could make here too:
For now, though, both the GUI and web-based interfaces we've coded get the job done. |
