Section 13.3. Socket Programming


13.3. Socket Programming

Now that we've seen how sockets figure into the Internet picture, let's move on to explore the tools that Python provides for programming sockets with Python scripts. This section shows you how to use the Python socket interface to perform low-level network communications. In later chapters, we will instead use one of the higher-level protocol modules that hide underlying sockets. Python's socket interfaces can be used directly, though, to implement custom network dialogs and to access standard protocols directly.

The basic socket interface in Python is the standard library's socket module. Like the os POSIX module, Python's socket module is just a thin wrapper (interface layer) over the underlying C library's socket calls. Like Python files, it's also object-basedmethods of a socket object implemented by this module call out to the corresponding C library's operations after data conversions. For instance, the C library's send and recv function calls become methods of socket objects in Python.

Python's socket module supports socket programming on any machine that supports BSD-style socketsWindows, Macs, Linux, Unix, and so onand so provides a portable socket interface. In addition, this module supports all commonly used socket typesTCP/IP, UDP, datagram, and Unix domainand can be used as both a network interface API and a general IPC mechanism between processes running on the same machine.

Beyond basic data communication tasks, this module also includes a variety of more advanced tools. For instance, it has calls for:

  • Converting bytes to a standard network ordering (ntohl, htonl)

  • Wrapping socket objects in a file object interface (sockobj.makefile)

  • Making socket calls nonblocking (sockbj.setblocking)

  • Setting socket timeouts (sockobj.settimeout)

and more. Provided your Python was compiled with Secure Sockets Layer (SSL) support, this module now also supports encrypted transfers with its socket.ssl. This call is used in turn by other standard library modules to support the HTTPS secure web site protocol (httplib, urllib, and urllib2), secure email transfers (poplib and smtplib), and more. We'll meet some of these other modules later in this part of the book, but we won't study all of the socket module's advanced features in this text; see the Python library manual for usage details omitted here.

13.3.1. Socket Basics

Although we won't get into advanced socket use in this chapter, basic socket transfers are remarkably easy to code in Python. To create a connection between machines, Python programs import the socket module, create a socket object, and call the object's methods to establish connections and send and receive data.

Sockets are inherently bidirectional in nature, and socket object methods map directly to socket calls in the C library. For example, the script in Example 13-1 implements a program that simply listens for a connection on a socket, and echoes back over a socket whatever it receives through that socket, adding 'Echo=>' string prefixes.

Example 13-1. PP3E\Internet\Sockets\echo-server.py

 ############################################################################ # Server side: open a TCP/IP socket on a port, listen for a message from # a client, and send an echo reply; this is a simple one-shot listen/reply # conversation per client, but it goes into an infinite loop to listen for # more clients   as long as this server script runs; the client may run on # a remote machine, or on same computer if it uses 'localhost' for server ############################################################################ from socket import *                    # get socket constructor and constants myHost = ''                             # server machine, '' means local host myPort = 50007                          # listen on a non-reserved port number sockobj = socket(AF_INET, SOCK_STREAM)       # make a TCP socket object sockobj.bind((myHost, myPort))               # bind it to server port number sockobj.listen(5)                            # listen, allow 5 pending connects while True:                                    # listen until process killed     connection, address = sockobj.accept( )   # wait for next client connect     print 'Server connected by', address       # connection is a new socket     while True:         data = connection.recv(1024)         # read next line on client socket         if not data: break                   # send a reply line to the client         connection.send('Echo=>' + data)     # until eof when socket closed     connection.close( ) 

As mentioned earlier, we usually call programs like this that listen for incoming connections servers because they provide a service that can be accessed at a given machine and port on the Internet. Programs that connect to such a server to access its service are generally called clients. Example 13-2 shows a simple client implemented in Python.

Example 13-2. PP3E\Internet\Sockets\echo-client.py

 ############################################################################ # Client side: use sockets to send data to the server, and print server's # reply to each message line; 'localhost' means that the server is running # on the same machine as the client, which lets us test client and server # on one machine;  to test over the Internet, run a server on a remote # machine, and set serverHost or argv[1] to machine's domain name or IP addr; # Python sockets are a portable BSD socket interface, with object methods # for the standard socket calls available in the sytstem's C library; ############################################################################ import sys from socket import *              # portable socket interface plus constants serverHost = 'localhost'          # server name, or: 'starship.python.net' serverPort = 50007                # non-reserved port used by the server message = ['Hello network world']           # default text to send to server if len(sys.argv) > 1:     serverHost = sys.argv[1]                # or server from cmd line arg 1     if len(sys.argv) > 2:                   # or text from cmd line args 2..n         message = sys.argv[2:]              # one message for each arg listed sockobj = socket(AF_INET, SOCK_STREAM)      # make a TCP/IP socket object sockobj.connect((serverHost, serverPort))   # connect to server   machine and port for line in message:     sockobj.send(line)                      # send line to server over socket     data = sockobj.recv(1024)               # receive line from server: up to 1k     print 'Client received:', repr(data)    # make sure it is quoted, was 'x' sockobj.close( )                             # close socket to send eof to server 

13.3.1.1. Server socket calls

Before we see these programs in action, let's take a minute to explain how this client and server do their stuff. Both are fairly simple examples of socket scripts, but they illustrate the common call patterns of most socket-based programs. In fact, this is boilerplate code: most socket programs generally make the same socket calls that our two scripts do, so let's step through the important points of these scripts line by line.

Programs such as Example 13-1 that provide services for other programs with sockets generally start out by following this sequence of calls:


sockobj = socket(AF_INET, SOCK_STREAM)

Uses the Python socket module to create a TCP socket object. The names AF_INET and SOCK_STREAM are preassigned variables defined by and imported from the socket module; using them in combination means "create a TCP/IP socket," the standard communication device for the Internet. More specifically, AF_INET means the IP address protocol, and SOCK_STREAM means the TCP transfer protocol.

If you use other names in this call, you can instead create things like UDP connectionless sockets (use SOCK_DGRAM second) and Unix domain sockets on the local machine (use AF_UNIX first), but we won't do so in this book. See the Python library manual for details on these and other socket module options. Using other socket types is mostly a matter of using different forms of boilerplate code.


sockobj.bind((myHost, myPort))

Associates the socket object to an addressfor IP addresses, we pass a server machine name and port number on that machine. This is where the server identifies the machine and port associated with the socket. In server programs, the hostname is typically an empty string (""), which means the machine that the script runs on and the port is a number outside the range 0 to 1023 (which is reserved for standard protocols, described earlier).

Note that each unique socket dialog you support must have its own port number; if you try to open a socket on a port already in use, Python will raise an exception. Also notice the nested parentheses in this callfor the AF_INET address protocol socket here, we pass the host/port socket address to bind as a two-item tuple object (pass a string for AF_UNIX). Technically, bind takes a tuple of values appropriate for the type of socket created (but see the next Note box about the older and deprecated convention of passing values to this function as distinct arguments).


sockobj.listen(5)

Starts listening for incoming client connections and allows for a backlog of up to five pending requests. The value passed sets the number of incoming client requests queued by the operating system before new requests are denied (which happens only if a server isn't fast enough to process requests before the queues fill up). A value of 5 is usually enough for most socket-based programs; the value must be at least 1.

At this point, the server is ready to accept connection requests from client programs running on remote machines (or the same machine), and falls into an infinite loopwhile True:, or the equivalent while 1: on older Pythonswaiting for them to arrive:


connection, address = sockobj.accept( )

Waits for the next client connection request to occur; when it does, the accept call returns a brand-new socket object over which data can be transferred from and to the connected client. Connections are accepted on sockobj, but communication with a client happens on connection, the new socket. This call actually returns a two-item tupleaddress is the connecting client's Internet address. We can call accept more than one time, to service multiple client connections; that's why each call returns a new, distinct socket for talking to a particular client.

Once we have a client connection, we fall into another loop to receive data from the client in blocks of 1,024 bytes at a time, and echo each block back to the client:


data = connection.recv(1024)

Reads at most 1,024 more bytes of the next message sent from a client (i.e., coming across the network), and returns it to the script as a string. We get back an empty string when the client has finishedend-of-file is triggered when the client closes its end of the socket.


connection.send('Echo=>' + data)

Sends the latest data block back to the client program, prepending the string 'Echo=>' to it first. The client program can then recv what we send herethe next reply line. Technically this call sends as much data as possible, and returns the number of bytes actually sent. To be fully robust, programs need to resend unsent portions or use connection.sendall to force all bytes to be sent.


connection.close( )

Shuts down the connection with this particular client.

After talking with a given client, the server goes back to its infinite loop and waits for the next client connection request.

13.3.1.2. Client socket calls

On the other hand, client programs like the one shown in Example 13-2 follow simpler call sequences. The main thing to keep in mind is that the client and server must specify the same port number when opening their sockets, and the client must identify the machine on which the server is running (in our scripts, server and client agree to use port number 50007 for their conversation, outside the standard protocol range):


sockobj = socket(AF_INET, SOCK_STREAM)

Creates a Python socket object in the client program, just like the server.


sockobj.connect((serverHost, serverPort))

Opens a connection to the machine and port on which the server program is listening for client connections. This is where the client specifies the string name of the service to be contacted. In the client, we can either specify the name of the remote machine as a domain name (e.g., starship.python.net) or numeric IP address. We can also give the server name as localhost (or the equivalent IP address 127.0.0.1) to specify that the server program is running on the same machine as the client; that comes in handy for debugging servers without having to connect to the Net. And again, the client's port number must match the server's exactly. Note the nested parentheses againjust as in server bind calls we really pass the server's host/port address to connect in a tuple object.

Once the client establishes a connection to the server, it falls into a loop, sending a message one line at a time and printing whatever the server sends back after each line is sent:


sockobj.send(line)

Transfers the next message line to the server over the socket.


data = sockobj.recv(1024)

Reads the next reply line sent by the server program. Technically, this reads up to 1,024 bytes of the next reply message and returns it as a string.


sockobj.close( )

Closes the connection with the server, sending it the end-of-file signal.

And that's it. The server exchanges one or more lines of text with each client that connects. The operating system takes care of locating remote machines, routing bytes sent between programs across the Internet, and (with TCP) making sure that our messages arrive intact. That involves a lot of processing tooour strings may ultimately travel around the world, crossing phone wires, satellite links, and more along the way. But we can be happily ignorant of what goes on beneath the socket call layer when programming in Python.

In older Python code, you may see the AF_INET server address passed to the server-side bind and client-side connect socket methods as two distinct arguments, instead of as a two-item tuple:

 soc.bind(host,port)     vs soc.bind((host,port)) soc.connect(host,port)  vs soc.connect((host,port)) 

This two-argument form is now deprecated, and only worked at all due to a shortcoming in earlier Python releases (unfortunately, the Python library manual's socket example used the two-argument form too!). The tuple server address form is preferred and, in a rare Python break with full backward-compatibility, will likely be the only one that will work in future Python releases.


13.3.1.3. Running socket programs locally

Okay, let's put this client and server to work. There are two ways to run these scriptson either the same machine or two different machines. To run the client and the server on the same machine, bring up two command-line consoles on your computer, start the server program in one, and run the client repeatedly in the other. The server keeps running and responds to requests made each time you run the client script in the other window.

For instance, here is the text that shows up in the MS-DOS console window where I've started the server script:

 C:\...\PP3E\Internet\Sockets>python echo-server.py Server connected by ('127.0.0.1', 1025) Server connected by ('127.0.0.1', 1026) Server connected by ('127.0.0.1', 1027) 

The output here gives the address (machine IP name and port number) of each connecting client. Like most servers, this one runs perpetually, listening for client connection requests. This server receives three, but I have to show you the client window's text for you to understand what this means:

 C:\...\PP3E\Internet\Sockets>python echo-client.py Client received: 'Echo=>Hello network world' C:\...\PP3E\Internet\Sockets>python echo-client.py localhost spam Spam SPAM Client received: 'Echo=>spam' Client received: 'Echo=>Spam' Client received: 'Echo=>SPAM' C:\...\PP3E\Internet\Sockets>python echo-client.py localhost Shrubbery Client received: 'Echo=>Shrubbery' 

Here, I ran the client script three times, while the server script kept running in the other window. Each client connected to the server, sent it a message of one or more lines of text, and read back the server's replyan echo of each line of text sent from the client. And each time a client is run, a new connection message shows up in the server's window (that's why we got three).

It's important to notice that client and server are running on the same machine here (a Windows PC). The server and client agree on the port number, but they use the machine names "" and localhost, respectively, to refer to the computer on which they are running. In fact, there is no Internet connection to speak of. Sockets also work well as cross-program communications tools on a single machine.

13.3.1.4. Running socket programs remotely

To make these scripts talk over the Internet rather than on a single machine, we have to do some extra work to run the server on a different computer. First, upload the server's source file to a remote machine where you have an account and a Python. Here's how I do it with FTP; your server name and upload interface details may vary, and there are other ways to copy files to a computer (e.g., email, web page post forms, etc.):[*]

[*] The FTP command is standard on Windows machines and most others. On Windows, simply type it in a DOS console box to connect to an FTP server (or start your favorite FTP program); on Linux, type the FTP command in an xterm window. You'll need to supply your account name and password to connect to a nonanonymous FTP site. For anonymous FTP, use "anonymous" for the username and your email address for the password (anonymous FTP sites are generally limited).

 C:\...\PP3E\Internet\Sockets>ftp starship.python.net Connected to starship.python.net. User (starship.python.net:(none)): lutz 331 Password required for lutz. Password: 230 User lutz logged in. ftp> put echo-server.py 200 PORT command successful. 150 Opening ASCII mode data connection for echo-server.py. 226 Transfer complete. ftp: 1322 bytes sent in 0.06Seconds 22.03Kbytes/sec. ftp> quit 

Once you have the server program loaded on the other computer, you need to run it there. Connect to that computer and start the server program. I usually Telnet into my server machine and start the server program as a perpetually running process from the command line.[*] The & syntax in Unix/Linux shells can be used to run the server script in the background; we could also make the server directly executable with a #! line and a chmod command (see Chapter 3 for details). Here is the text that shows up in a Window on my PC that is running a Telnet session connected to the Linux server where I have an account (minus a few deleted informational lines):

[*] Telnet is a standard command on Windows and Linux machines, too. On Windows, type it at a DOS console prompt or in the Start/Run dialog box (it can also be started via a clickable icon). Telnet usually runs in a window of its own. For some server machines, you'll need to use secure shell rather than Telnet to access a shell prompt.

 C:\...\PP3E\Internet\Sockets>telnet starship.python.net Red Hat Linux release 6.2 (Zoot) Kernel 2.2.14-5.0smp on a 2-processor i686 login: lutz Password: [lutz@starship lutz]$ python echo-server.py & [1] 4098 

Now that the server is listening for connections on the Net, run the client on your local computer multiple times again. This time, the client runs on a different machine than the server, so we pass in the server's domain or IP name as a client command-line argument. The server still uses a machine name of "" because it always listens on whatever machine it runs on. Here is what shows up in the server's Telnet window:

 [lutz@starship lutz]$ Server connected by ('166.93.68.61', 1037) Server connected by ('166.93.68.61', 1040) Server connected by ('166.93.68.61', 1043) Server connected by ('166.93.68.61', 1050) 

And here is what appears in the MS-DOS console box where I run the client. A "connected by" message appears in the server Telnet window each time the client script is run in the client window:

 C:\...\PP3E\Internet\Sockets>python echo-client.py starship.python.net Client received: 'Echo=>Hello network world' C:\...\PP3E\Internet\Sockets>python echo-client.py starship.python.net ni Ni NI Client received: 'Echo=>ni' Client received: 'Echo=>Ni' Client received: 'Echo=>NI' C:\...\PP3E\Internet\Sockets>python echo-client.py starship.python.net Shrubbery Client received: 'Echo=>Shrubbery' C:\...\PP3E\Internet\Sockets>ping starship.python.net Pinging starship.python.net [208.185.174.112] with 32 bytes of data: Reply from 208.185.174.112: bytes=32 time=311ms TTL=246 ctrl-C C:\...\PP3E\Internet\Sockets>python echo-client.py 208.185.174.112 Does she? Client received: 'Echo=>Does' Client received: 'Echo=>she?' 

The ping command can be used to get an IP address for a machine's domain name; either machine name form can be used to connect in the client. This output is perhaps a bit understateda lot is happening under the hood. The client, running on my Windows laptop, connects with and talks to the server program running on a Linux machine perhaps thousands of miles away. It all happens about as fast as when client and server both run on the laptop, and it uses the same library calls; only the server name passed to clients differs.

13.3.1.5. Socket pragmatics

Before we move on, there are three practical usage details you should know. First, you can run the client and server like this on any two Internet-aware machines where Python is installed. Of course, to run the client and server on different computers, you need both a live Internet connection and access to another machine on which to run the server. You don't need a big, expensive Internet link, thougha simple modem and dial-up Internet account will do for clients. When sockets are opened, Python is happy to use whatever connectivity you have, be it a dedicated T1 line or a dial-up modem account.

On a laptop PC with just dial-up access, for instance, Windows automatically dials out to your ISP when clients are started or when Telnet server sessions are opened. If a broadband connection is available, that is utilized instead. In this book's examples, server-side programs that run remotely are executed on a machine called starship.python.net. If you don't have an account of your own on such a server, simply run client and server examples on the same machine, as shown earlier; all you need then is a computer that allows sockets, and most do.

Second, the socket module generally raises exceptions if you ask for something invalid. For instance, trying to connect to a nonexistent server (or unreachable servers, if you have no Internet link) fails:

 C:\...\PP3E\Internet\Sockets>python echo-client.py www.nonesuch.com hello Traceback (innermost last):   File "echo-client.py", line 24, in ?     sockobj.connect((serverHost, serverPort))   # connect to server machine...   File "<string>", line 1, in connect socket.error: (10061, 'winsock error') 

Finally, also be sure to kill the server process before restarting it again, or else the port number will still be in use, and you'll get another exception:

 [lutz@starship uploads]$ ps -x   PID TTY      STAT   TIME COMMAND  5570 pts/0    S      0:00 -bash  5570 pts/0    S      0:00 -bash  5633 pts/0    S      0:00 python echo-server.py  5634 pts/0    R      0:00 ps -x [lutz@starship uploads]$ python echo-server.py Traceback (most recent call last):   File "echo-server.py", line 14, in ?     sockobj.bind((myHost, myPort))               # bind it to server port number socket.error: (98, 'Address already in use') 

A series of Ctrl-Cs will kill the server on Linux (be sure to type fg to bring it to the foreground first if started with an &):

 [lutz@starship uploads]$ python echo-server.py ctrl-c Traceback (most recent call last):   File "echo-server.py", line 18, in ?     connection, address = sockobj.accept( )   # wait for next client connect KeyboardInterrupt 

A Ctrl-C kill key combination won't kill the server on my Windows machine, however. To kill the perpetually running server process running locally on Windows, you may need to type a Ctrl-Alt-Delete key combination, and then end the Python task by selecting it in the process listbox that appears. Closing the window in which the server is running will also suffice on Windows, but you'll lose that window's command history. You can also usually kill a server on Linux with a kill -9 pid shell command if it is running in another window or in the background, but Ctrl-C requires less typing.

13.3.1.6. Spawning clients in parallel

To see how the server handles the load, let's fire up eight copies of the client script in parallel using the script in Example 13-3 (see the end of Chapter 5 for details on the launchmodes module used here to spawn clients).

Example 13-3. PP3E\Internet\Sockets\testecho.py

 import sys from PP3E.launchmodes import QuietPortableLauncher numclients = 8 def start(cmdline): QuietPortableLauncher(cmdline, cmdline)( ) # start('echo-server.py')              # spawn server locally if not yet started args = ' '.join(sys.argv[1:])          # pass server name if running remotely for i in range(numclients):     start('echo-client.py %s' % args)  # spawn 8? clients to test the server 

To run this script, pass no arguments to talk to a server listening on port 50007 on the local machine; pass a real machine name to talk to a server running remotely. On Windows, the clients' output is discarded when spawned from this script:

 C:\...\PP3E\Internet\Sockets>python testecho.py C:\...\PP3E\Internet\Sockets>python testecho.py starship.python.net 

If the spawned clients connect to a server run locally, connection messages show up in the server's window on the local machine:

 C:\...\PP3E\Internet\Sockets>python echo-server.py Server connected by ('127.0.0.1', 1283) Server connected by ('127.0.0.1', 1284) Server connected by ('127.0.0.1', 1285) Server connected by ('127.0.0.1', 1286) Server connected by ('127.0.0.1', 1287) Server connected by ('127.0.0.1', 1288) Server connected by ('127.0.0.1', 1289) Server connected by ('127.0.0.1', 1290) 

If the server is running remotely, the client connection messages instead appear in the window displaying the Telnet connection to the remote computer:

 [lutz@starship lutz]$ python echo-server.py Server connected by ('166.93.68.61', 1301) Server connected by ('166.93.68.61', 1302) Server connected by ('166.93.68.61', 1308) Server connected by ('166.93.68.61', 1309) Server connected by ('166.93.68.61', 1313) Server connected by ('166.93.68.61', 1314) Server connected by ('166.93.68.61', 1307) Server connected by ('166.93.68.61', 1312) 

Keep in mind, however, that this works for our simple scripts only because the server doesn't take a long time to respond to each client's requestsit can get back to the top of the server script's outer while loop in time to process the next incoming client. If it could not, we would probably need to change the server to handle each client in parallel, or some might be denied a connection. Technically, client connections would fail after five clients are already waiting for the server's attention, as specified in the server's listen call. We'll see how servers can handle multiple clients robustly in the next section.

13.3.1.7. Talking to reserved ports

It's also important to know that this client and server engage in a proprietary sort of discussion, and so use the port number 50007 outside the range reserved for standard protocols (0 to 1023). There's nothing preventing a client from opening a socket on one of these special ports, however. For instance, the following client-side code connects to programs listening on the standard email, FTP, and HTTP web server ports on three different server machines:

 C:\...\PP3E\Internet\Sockets>python >>> from socket import * >>> sock = socket(AF_INET, SOCK_STREAM) >>> sock.connect(('mail.rmi.net', 110))         # talk to RMI POP mail server >>> print sock.recv(40) +OK Cubic Circle's v1.31 1998/05/13 POP3 >>> sock.close( ) >>> sock = socket(AF_INET, SOCK_STREAM) >>> sock.connect(('www.python.org', 21))        # talk to Python FTP server >>> print sock.recv(40) 220 python.org FTP server (Version wu-2. >>> sock.close( ) >>> sock = socket(AF_INET, SOCK_STREAM) >>> sock.connect(('starship.python.net', 80))   # starship HTTP web server >>> sock.send('GET /\r\n')                      # fetch root web page 7 >>> sock.recv(60) '<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">\012<HTM' >>> sock.recv(60) 'L>\012 <HEAD>\012  <TITLE>Starship Slowly Recovering</TITLE>\012 </HE' 

If we know how to interpret the output returned by these ports' servers, we could use raw sockets like this to fetch email, transfer files, and grab web pages and invoke server-side scripts. Fortunately, though, we don't have to worry about all the underlying detailsPython's poplib, ftplib, httplib, and urllib modules provide higher-level interfaces for talking to servers on these ports. Other Python protocol modules do the same for other standard ports (e.g., NNTP, Telnet, and so on). We'll meet some of these client-side protocol modules in the next chapter.[*]

[*] You might be interested to know that the last part of this example, talking to port 80, is exactly what your web browser does as you surf the Web: followed links direct it to download web pages over this port. In fact, this lowly port is the primary basis of the Web. In Chapter 16, we will meet an entire application environment based upon sending formatted data over port 80CGI server-side scripting. At the bottom, though, the Web is just bytes over sockets, with a user interface. The wizard behind the curtain is not as impressive as he may seem!

By the way, it's all right to open client-side connections on reserved ports like this, but you can't install your own server-side scripts for these ports unless you have special permission:

 [lutz@starship uploads]$ python >>> from socket import * >>> sock = socket(AF_INET, SOCK_STREAM) >>> sock.bind(('', 80)) Traceback (most recent call last):   File "<stdin>", line 1, in ? socket.error: (13, 'Permission denied') 

Even if run by a user with the required permission, you'll get the different exception we saw earlier if the port is already being used by a real web server. On computers being used as general servers, these ports really are reserved.




Programming Python
Programming Python
ISBN: 0596009259
EAN: 2147483647
Year: 2004
Pages: 270
Authors: Mark Lutz

flylib.com © 2008-2017.
If you may any questions please contact us: flylib@qtcs.net