Recipe2.28.File Locking Using a Cross-Platform API

Recipe 2.28. File Locking Using a Cross-Platform API

Credit: Jonathan Feinberg, John Nielsen

Problem

You need to lock files in a program that runs on both Windows and Unix-like systems, but the Python Standard Library offers only platform-specific ways to lock files.

Solution

When the Python Standard Library itself doesn't offer a cross-platform solution, it's often possible to implement one ourselves:

import os # needs win32all to work on Windows (NT, 2K, XP, _not_ /95 or /98) if os.name == 'nt':     import win32con, win32file, pywintypes     LOCK_EX = win32con.LOCKFILE_EXCLUSIVE_LOCK     LOCK_SH = 0 # the default     LOCK_NB = win32con.LOCKFILE_FAIL_IMMEDIATELY     _ _overlapped = pywintypes.OVERLAPPED( )     def lock(file, flags):         hfile = win32file._get_osfhandle(file.fileno( ))         win32file.LockFileEx(hfile, flags, 0, 0xffff0000, _ _overlapped)     def unlock(file):         hfile = win32file._get_osfhandle(file.fileno( ))         win32file.UnlockFileEx(hfile, 0, 0xffff0000, _ _overlapped) elif os.name == 'posix':     from fcntl import LOCK_EX, LOCK_SH, LOCK_NB     def lock(file, flags):         fcntl.flock(file.fileno( ), flags)     def unlock(file):         fcntl.flock(file.fileno( ), fcntl.LOCK_UN) else:     raise RuntimeError("PortaLocker only defined for nt and posix platforms")

Discussion

When multiple programs or threads have to access a shared file, it's wise to ensure that accesses are synchronized so that two processes don't try to modify the file contents at the same time. Failure to synchronize accesses could even corrupt the entire file in some cases.

This recipe supplies two functions, lock and unlock, that request and release locks on a file, respectively. Using the portalocker.py module is a simple matter of calling the lock function and passing in the file and an argument specifying the kind of lock that is desired:

Shared lock (default): This lock denies all processes, including the process that first locks the file, write access to the file. All processes can read the locked file.
Exclusive lock: This denies all other processes both read and write access to the file.
Nonblocking lock: When this value is specified, the function returns immediately if it is unable to acquire the requested lock. Otherwise, it waits. LOCK_NB can be ORed with either LOCK_SH or LOCK_EX by using Python's bitwise-or operator, the vertical bar (|).

For example:

import portalocker afile = open("somefile", "r+") portalocker.lock(afile, portalocker.LOCK_EX)

The implementation of the lock and unlock functions is entirely different on different systems. On Unix-like systems (including Linux and Mac OS X), the recipe relies on functionality made available by the standard fcntl module. On Windows systems (NT, 2000, XPit doesn't work on old Win/95 and Win/98 platforms because they just don't have the needed oomph in the operating system!), the recipe uses the win32file module, part of the very popular PyWin32 package of Windows-specific extensions to Python, authored by Mark Hammond. But the important point is that, despite the differences in implementation, the functions (and the flags you can pass to the lock function) are made to behave in the same way across platforms. Such cross-platform packaging of differently implemented but equivalent functionality enables you to easily write cross-platform applications, which is one of Python's strengths.

When you write a cross-platform program, it's nice if the functionality that your program uses is, in turn, encapsulated in a cross-platform way. For file locking in particular, it is especially helpful to Perl users, who are used to an essentially transparent lock system call across platforms. More generally, if os.name== just does not belong in application-level code. Such platform testing ideally should always be in the standard library or an application-independent module, as it is here.

Recipe2.28.File Locking Using a Cross-Platform API