Section 13.16. Advanced Features of New-Style Classes (Python 2.2)
13.16. Advanced Features of New-Style Classes (Python 2.2+)13.16.1. General Features of New-Style ClassesWe have already discussed some of the features tied to new-style classes. With the unification of types and classes, the most significant of these features is the ability to subclass Python data types. One side effect of this is that all of the Python "casting" or conversion built-in functions are now factory functions. When you call them, you are really instantiating an instance of the corresponding type. The following built-in function, which have been around Python for a while, have been quietly (or perhaps not) converted to factory functions:
In addition, several new ones have been added to round out the posse:
These class names and factory functions have flexible usage. In addition to creating new objects of those types, they can be used as base classes when subclassing types, and they can now be used with the isinstance() built-in function. Using isinstance() can help replace tired old idioms with one that requires fewer functions calls resulting in cleaner code. For example, to test if an object is an integer, we had to call type() twice or import the types module and use its attributes; but now we can just use isinstance() and even gain in performance:
Keep in mind that although isinstance() is flexible, it does not perform an "exact match" comparisonit will also return true if obj is an instance of the given type or an instance of a subclass of the given type. You will still need to use the is operator if you want an exact class match. Please review Section 13.12.2 above for a deeper explanation of isinstance() as well as its introduction in Chapter 4 and how these calls evolved along with Python. 13.16.2. __slots__ Class AttributeA dictionary is at the heart of all instances. The __dict__ attribute keeps track of all instance attributes. For example, when you have an instance inst with an attribute foo, recognize that accessing it with inst.foo is the same as doing it with inst.__dict__['foo']. This dictionary takes up a good amount of memory, and if you have a class with very few attributes but a significant number of instances of this object, then you are taking a substantial hit. To combat this, users are now able to use the __slots__ attribute to substitute for __dict__. Basically, __slots__ is a class variable consisting of a sequence-like object representing the set of valid identifiers that make up all of an instance's attributes. This can be a list, tuple, or iterable. It can also be a single string identifying the single attribute that an instance can have. Any attempt to create an instance attribute with a name not in __slots__ will result in an AttributeError exception: class SlottedClass(object): __slots__ = ('foo', 'bar') >>> c = SlottedClass() >>> >>> c.foo = 42 >>> c.xxx = "don't think so" Traceback (most recent call last): File "<stdin>", line 1, in ? AttributeError: 'SlottedClass' object has no attribute 'xxx'The primary reason for this feature is the conservation of memory. A side effect is a type of security preventing users from adding instances attributes dynamically in an ad hoc manner. A class defined with a __slots__ attribute will not have a __dict__ (unless you add '__dict__' as an element of __slots__). For more information on __slots__, see the Data Model chapter of the Python (Language) Reference Manual. 13.16.3. __getattribute__() Special MethodPython classes have a special method named __getattr__(), which is called only when an attribute cannot be found in an instance's __dict__ or its class (class's __dict__), or ancestor class (its __dict__). One place where we saw __getattr__() used was for implementing delegation. The problem that many users encountered was that they wanted a certain function to execute for every attribute access, not just when one cannot be found. This is where __getattribute__() comes in. It works just like __getattr__() except that it is always called when an attribute is accessed, not just when it cannot be found. If a class has both __getattribute__() and __getattr__() defined, the latter will not be called unless explicitly called from __getattribute__() or if __getattribute__() raises AttributeError. Be very careful if you are going to access attributes in here... attributes of this class or an ancestor. If you cause __getattribute__() to somehow call __getattribute__()again, you will have infinite recursion. To avoid infinite recursion using this method, you should always call an ancestor class method that shares the same name in order to access any attributes it needs safely; for example, super(obj, self).__getattribute__(attr). This special method is only valid with new-style classes. As with __slots__, you can get more information on __getattribute__() by referring to the Data Model chapter of the Python (Language) Reference Manual. 13.16.4. DescriptorsDescriptors are one of the keys behind Python's new-style classes. They provide a powerful API to object attributes. You can think of a descriptor as an agent that presents object attributes. Depending on which situation you encounter when you need an attribute, you can get to it via its descriptor (if there is one for it) or in the normal way (dotted attribute notation). If there is an agent for your object and it has a "get" attribute (really spelled __get__), it is invoked, and you get back all you need to access the object in question. The same thing applies if you are attempting to assign a value to an object with a descriptor (set) or removing an attribute (delete). __get__(), __set__(), __delete__() Special MethodsStrictly speaking, a descriptor is really any (new-style) class that implements at least one of three special methods that serve as the descriptor protocol: __get__(), __set__(), and __delete__(). As mentioned just above, __get__() is used to get the value of an attribute, __set__() is used to assign a value to an attribute, and __delete__() is called when an attribute is explicitly removed using the del statement (or rather, its reference count decremented). Of the three, the latter is rarely implemented. Also, not all descriptors implement the __set__() method either. These are referred to as method descriptors, or more accurately, non-data descriptors. Those that override both __get__() and __set__() are called data descriptors, and they are more powerful than non-data descriptors. The signatures for __get__(), __set__(), and __delete__() look like this:
When you want to use an agent for an attribute, you install it as a class attribute, and let the agent handle all the dirty work. Anytime you try to do something to an attribute with that name, you will get the descriptor that proxies all functionality. We covered wrapping in the previous section. This is just more wrapping going on. Instead of just delegating everything to objects in your class, we are delegating slightly more complex attribute access here. __getattribute__() Special Method (again)Ordering matters when using descriptors, and certain aspects have precedence over others. The heart of the entire system is __getattribute__() since that special method is called for every attribute instance. It is the one that finds a class attribute or an agent to call on your behalf to access an attribute, etc. Reviewing the signatures just above, if __get__() is called for an instance, the object is passed in and perhaps a type or class. For example, given a class X and an instance x, x.foo is translated by __getattribute__() to: type(x).__dict__['foo'].__get__(x, type(x)) If __get__() is called for a class, then None is passed in as the object (which would be self for an instance): X.__dict__['foo'].__get__(None, X) Finally, if super() is called, for example given Y as a subclass of X, then super(Y, obj).foo looks in obj.__class__.__mro__ for the class right next to Y going up the tree to find class X, and then calls: X.__dict__['foo'].__get__(obj, X) Then it is up to that descriptor to return the desired object. PrecedenceThe way __getattribute__() works needs to be covered, as it was implemented to behave in a very specific way. Thus it is very important to recognize this ordering:
A descriptor is a class attribute, so all class attributes have the highest priority. You can even replace a descriptor by simply reassigning its original reference to other objects. They are followed closely behind by descriptors with __get__() and __set__() implemented. If you have an agent, it will do all your work for you! Otherwise, it should just default to the local object's __dict__, meaning that it will be an instance attribute. The non-data descriptors come next. This may sound surprising because on first glance, one would think that these should be higher up in the food chain than instance attributes, but that is not the case. The purpose of non-data descriptors is only to provide a value if one is not already part of an instance, sort of how __getattr__() is only called if an attribute cannot be found in an instance's __dict__! Speaking of __getattr__(), if no non-data descriptor is found, then __getattribute__() raises an AttributeError, and that in turn causes __getattr__() to be invoked as the last stand before AttributeError is raised to the user. Descriptor ExamplesLet us start with a very boring example... a descriptor that just discards any attempt to retrieve or set a value from and to an attribute, respectively. Actually, all of the examples here just ignore all requests, but they are incremental, and we hope that you can figure out a little more about descriptors for each one: class DevNull1(object): def __get__(self, obj, typ=None): pass def __set__(self, obj, val): pass We create a class that uses this descriptor and try to assign something to it as well as display its value: >>> class C1(object): ... foo = DevNull1() ... >>> c1 = C1() >>> c1.foo = 'bar' >>> print 'c1.foo contains:', c1.foo c1.foo contains: None That was not too terribly exciting... how about one where the descriptor methods at least give some output to show what is going on? class DevNull2(object): def __get__(self, obj, typ=None): print 'Accessing attribute... ignoring' def __set__(self, obj, val): print 'Attempt to assign %r... ignoring' % (val) Now let us see this one in action: >>> class C2(object): ... foo = DevNull2() ... >>> c2 = C2() >>> c2.foo = 'bar' Attempt to assign 'bar'... ignoring >>> x = c2.foo Accessing attribute... ignoring >>> print 'c2.foo contains:', x c2.foo contains: None For our final example, let us add a placeholder in our descriptor class that holds some useful information about the descriptor: class DevNull3(object): def __init__(self, name=None): self.name = name def __get__(self, obj, typ=None): print 'Accessing [%s]... ignoring' % self.name) def __set__(self, obj, val): print 'Assigning %r to [%s]... ignoring' % val, self.name) In the output below, we show you the importance of the hierarchy mentioned above, especially where we state that a full data descriptor has precedence over an instance attribute: >>> class C3(object): ... foo = DevNull3('foo') ... >>> c3 = C3() >>> c3.foo = 'bar' Assigning 'bar' to [foo]... ignoring >>> x = c3.foo Accessing [foo]... ignoring >>> print 'c3.foo contains:', x c3.foo contains: None >>> print 'Let us try to sneak it into c3 instance...' Let us try to sneak it into c3 instance... >>> c3.__dict__['foo'] = 'bar' >>> x = c3.foo Accessing [foo]... ignoring >>> print 'c3.foo contains:', x c3.foo contains: None >>> print "c3.__dict__['foo'] contains: %r" % \ c3.__dict__['foo'], "... why?!?" c3.__dict__['foo'] contains: 'bar' ... why?!? Notice how we were able to sneak in an attribute to our instance. We were able to assign the string "bar" to c3.foo, but because the data descriptor is more important, it overrides or effectively hides our assignment. Likewise, because instance attributes have a higher precedence than non-data attributes, you can also hide such a descriptor, just as you can hide a class attribute, by assigning an instance attribute with the same name: >>> class FooFoo(object): ... def foo(self): ... print 'Very important foo() method.' ... >>> >>> bar = FooFoo() >>> bar.foo() Very important foo() method. >>> >>> bar.foo = 'It is no longer here.' >>> bar.foo 'It is no longer here.' >>> >>> del bar.foo >>> bar.foo() Very important foo() method. This was a pretty transparent example because we called it as a function, then accessed it as a string, but we could have used another function and kept the same calling mechanism, too: >>> def barBar(): ... print 'foo() hidden by barBar()' ... >>> bar.foo = barBar >>> bar.foo() foo() hidden by barBar() >>> >>> del bar.foo >>> bar.foo() Very important foo() method. The point was to emphasize that because functions are non-data descriptors, instance attributes are ranked higher, and we can shadow any non-data descriptor simply by assigning an object to the instance (of the same name). Our final example does a little bit more. It is a crude attempt at using the filesystem as a means of storing the contents of an attribute. Lines 110After the usual setup, we create our descriptor class with a class attribute (saved) that keeps track of all attributes with descriptor access. When a descriptor is created, it registers and saves the name of the attribute (passed in from the user). Lines 1226When fetching an attribute, we need to ensure that users do not use it before they have even assigned a value to it. If it passes that test, then we attempt to open the pickle file to read in the saved value. An exception is raised if somehow the file cannot be opened, either because it was erased (or never created), or if it was corrupted or somehow cannot be unserialized by the pickle module. Lines 2838Saving the attribute takes several steps: open the pickle file for write (either creating it for the first time or wiping out one that was already there), serializing the object to disk, and registering the name so users can retrieve the value. An exception is thrown if the object cannot be pickled. Note that if you are using Python 2.5 you can never merge the TRy-except and try finally statements (lines 30-38) together. Example 13.9. Using a File to Store an Attribute (descr.py)
Lines 4045Finally, if the attribute is explicitly deleted, the file is removed, and the name unregistered. Here is some sample usage of this class: >>> class MyFileVarClass(object): ... foo = FileDescr('foo') ... bar = FileDescr('bar') ... >>> fvc = MyFileVarClass() >>> print fvc.foo Traceback (most recent call last): File "<stdin>", line 1, in ? File "descr.py", line 14, in __get__ raise AttributeError, \ AttributeError: 'foo' used before assignment >>> >>> fvc.foo = 42 >>> fvc.bar = 'leanna' >>> >>> print fvc.foo, fvc.bar 42 leanna >>> >>> del fvc.foo >>> print fvc.foo, fvc.bar Traceback (most recent call last): File "<stdin>", line 1, in ? File "descr.py", line 14, in __get__ raise AttributeError, \ AttributeError: 'foo' used before assignment >>> >>> fvc.foo = __builtins__ Traceback (most recent call last): File "<stdin>", line 1, in ? File "descr.py", line 35, in __set__ raise AttributeError, \ AttributeError: could not pickle 'foo' Attribute access appears normal, and the programmer cannot really tell that an object is pickled and stored to the filesystem (except in the last example where we tried to pickle a module, a no-no). We also put in a handler for cases when the pickle file gets corrupted. This is also the first descriptor where we have implemented __delete__(). One thing to keep in mind with all of our examples is that we did not use the instance obj at all. Do not confuse obj with self as the latter is the instance of the descriptor, not the instance of the original class. Descriptor SummaryBelieve it or not, you have already seen descriptors at work. Static methods, class methods, properties (see next section below), and even functions themselves are all descriptors. Think about this: functions are very generic objects in Python. There are built-in ones, user-defined ones, methods defined in classes, static methods, and class methods. Those are all examples of functions. The only difference between them is how they are called. Functions are normally unbound. So are static methods, even though they are defined in classes. But methods need to be bound to an instance, and class methods need to be bound to a class, right? The descriptor for a function object knows all this, so depending on what type of function it is, a descriptor can "wrap up" a function object along with whatever it needs to be bound to, if applicable, and then returns that back to the caller. The way it works is that the function itself is a descriptor, and its __get__() method is what puts together the callable that it returns for you. It is quite an amazing generality, which does not break the way Python has been working all this time! Properties and property() Built-in FunctionProperties are a very useful and specific type of descriptor. They were meant to handle all accesses to instance attributes in a similar manner that we described for descriptors, above. When you access an instance attribute "normally," you use the dotted-attribute notation. You were updating an instance's __dict__ attribute. With properties, although your usage resembles normal attribute access, the actual implementation of such access uses function (or method) calls. In earlier versions of Python, as seen earlier this chapter, you could use __getattr__() and __setattr__() to play with attributes in general. The problem is that all attribute access goes through those special methods (and __getattribute__()), but with properties, you can give a property specific functions to execute for getting, setting, and deleting instance attributes, so you no longer have to use those other special methods (which became quite large actually if you had many instance attributes you were trying to manage). The property() built-in function can take up to four arguments. Its signature is: property(fget=None, fset=None, fdel=None, doc=None) Keep in mind that although normal usage of property() is within a class definition where the functions passed in are actually methods, property() can accept functions. In fact, at the time that property() is called when a class is declared, those methods are unbound and thus really are functions! Here is a simple example that creates a read-only integer attribute but hides it within the class by barely encrypting it using the bitwise XOR operator: class ProtectAndHideX(object): def __init__(self, x): assert isinstance(x, int), \ '"x" must be an integer!' self.__x = ~x def get_x(self): return ~self.__x x = property(get_x) If we try it out, we see that it saves the first value we give it but does not allow us to set it again: >>> inst = ProtectAndHideX('foo') Traceback (most recent call last): File "<stdin>", line 1, in ? File "prop.py", line 5, in __init__ assert isinstance(x, int), \ AssertionError: "x" must be an integer! >>> inst = ProtectAndHideX(10) >>> print 'inst.x =', inst.x inst.x = 10 >>> inst.x = 20 Traceback (most recent call last): File "<stdin>", line 1, in ? AttributeError: can't set attributeHere is another example, but with a setter: class HideX(object): def __init__(self, x): self.x = x def get_x(self): return ~self.__x def set_x(self, x): assert isinstance(x, int), \ '"x" must be an integer!' self.__x = ~x x = property(get_x, set_x) Here is the output of this example: >>> inst = HideX(20) >>> print inst.x 20 >>> inst.x = 30 >>> print inst.x 30 This property works because by the time the constructor is called to set the initial value of x, the getter already saves it as ~x to self.__x. You can even stick in a documentation string for your attribute, as shown here in this next example: from math import pi def get_pi(dummy): return pi class PI(object): pi = property(get_pi, doc='Constant "pi"') Here we are using a function instead of a method for our property, just to show it can be done. When it is called, however, we have to keep in mind that self is going to be passed in as the first (and only) argument, so we still need to have a dummy variable to discard it. Here is the corresponding output: >>> inst = PI() >>> inst.pi 3.1415926535897931 >>> print PI.pi.__doc__ Constant "pi"Can you see how properties take your functions (fget, fset, and fdel) and map them as descriptor methods __get__(), __set__(), and __delete__()? You did not have to create a descriptor class and define these callables as methods of your descriptor class. You just created functions (or methods) and gave them all to property(). One drawback to creating your descriptor methods inside your class definition is that it clutters up the class namespace. Not only that, but isn't the point of having a property to control access to an attribute? But this control does not exist if they are not forced to use the property. Our second example does not enforce this because it allows access to our property methods (since they are part of the class definition): >>> inst.set_x(40) # can we require inst.x = 40? >>> print inst.x 40A clever idiom in a recipe in the ActiveState Programmer Network Python Cookbook (http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/205183) solves both of these problems by:
There is no method clutter in the class's namespace because the methods were defined as inner functions in someone else's namespace. The user has no access to the methods because the namespace in which they were defined was destroyed (by going out-of-scope), thus they are compelled to use the property as that is now the one and only way for them to access the attribute. Here is our modified class inspired by the recipe: class HideX(object): def __init__(self, x): self.x = x @property def x(): def fget(self): return ~self.__x def fset(self, x): assert isinstance(x, int), \ '"x" must be an integer!' self.__x = ~x return locals() Our code works exactly as before, but there two big differences: (1) the namespace for the class is much smaller and consists (only) of ['__doc__', '__init__', '__module__', 'x'], and (2), the user can no longer use inst.set_x(40) to set the attribute ... they have to use init.x = 40. We also use a function decorator (@property) to reassign x from a function to a property object. Since decorators were introduced in Python 2.4, those of you using 2.2.x or 2.3.x need to replace the decorator with the following assignment after the x() function declaration with x = property(**x()). 13.16.5. Metaclasses and __metaclass__What Are Metaclasses?Metaclasses are probably the most mind-bending feature that was added with new-style classes. Metaclasses are classes that let you define how certain classes can be constructed, basically giving you a level of control over how classes are created. (You do not even need to think at the instance level.) They have been talked about since before the days of Python 1.5 (when many minds were bent), but they are finally a reality. Basically, you can think of a metaclass as the class of a class, or rather, a class whose instances are other classes. Believe it or not, whenever you create a class now, you are actually employing the default metaclass, which is a (or rather, the) type object. (If classic classes are used, the metaclasses for those are types.ClassType.) Take any class and call type() on it, and you will see what it is an instance of: class C(object): pass class CC: pass >>> type(C) <type 'type'> >>> >>> type(CC) <type 'classobj'> >>> >>> import types >>> type(CC) is types.ClassType True When Are Metaclasses Used?Metaclasses are always used when creating classes. When executing a class definition, the interpreter has to know the correct metaclass to use. It will look for a class attribute named __metaclass__ first, and if it is there, it will use the class that is assigned to that attribute as the metaclass. If that attribute has not been defined, it will go up and search an ancestor class for __metaclass__. All new-style classes must inherit from object or type if there are no other base classes (type (object) is type anyway). If that is not found, it checks for a global variable named __metaclass__ and uses it if it exists. Otherwise, the class is a classic class, and types.ClassType is used as the metaclass. (Note you can do some trickery here... if you define a classic class and set __metaclass__ = type, you have parlayed it into a new-style class!) Any time a class declaration is executed, the correct (and usually default) metaclass is determined, and that metaclass (always) passes three arguments (to its constructor): the class name, the tuple of base classes to inherit from, and the (class) attribute dictionary. Who Are Metaclass Users?To many, the subject of metaclasses belongs in the realm of the theoretical or pure object-oriented thinking and has no place in everyday programming. To some extent that is true; however, the most important thing to keep in mind is that the end consumers of metaclasses are programmers themselves, not application users. You can define metaclasses that "force" programmers to implement solution classes in specific ways, which can either simplify their work or make them program to a target specification. When Are Metaclasses Created?Metaclasses are created for the situations described just above, when you want to change the default behavior of how classes can and are created. Most Python users will not be creating or explicitly using metaclasses. The standard behavior in creating new-style or classic classes is to just take the default behavior by using the system-supplied metaclasses. In most cases, users will not even be aware that metaclasses are providing the templated default behavior of class creation (or metaclass instantiation). Although metaclasses will not be created on a regular basis, let us take a look at a simple example below. (More examples can be found in the documents listed at the end of this subsection.) Metaclass Example 1The first example of metaclasses we are presenting here is (hopefully) very simple. It does not do anything at all except timestamp when a class is created using the metaclass. (As you know now, it happens when the class is created.) Take a look at the following script. It contains print statements scattered throughout so that you can track the events as they occur: #!/usr/bin/env python from time import ctime print '*** Welcome to Metaclasses!' print '\tMetaclass declaration first.' class MetaC(type): def __init__(cls, name, bases, attrd): super(MetaC, cls).__init__(name, bases, attrd) print '*** Created class %r at: %s' % ( name, ctime()) print '\tClass "Foo" declaration next.' class Foo(object): __metaclass__ = MetaC def __init__(self): print '*** Instantiated class %r at: %s' % ( self.__class__.__name__, ctime()) print '\tClass "Foo" instantiation next.' f = Foo() print '\tDONE' If we run this script, we get the following output: *** Welcome to Metaclasses! Metaclass declaration first. Class "Foo" declaration next. *** Created class 'Foo' at: Tue May 16 14:25:53 2006 Class "Foo" instantiation next. *** Instantiated class 'Foo' at: Tue May 16 14:25:53 2006 DONE Once you are comfortable with the fact that a class declaration actually causes some work to be done, then you are well under way. Metaclass Example 2In this second example, we are going to create a metaclass that forces programmers to supply a __str__() method in their classes so that their users can see something more useful than the generic Python object string (< object object at id>) we saw earlier in this chapter. Our metaclass will also (strongly) suggest users override __repr__() if they have not done that either, but it is only a warning. Not implementing __str__() will result in a TypeError exception being thrown, forcing users to create a special method with that name. Here is the code for the metaclass: from warnings import warn class ReqStrSugRepr(type): def __init__(cls, name, bases, attrd): super(ReqStrSugRepr, cls).__init__( name, bases, attrd) if '__str__' not in attrd: raise TypeError( "Class requires overriding of __str__()") if '__repr__' not in attrd: warn( 'Class suggests overriding of __repr__()\n', stacklevel=3) We will create three example classes that use our metaclass, one that overrides both __str__() and __repr__() special methods (Foo), one that only implements the __str__() special method (Bar), and one that implements neither (FooBar), an error situation. The full application is presented here as Example 13.10. Example 13.10. Metaclass Example (meta.py)
Running this script, we get the following output: $ python meta.py *** Defined ReqStrSugRepr (meta)class *** Defined Foo class sys:1: UserWarning: Class suggests overriding of __repr__() *** Defined Bar class Traceback (most recent call last): File "meta.py", line 43, in ? class FooBar(object): File "meta.py", line 12, in __init__ raise TypeError( TypeError: Class requires overriding of __str__() Note how we got past declaring Foo without incident. With Bar, we received the warning for not implementing __repr__(), and FooBar did not pass the security check, hence the reason why the application failed to make it to the (final) print statement. Another important thing to note is that we did not even create any instances of the test classes... they are not even part of our picture. However, keep in mind that those classes themselves are instances of our metaclass. This is but one example of the power of metaclasses. There are many more examples online and in the Python documentation, PEPs 252 and 253, the What's New in Python 2.2 document, and Guido van Rossum's essay entitled, "Unifying Types and Classes in Python 2.2." You can find a link to that document from the main Python release page for 2.2.3. |
value
None
EAN: 2147483647
Pages: 334