The unicodedata Module

(New in 2.0) The unicodedata module contains Unicode character properties, such as character categories, decomposition data, and numerical values. Its use is shown in Example 8-3.

Example 8-3. Using the unicodedata Module

File: unicodedata-example-1.py

import unicodedata

for char in [u"A", u"-", u"1", u"N{LATIN CAPITAL LETTER O WITH DIAERESIS}"]:
 print repr(char),
 print unicodedata.category(char),
 print repr(unicodedata.decomposition(char)),

 print unicodedata.decimal(char, None),
 print unicodedata.numeric(char, None)

u'A' Lu '' None None
u'-' Pd '' None None
u'1' Nd '' 1 1.0
u'303226' Lu '004F 0308' None None

Note that in Python 2.0, properties for CJK ideographs and Hangul syllables are missing. This affects characters in the range 0x3400-0x4DB5, 0x4E00-0x9FA5, and 0xAC00-D7A3. The first character in each range has correct properties, so you can work around this problem by simply mapping each character to the beginning:

def remap(char):
 # fix for broken unicode property database in Python 2.0
 c = ord(char)
 if 0x3400 <= c <= 0x4DB5:
 return unichr(0x3400)
 if 0x4E00 <= c <= 0x9FA5:
 return unichr(0x4E00)
 if 0xAC00 <= c <= 0xD7A3:
 return unichr(0xAC00)
 return char

This bug has been fixed in Python 2.1.