Section 20.1. Dealing with Unicode in Python

20.1. Dealing with Unicode in Python

Even though TurboGears provides comprehensive support for internationalization, if you want your TurboGears application to be easily translatable, there are a few things you need to remember. Your application, like any Python program, requires some attention with respect to how you handle Unicode strings.

In the current version of Python there are strings and Unicode strings. The best piece of advice you can get if you are planning to internationalize your application is:

Always use Unicode strings everywhere!

Every string in your application should be a Unicode string. The only time you should ever convert to/from Unicode is at the IO boundaries of the application. So, if you receive a string from an email or other source, convert it to Unicode right away.

If you do this, it greatly reduces the chances that you'll get the infamous UnicodeDecodeError, which happens whenever Python tries to decode a Unicode string into an output format that can't handle all of the Unicode characters in that string.

The default string type in Python is ASCII (because ASCII works everywhere!). It's pretty likely that your Unicode string will contain some characters that can't be decoded into an ASCII output. This is easy to avoid as long as you use Unicode everywhere.

TurboGears libraries all use Unicode by default; but unfortunately, you still have to be careful when interfacing with third-party Python libraries. Many of them are still Unicode-ignorant and can pop up the UnicodeDecodeError you'll learn to hate.

The second most important rule of handling Unicode characters in your Turbo-Gears application is:

Always include strings with non-ASCII characters in your unit tests.

If you have appropriate tests, you'll be able to detect places in which Unicode strings get forced into ASCII right away. With the right tests, you'll find out that an imported library doesn't handle Unicode as soon as you attempt to use it. This is a lot better than finding out that something is broken when a user from Helsinki sends you an email telling you that your application doesn't work.