Section 20.3. Translating Your Application

20.3. Translating Your Application

Now that you have a firm grasp on the basics of handling Unicode strings in Turbo-Gears, we can start delving into the process that TurboGears uses to create and maintain translation files.

There are several steps in the translation process, from marking what needs to be translated to creating catalogs of translated strings and making them available to the application. TurboGears uses the standard tool used in the GNU/Open Source world: GNUgettext.

There are several steps in the process:

1. Marking translatable strings in your source code (Python source, Kid/Cheetah templates, etc.)

2. Collecting these strings into so called "message catalog templates"

3. Creating or updating locale-specific message catalogs from the template file

4. Compiling message catalogs

5. Using run-time functions to set up the correct locale and read translations of the desired string

The message catalog is a set of text files that contains all the strings for translations and (usually) references the source file the string comes from. The reference makes it easier to track the changes down the road. The message catalog consists of two different types of files, the "master" catalog or template (with .pot extension) and "locale" catalog files (one per language). Each of these additional catalog files contains the translated strings for its language.

To translate your application's text, you edit each locale catalog file to provide all necessary translations. When you are done with the translations, you compile that locale into a binary representation for efficient run-time lookup.

As your source code evolves, you rerun the string collection process, update the locale catalog files, update translations, and recompile catalogs.

If you think this process sounds painful, you're right. Well, at least you would be right if you had to do it all by hand. While it may seem daunting to translate your application into a few dozen languages, TurboGears does everything it can to make the process as painless as possible by automating away the most tedious work.

20.3.1. Localization of Python Sources

Before we can set up the .pot files, we have to mark all of the translatable strings in our application. We also need to set up functions to return the right string for the user's locale whenever a string is generated that could go out to the user. Luckily for us, we do both of these things the same way!

There is a de facto standard for such things, and TurboGears follows that established standard rather than trying to invent something new. To mark a string for translation, you just use the"underscore" functionwritten as _(). Inside your TurboGears application, _() is a "built in" function, so you do not have to import anything in order to call it. The same is true for Python excerpts in Kid templates. Actually, Kid also has other even simpler localization options that we'll get back to shortly.

When using the _() function, also called the gettext function, sometimes you don't want to translate the text right away, instead you want to defer the results of its evaluation until later, when you know the user's locale. This is typical for declarative statements like the ones you use to define a widget or form:

signup_form = TableForm('signup', submit_text=_("Sign me Up"), fields=[...])

Here, gettext("Sign me Up") is called when the form is defined, not when it is actually rendered in a web page. This would give you a static message that may not be relevant for the user's locale because the same message is sent to all locales. Even worse, when the form was defined, no locale had yet been specified, so TurboGears would fallback to the default.

To overcome this issue, TurboGears uses a special flavor of the gettext function that is sometimes called lazy_gettext, which does not attempt to translate the string immediately, but instead returns a callable that knows which strings to return a translated value for and performs a translation only when needed. This "something" can be evaluated any number of times, and each time it will perform a new lookup for the message and locale specified.

Usually the lazy_gettext TRick is transparent to you as a programmer. But, occasionally the trick can get you in trouble. As a result, it can be helpful to understand what is happening under the hood. When you want to control exactly what happens, TurboGears provides the plain_gettext and lazy_gettext functions, which work as you would expect: plain_gettext TRanslates the message right away, while lazy_gettext returns the magic callable just described.

20.3.2. Localization of Kid Templates

As noted previously, you can use _() inside your Kid template. It requires some extra typing, but works fine, as illustrated below:

<p>${_('Welcome, %s!') % name}</p>

This is the only option if you need to have a parameterized string, as we do in the above example.

To translate static strings in your Kid template, there are some even simpler options.

You can have TurboGears translate all of your templates by using the i18n.run_template_filter = on configuration setting. This setting incurs small performance costs to process a template, but is very straightforward to set up and use.

For instance, in the following Kid excerpt :

<p>Welcome, <em>guest</em>!<br/> Nice to see you again on my site. </p>

The template filter would automatically extract these translatable strings:

'Welcome,' 'guest' '!' 'Nice to see you again on my site.'

Note how the XHTML markup breaks the string and that whitespace is always stripped off.

If you don't want to translate everything, you can use more fine-grained control with the turbogears.i18n.translate helper function. To use it, you add the right py:match processing directive at the top of your Kid template and mark translatable text by adding lang attributes to corresponding nodes. Here is a fully functional example:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/ TR/xhtml1/DTD/xhtml1-transitional.dtd"> <?python from turbogears.i18n import translate ?> <html xmlns="http://www.w3.org/1999/xhtml" xmlns:py="http://purl.org/kid/ns#"     py:extends="'master.kid'">   <translate py:match="item.attrib.has_key('lang')" py:replace="translate(item, 'lang')"/> <body>   <div lang="">     <span>This message will be translated.</span>     <span lang="en">And this one as well.</span>   </div>   <p>This is "normal" text that is not translated.</p> </body>

An empty value in the lang attribute means "current user locale," which is what you want most of the time. You can override the selection of the current user locale by specifying a locale code "en" or "fi" instead. In this case, translation to this locale will be used, ignoring the current user locale (this may be useful, for example, for a site language selector UI).

The command-line version (tg-admin i18n) at least tries to support both approaches to translating Kid templates withstrict-kid-support/loose-kid-support, while Toolbox admi18n simply grabs everything. Unfortunately, neither tool will automatically detect calling gettext (using the _() function) directly from Kid templates.

20.3.3. Auto-Detect User Language Preference

The turbogears.i18n.get_locale() function tries to guess the user's locale. It looks for locale information in several places: in the session, the HTTP Accept-Language header, and finally the i18n.default_locale configuration option. If none of these places contains a locale, TurboGears will fall back to a default locale of "en."

If the default lookup algorithm doesn't suit you (for instance, if you store locale information in the database), you can specify your own algorithm by assigning a lookup function to the i18n.get_locale configuration variable. The function is parameterless and returns a locale name.

Example:

def get_locale_from_db():         user = identity.current.user         if user and user.locale:                 return user.locale         return turbogears.i18n.utils._get_locale() # fallback to default

If you store the user's locale information in session data, use turbogears.i18n set_session_locale. The default key is locale, but you can override that with the i18n.session_key variable. Make sure you have session_filter.on before using it.

20.3.4. Locale-Specific Objects

To localize an application completely, simply translating user interface messages often is not enough.

Different locales often use different representations for things like dates, numbers, currencies, and so forth. All locale-specific data resides in the turbogears.i18n.format module.

These functions also take an optional locale argument; if it is None, the current user locale is used.

These low-level functions might be of help if you are building your own locale libraries.

Higher-level functions solve more complete tasks:

• format_number Formats a number using a locale-specific group separator symbol

• format_decimal Like format_number, handles locale-specific separators, but also handles decimals with a specified precision (decimal symbol is read from the locale information)

• format_currency Formats a number with a fixed precision (two digits)

• parse_number/parse_decimal Counterparts of the format_number/format_decimal functions, reads locale-specific string representations of a number into Python long or float value

• format_date Formats the datetime object to match one of the predefined locale-specific formats (full, long, medium, short) or a custom format template.

20.3.5. Admin i18n Interface

There are two tools in TurboGears that help you solve the mundane problems of dealing with i18n. One is web-based, the admi18n component of the Toolbox. The other tool is a command-line tool: the i18n subcommand of the tg-admin tool.

Both commands work in very much the same manner, so you can pick one based on your personal preferences. Generally speaking, the web interface is easier to use, while the command-line version provides some advanced options not found in the web version (not that these will be of interest to most folks).

20.3.5.1. Toolbox admi18n Tool

The tool includes built-in usage instructions so it should be straightforward to use.

20.3.5.2. tg-admin i18n Command-line Tool

The following sub-commands are available:

• add Creates a message catalog for specified locale

• collect Scans source files to gather translatable strings to a .pot file

• merge Syncs message catalogs in different languages with the master .pot file

• compile Compiles message catalogs (.po to .mo)

• clean Deletes backups and compiled files

Note

You'll need to run tg-admin i18n collect first to grab new strings from the source files. Once you've added new translations, run tg-admin i18n compile to actually see the changes.

The tg-admin i18n collect command processes a project's Python and Kid sources to find strings marked for translation. If a translation file exists, its content is updated with new data and a backup is made.

There are some options related to Kid template processing, run tg-admin i18n -h to get them.

To add another locale to your project, run tg-admin i18n add and supply an appropriate locale code, like es or uk_UA. This will create the locale's directory and the locale's message catalog file which contains translations.

The command tg-admin i18n compile compiles every locale's message catalog file into a machine efficient .mo file.

The command tg-admin i18n merge copies newly-added message strings from the master .pot file to each locale's message catalog, keeping existing translations intact.