Encoding
| A computer stores text as a series of numbers. For each letter, punctuation mark, or space there is a corresponding number, called a code point, which represents that letter. An encoding is simply a system that is used to identify characters using numbers. Binary data can be encoded as well, and later on in this chapter you will learn how to encode binary data using REALbasic's Base64 classes. Encoding has been encountered previously in this book, but only superficially; however, it is a crucial part of developing programs, especially ones that run on multiple platforms, because you will no doubt be confronted with the full range of encoding possibilities. As it turns out, understanding character encoding is an important part of being able to make effective use of XML. In this section, I will discuss REALbasic's encoding classes and related tools and share a sample utility application I have used to explore encoding on REALbasic. There are a lot of different encodings and if you do an appreciable amount of working with text, you will inevitably run into encoding headaches. One of the earliest encodings is ASCII (American Standard Code for Information Interchange, first standardized in 1963), which was limited to a 7-bit character set, or 127 characters. The first 32 characters, 031, are control characters, and 32 to 126 are the characters that make up the basic American-English alphabet. Following is a list of all the printable ASCII characters, in numeric order: !"#$%&'()*+,-./0123456789:;<=>? @ ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_ ` abcdefghijklmnopqrstuvwxyz{|}~ASCII is fine if you speak English and need only 127 different characters, but not everyone in the world is English and not everyone in the world speaks English, so it quickly became evident that either additional encodings were required, or a new approach to encoding altogether would be needed. At first, a range of encodings emerged. Both Macintosh and Windows computers used 1-byte character encodings, which provided space for 256 characters. Although both systems shared a common ASCII heritage, the code points above 127 represented different characters on each platform. Macintosh used MacRoman (plus many variants for different languages) and Windows used Latin-1 (ISO 8859-1). Since that time, a new standard has emerged (or perhaps more accurately, is emerging) that rationalizes character encoding and that allocates a large enough pool of code points to represent all languages. The standard is called Unicode, and it is managed by the Unicode Consortium. The first Unicode standard was first published in 1991. Unicode uses up to four bytes to represent a character, and there is room in the standard for 1,114,112 code points. These code points are organized into 17 planes, each one representing 65536 code points (216). The first plane, plane "0", is called the Basic Multilingual Plane (BMP); this is the most commonly used plane, where code points have been assigned to a large portion of modern languages. There is a standard approach to representing Unicode code points, which consists of "U+", followed by a hexadecimal digit. The range of code points in the BMP is U+0000 to U+FFFF (Unicode assigns code points beyond the BMP up to U+10FFFF, and the original specification allowed for ranges up to U+7FFFFFFF). The first 256 code points in the BMP are identical to Latin-1, which also means that the first 127 code points are identical to ASCII. The Unicode standard uses several formats to encode code points. They come in two campsUCS, the Universal Character Set, and UTF, the Unicode Transformation Format. UCSUniversal Character Set (UCS-2, UCS-4)The Universal Character Set uses either 2 bytes or 4 bytes to represent characters. In UCS-2, every character is represented by a 2-byte code point (which, of necessity, limits the number of characters available and does not represent the complete range of Unicode). UCS-4, on the other hand, uses 4 bytes. There are two problems with UCS ("problems" is probably not the right wordbut these are the reasons that UCS is not used in practice as much as UTF). First off, early non-Unicode character sets (ASCII and LATIN-1) used 8 bytes. Using UCS means that legacy documents would have to be converted to either 2-byte or 4-byte formats to be viewed. Second, using either 2 bytes or 4 bytes for all characters means that your text will take up a lot more space than it would if you were able to use 1 byte for some characters, 2 bytes for others, and so on, which is exactly what UTF does. UTFUnicode Transformation Format (UTF-8, UTF-16, UTF-32)REALbasic supports UTF-8 and UTF-16 as native formats. There are also two additional UTF formats, UTF-7 and UTF-32. In UTF, characters are represented by code points of varying sizes, and this is what differentiates UTF from UCS. UTF-8, for example, identifies some characters with 1 byte, others with 2, all the way up to 4 bytes. UTF-16 starts with 2-byte codepoints, but represents some characters with 4 bytes and UTF-32 starts with 4-byte codepoints. The following table shows the range of code points and the number of bytes used by UTF-16 and UTF-8 to represent it. The first three rows represent the BMP.
Byte Order MarkThe byte order mark (BOM) is a character that is placed at the beginning of a file that can used to identify the byte order of the document; byte order is simply a reference to endianness, which is one of those topics that keeps popping up in any kind of cross-platform development. The character in question is supposed to be a zero-width non-breaking space. UCS-2 and UTF-16 are the two Unicode formats that use the BOM for the determination of endianness. When used with UTF-8 and others, it's used to identify the encoding of the file itself. In other words, based on the value of the first four bytes, you can determine the encoding of the stringthat is, if the BOM has been has been set. REALbasic usually makes this determination for you, but in case you want to check directly, here are the values and what they mean:
Converting EncodingsNow that the basics of Unicode have been reviewed, it's time to turn to REALbasic's encoding classes and learn how to use the tools provided to effectively manage character encoding in your application. TextEncoding ClassThe TextEncoding class represents a particular encodingwhether Unicode or some native encoding like MacRoman. TextEncoding.Base as Integer TextEncoding.Code as Integer TextEncoding.Variant as Integer TextEncoding.Format as Integer TextEncoding.InternetName as String The TextEncoding class offers an alternative to the global Chr function discussed earlier in the book. When using the global function, it is assumed that you are using UTF-8, but that may not be what you want. If, for whatever reason, you want to get a character using a codepoint for another encoding, you can use this method on a TextEncoding instance that represents the encoding you want to use: TextEncoding.Chr(codepoint as Integer) as String You can test to see if the encoding of one string is equal to that of another this way: TextEncoding.Equals(otherEncoding as TextEncoding) as Boolean Encodings ObjectThe Encodings object is always available, and it is used to get a reference to a particular TextEncoding object. You can get a reference to the encoding using the encoding's name: Encodings.EncodingName as TextEncodingYou can also get a reference to a particular encoding using the code for that encoding. The code is an integer that represents a particular encoding. This method lets you get a reference to a TextEncoding object by passing the code as an argument. Encodings.GetFromCode(aCode as Integer) as TextEncoding You can get references to characters through encoding, by calling the following method: Encodings.UTF-8.Chr(aCodePoint as Integer) To use the Encodings object, you need to know the names of the available encodings and, optionally, their codes. The following table provides a list of all the encodings recognized by REALbasic's Encodings object and the values associated with each encoding.
After you have a reference to a particular encoding, you can use it to define the encoding of a string or to convert the encoding from one String to another. DefineEncoding FunctionThe DefineEncoding global function is used to apply an encoding to a string whose encoding you already know. In simpler terms, it adds a byte order mark at the beginning of the string so that the encoding can be determined by checking it. DefineEncoding(aString as String, enc as TextEncoding) as String ConvertEncoding FunctionThis global function lets you convert a string from one encoding to another. This is a more recent addition to the language than the TextConverter class and is generally easier to use. It assumes that REALbasic is able to figure out the encoding of the string that is being passed so that it can accurately convert it into the encoding passed in the second argument. ConvertEncoding(aString as String, enc as TextEncoding) as String Some additional global functions can be used, but they have largely been replaced by the methods outlined previously. There is a GetTextConverter function and a GetTextEncoding function, but it is generally simpler to use the Encodings object and ConvertEncoding function outlined earlier. Base64 EncodingBase64 encoding is unrelated to Unicode, but it is quite relevant to our discussion of XML and Internet applications in general. Basically, Base64 is a means of encoding arbitrary binary information into a string of ASCII characters that can then be emailed or otherwise transferred through the Internet. REALbasic provides global functions for your encoding and decoding pleasure: REALbasic.EncodeBase64(aString as String, [linewrap as Integer]) as String REALbasic.DecodeBase64(aString as String, [linewrap as Integer]) as String Consider the following code snippet: Dim s as String s = "ABCs are fun to learn." MsgBox EncodeBase64(s) The MsgBox will display the following: QUJDcyBhcmUgZnVuIHRvIGxlYXJuLg== XML and HTML Entity EncodingXML and HTML are encoded in UTF-8 or UTF-16 by default; if another encoding is used, it must be declared at the beginning of the document. However, because some Internet-related protocols can work only with 8-bit character sets, you sometimes need to encode characters outside that range in a special format. You can refer to specific characters using an XML entity reference. Entity references always start with "&" and end with ";" and every time an XML parser encounters an entity, it tries to expand itwhich means that it tries to replace it with the appropriate replacement text for that entity. An entity reference can refer to a long string of text, either declared within the document or residing outside the document, but the references used for character encoding are automatically expanded. The generic way to refer to a particular character is to use a numeric entity reference, like so: ' This particular reference refers to the apostrophe character. The ampersand is followed by "#", which identifies it as a numeric character reference and the numbers that follow it are decimal representations of a particular code point. In addition to using entities to refer to characters that are outside the 1-byte range, there are also some characters that need to be encoded when you want to refer to them literally because they are used to mark up an XML document. The following table specifies those entities and what they represent:
Detecting Encoding in XML DocumentsIf an XML document does not declare an encoding, it is supposed to be either UTF-8 or UTF-16. (The BOM is optional.) However, despite the fact that they are supposed to declare their encoding, some do not, and others may declare an encoding that is different from what the text is actually encoded in. This is surprisingly common (in my experience, at least), so checking is helpful. XML documents have to start with an XML declaration: <?xml version="1.0" encoding="UTF-8"?> Regardless of the version or encoding, you definitely know that each XML document will start with <?xml. Because you know what those characters have to be, you can deduce the encoding being used by examining the data in the first four positions. In some cases, you are just narrowing down the options, so you need to check the encoding declaration in the XML document as well to make sure it is consistent with what you have found.
|
EAN: 2147483647
Pages: 149