Numeric Conversions
OOo Basic tries to convert arguments to an appropriate type before performing an operation. However, it is safer to explicitly convert data types using conversion functions, as presented in this chapter, than to rely on the default behavior, which may not be what you want. When an Integer argument is required and a floating-point number is provided, the default behavior is to round the number. For example, 16.8 MOD 7 rounds 16.8 to 17 before performing the operation. The Integer division operator, however, truncates the operands. For example, "Print 4 \ 0.999" truncates 0.999 to 0, causing a division-by-zero error.
There are many different methods and functions to convert to numeric types. The primary conversion functions convert numbers represented as strings based on the computer's locale. The conversion functions in Table 4 convert any string or numeric expression to a number. String expressions containing hexadecimal or octal numbers must represent them using the standard OOo Basic notation. For example, the hexadecimal number 2A must be represented as "&H2A".
| Function | Type | Description |
|---|---|---|
| CByte(expression) | Byte | Round the String or numeric expression to a Byte. |
| CInt(expression) | Integer | Round the String or numeric expression to the nearest Integer. |
| CLng(expression) | Long | Round the String or numeric expression to the nearest Long. |
| CDbl(expression) | Double | Convert a String or numeric expression to a Double. |
| CSng(expression) | Single | Convert a String or numeric expression to a Single. |
The functions that return a whole number all have similar behavior. Numeric expressions are rounded rather than truncated. A string expression that does not contain a number evaluates to zero. Only the portion of the string that contains a number is evaluated, as shown in Listing 10 .
 |
Print CInt(12.2) ' 12 Print CLng("12.5") ' 13 Print CInt("xxyy") ' 0 Print CLng("12.1xx") ' 12 Print CInt(-12.2) '-12 Print CInt("-12.5") '-13 Print CLng("-12.5xx") '-13 | |
CLng and CInt have similar, but not identical, behavior for different types of overflow conditions. Decimal numbers in strings that are too large cause a run-time error. For example, CInt("40000") and CLng("999999999999") cause a run-time error, but CLng("40000") does not. CLng never causes an overflow if a hexadecimal or octal number is too large; it just silently returns zero without complaint. CInt, however, interprets hexadecimal and octal numbers as a Long and then converts them to a String. The result is that a valid Long generates a run-time error when it is converted to an Integer. A hexadecimal value that is too large to be valid returns zero with no complaints and then is cast to an Integer (see Listing 11 ).
| |
Print CLng("&HFFFFFFFFFFE") '0 Overflow on a Long Print CInt("&HFFFFFFFFFFE") '0 Overflow on a Long then convert to Integer Print CLng("&HFFFFE") '1048574 Print CInt("&HFFFFE") 'Run-time error, convert to Long then overflow | |
The code in Listing 12 converts numerous hexadecimal numbers to a Long using CLng. See Table 5 for an explanation of the output in Listing 12.
| Input | CLng Output | Explanation |
|---|---|---|
| F | 15 | Correct hexadecimal value. |
| FF | 255 | Correct hexadecimal value. |
| FFF | 4095 | Correct hexadecimal value. |
| FFFF | 65535 | Correct hexadecimal value. |
| FFFFF | 1048575 | Correct hexadecimal value. |
| FFFFFF | 16777215 | Correct hexadecimal value. |
| FFFFFFF | 268435455 | Correct hexadecimal value. |
| FFFFFFFF | -1 | Correct hexadecimal value. A Long supports eight hexadecimal digits. |
| FFFFFFFFF |
| Overflow returns zero; this is nine hexadecimal digits. |
| E | 14 | Correct hexadecimal value. |
| FE | 254 | Correct hexadecimal value. |
| FFE | 4094 | Correct hexadecimal value. |
| FFFE | 65534 | Correct hexadecimal value. |
| FFFFE | 1048574 | Correct hexadecimal value. |
| FFFFFE | 16777214 | Correct hexadecimal value. |
| FFFFFFE | 268435454 | Correct hexadecimal value. |
| FFFFFFFE | -2 | Correct hexadecimal value. A Long supports eight hexadecimal digits. |
| FFFFFFFFE |
| Overflow returns zero; this is nine hexadecimal digits. |
| |
Sub ExampleCLngWithHex Dim s$, i% Dim v() v() = Array("&HF", "&HFF", "&HFFF", "&HFFFF",_ "&HFFFFF", "&HFFFFFF", "&HFFFFFFF", "&HFFFFFFFF",_ "&HFFFFFFFFF",_ "&HE", "&HFE", "&HFFE", "&HFFFE",_ "&HFFFFE", "&HFFFFFE", "&HFFFFFFE", "&HFFFFFFFE",_ "&HFFFFFFFFE") For i = LBound(v()) To UBound(v()) s = s & "CLng(" & v(i) & ") = " & CLng(v(i)) & CHR$(10) Next MsgBox s End Sub | |
When writing numbers, you don't need to include leading zeros. For example, 3 and 003 are the same number. A Long Integer can contain eight hexadecimal digits. If only four digits are written, you can assume there are leading zeros. When the hexadecimal number is too large for a Long, a zero is returned. The negative numbers are just as easily explained. The computer's internal binary representation of a negative number has the first bit set. The hexadecimal digits 8, 9, A, B, C, D, E, and F all have the high bit set when represented as a binary number. If the first hexadecimal digit has the high bit set, the returned Long is negative. A hexadecimal number is positive if it contains fewer than eight hexadecimal digits, and it is negative only if it contains eight hexadecimal digits and the first digit is 8, 9, A, B, C, D, E, or F.
The CByte function has the same behavior as CInt and CLng, albeit with a few caveats. The return type, Byte, is interpreted as a character unless it is explicitly converted to a number. A Byte is a Short Integer that uses only eight bits rather than the 16 used by an Integer.
Print CByte("65") 'A has ASCII value 65 Print CInt(CByte("65xx")) '65 directly converted to a number. | Compatibility | Visual Basic contains more functions, such as CCur, to convert to the Currency type, and CVar to convert to a Variant. These are scheduled to be supported in OOo 2.0. There are differences in the whole number types. For example, although Clnt returns an Integer in both languages, an Integer in VB .NET is equivalent to an OOo Basic Long. The rounding rules are different in VB: Numbers are rounded to the nearest even number when the decimal point is exactly 0.5; this is called IEEE rounding. |
The functions that return a floating-point number all have similar behavior. Numeric expressions are converted to the closest representable value. Strings that contain non-numeric components generate a run-time error. For example, CDbl("13.4e2xx") causes a run-time error. CDbl and CSng both generate a run-time error for hexadecimal and octal numbers that are too large.
Print CDbl(12.2) ' 12.2 Print CSng("12.55e1") ' 125.5 Print CDbl("-12.2e-1")'-1.22 Print CSng("-12.5") '-12.5 Print CDbl("xxyy") ' run-time error Print CSng("12.1xx") ' run-time error The functions CDbl and CSng both fail for string input that contains non-numeric data; the Val function does not. Use the Val function to convert a string to a Double that may contain other characters. The Val function looks at each character in the string, ignoring spaces, tabs, and new lines, stopping at the first character that isn't part of a number. Symbols and characters often considered to be parts of numeric values, such as dollar signs and commas, are not recognized. The function does, however, recognize octal and hexadecimal numbers prefixed by &O (for octal) and &H (for hexadecimal).
| Note | The treatment of spaces by the Val function is different than the other functions. For example, Val(" 12 34") returns the number 1234; CDbl and CSng generate a run-time error, and Clnt returns 12 for the same input. |
| Warning |
| |
Print Val(" 12 34") '1234 Print Val("12 + 34") '12 Print Val("-1.23e4") '-12300 Print Val(" &FF") '0 Print Val(" &HFF") '255 Print Val("&HFFFF") '-1 Print Val("&HFFFE") '-2 Print Val("&H3FFFE") '-2, yes, it really converts this to -2 | |
As of version 1.1.1, the behavior of the Val function while recognizing hexadecimal or octal numbers is strange enough that I call it a bug. Internally, hexadecimal and octal numbers are converted to a 32-bit Long Integer and then the least significant 16 bits are converted to an Integer. This explains why in Listing 13 the number H3FFFE is converted to -2, because only the least significant 16 bits are recognized-in case you forgot, this means the rightmost four hexadecimal digits. This strange behavior is demonstrated in Listing 14 . The output is explained in Table 6 .
| Input | Output | Explanation |
|---|---|---|
| F | 15 | Hexadecimal F is 15. |
| FF | 255 | Hexadecimal FF is 255. |
| FFF | 4095 | Hexadecimal FFF is 4095. |
| FFFF | -1 | Hexadecimal FFFF is -1 for a 16-bit (two-byte) integer. |
| FFFFF | -1 | Only the rightmost four bytes are recognized. |
| E | 14 | Hexadecimal E is 14. |
| FE | 254 | Hexadecimal FE is 254. |
| FFE | 4094 | Hexadecimal FFE is 4094. |
| FFFE | -2 | Hexadecimal FFFE is -2 for a 16-bit (two-byte) integer. |
| FFFFE | -2 | Only the rightmost four bytes are recognized; this is five bytes. |
| FFFFFE | -2 | Only the rightmost four bytes are recognized; this is six bytes. |
| FFFFFFE | -2 | Only the rightmost four bytes are recognized; this is seven bytes. |
| FFFFFFFE | -1 | If more than seven hexadecimal digits are present, -1 is returned. |
| 111111111 | -1 | If more than seven hexadecimal digits are present, -1 is returned. |
| |
Sub ExampleValWithHex Dim s$, i% Dim 1 As Long Dim v() v() = Array("&HF", "&HFF", "&HFFF", "&HFFFF",_ "&HFFFFF", "&HFFFFFF", "&HFFFFFFF", "&HFFFFFFFF",_ "&HFFFFFFFFF",_ "&HE", "&HFE", "&HFFE", "&HFFFE",_ "&HFFFFE", "&HFFFFFE", "&HFFFFFFE", "&HFFFFFFFE",_ "&HFFFFFFFFE") For i = LBound(v()) To UBound(v()) s = s & "Val(" & v(i) & ") = " & Val(v(i)) & CHR$(10) Next 1 = "&H" & Hex(-2) s = s & CHR$(10) & "Hex(-1) _ " & Hex(-1) & CHR$(10) s = s & "Hex(-2) = " & Hex(-2) & CHR$(10) s = s & "1 = &H" & Hex(-2) & " ==> " & 1 & CHR$(10) MsgBox s End Sub | |
To summarize the behavior of how Val works with hexadecimal numbers:
-
Return -1 if more than seven hexadecimal digits are present.
-
If more than four and less than eight hexadecimal digits are present, use the least significant four and interpret them as a 16-bit integer.
-
If four or fewer hexadecimal digits are present, interpret them as a 16-bit integer.
| Bug | The Val function interprets a hexadecimal or octal number as a 16-bit Integer rather than a 32-bit Long before converting to a Double, but it accepts more digits than are allowed for an Integer but not as many as are allowed for a Long. Avoid this by using CLng to convert to a Long and then convert this to a Double rather than using Val to convert hexadecimal and octal numbers represented as a String to a Double. |
Use the functions CByte, CInt, CLng, CSng, and CDbl to convert a number, string, or expression to a specific numeric type. Use the functions Int and Fix to remove the decimal portion and return a Double. A string expression that does not contain a number evaluates to zero. Only the portion of the string that contains a number is evaluated. See Table 7 .
| Function | Type | Description |
|---|---|---|
| Int | Double | Round the number toward negative infinity. |
| Fix | Double | Chop off the decimal portion. |
The functions Int and Fix differ only in their treatment of negative numbers. Fix always discards the decimal, which is equivalent to rounding toward zero. Int, on the other hand, rounds toward negative infinity. In other words, "Int(12.3)" is 12 and "Int(-12.3)" is -13.
Print Int(12.2) ' 12 Print Fix(12.2) ' 12 Print Int("12.5") ' 12 Print Fix("12.5") ' 12 Print Int("xxyy") ' 0 Print Fix("xxyy") ' 0 Print Int(-12.4) '-13 Print Fix(-12.4) '-12 Print Fix("-12.1xx")'-12 Print Int("-12.1xx")'-13 | Warning | The help included with OOo is misleading when it states that the fractional part is rounded. It should say that it is rounded toward negative infinity. For example, 12.5 rounds to 12 but -12.5 rounds to -13. |
EAN: 2147483647
Pages: 203