Core Technologies / March 1995

How To Bruise an Integer

Tom R. Halfhill

The "bruised," or truncated, integers Vaughan Pratt describes are numbers that should be integers and may appear as integers when displayed on the screen but internally are represented as floating-point values that fall just shy of a whole number. To see how this happens, try the following experiment using the Calculator program in Windows 3.1. This works on any PC, not just a Pentium.

1. Open the Calculator and select scientific mode from the View menu. Select decimal mode by clicking the button marked DEC.

2. Enter the integer 3.

3. Switch to binary mode by clicking the button marked BIN. Notice that Calculator correctly displays 11, the binary equivalent of 3.

4. Switch back to decimal mode and clear the display.

5. Enter 4.1 minus 1.1. Notice what appears to be the correct answer, 3.

6. Switch to binary mode. Calculator incorrectly displays 10, which is the binary equivalent of 2, not 3.

7. Switch back to decimal mode. The display now reads 2, not 3.

What happened? The floating-point subtraction in step 5 appeared to yield the correct result, but internally the number 3 was stored as a floating-point value that didn't quite equal 3 (probably 2.9999999...). When this "bruised integer" was converted to binary, Calculator evidently rounded down instead of up, causing an error in the displayed result. Microsoft corrected this problem in a new version of Calculator posted on various on-line services in January. However, the main point here is that the Calculator program illustrates how integer bruising occurs.

Copyright 1994-1998 BYTE

Return to Tom's BYTE index page