Landing : Athabascau University
  • Blogs

All site blogs

Latest comments

  • In fact, calculators have the same floating point limitations as computers, which makes sense, because calculators are simple computers. Try adding 0.1 to 1,000,000,000 and you'll get 1,000,000,000, not 1,000,000,000.1 (unless your calculator uses 64-bit numbers, which can precisely display up to about 18 digits); or try cos(0.00001) and you'll get 1, not 0.99999999995. They seem to be more accurate because they can limit their outputs. It's worth noting that having a hardware FPU does not guarantee greater precision than using a floating point library, as Arduinos do. The highest precision operations are actually done in software. Take a look at Java's BigDecimal for a great example and explanation. 

    You are correct in thinking that there could be a loss of precision when working with base-10 numbers in a binary system. The IEEE 754 standard defines 5 binary and 3 decimal floating point formats, where the latter can store decimal numbers exactly. Since the libm.a floating point library (Arduino/avr-gcc, etc.) likely uses the "binary32" format, it does not represent decimal number exactly, and is accurate to about 7 digits. Any format is no more precise than the value of the least significant bit ("LSB") (big-endian only; MSB value for little-endien) in the significand. For binary32 numbers, this is 2^-23 times the exponent, which is related to its magnitude. A quick search found a library that may be able to introduce the decimal32/64/128 formats to an Arduino platform, [here]: (it is for the ESP8266, a 32-bit Xtensa system, however, so it wouldn't work out of the box, if at all, on the 8-bit AVR/etc. systems we are using).

    Tyler Lucas November 15, 2017 - 1:20pm