From: Jeff Johnson on 30 Apr 2010 14:59 You know, the one people post when someone doesn't understand why 22 / 10 yields 2.2000000000000000000001 instead of 2.2. From: dpb on 30 Apr 2010 15:27 Jeff Johnson wrote:> You know, the one people post when someone doesn't understand why 22 / 10 > yields 2.2000000000000000000001 instead of 2.2. > > Goldberg, I presume you mean??? -- From: Patrice on 30 Apr 2010 15:46 Hello, > You know, the one people post when someone doesn't understand why 22 / 10 > yields 2.2000000000000000000001 instead of 2.2. You could try : http://en.wikipedia.org/wiki/Floating_point#Accuracy_problems The short version is that floating point values are coded in a way that favor range over precision. The decimal data type allows to have full control over the precision. -- Patrice From: Jeff Johnson on 30 Apr 2010 16:27 "dpb" wrote in message news:hrfb2o\$ate\$1(a)news.eternal-september.org... >> You know, the one people post when someone doesn't understand why 22 / 10 >> yields 2.2000000000000000000001 instead of 2.2. > Goldberg, I presume you mean??? > > Urk. Is this the one Ken (I think it was Ken...maybe Rick?) used to post all the time? There's a ton of math in the one you linked to, and I think this other one was a bit more...approachable. From: Rick Rothstein on 30 Apr 2010 17:14 > Urk. Is this the one Ken (I think it was Ken...maybe Rick?) used to post > all the time? These are the three links I used to post (the last one being the one already posted)... INFO: Visual Basic and Arithmetic Precision http://support.microsoft.com/default.aspx?scid=http://support.microsoft.com:80/support/kb/articles/Q279/7/55.ASP&NoWebContent=1 (Complete) Tutorial to Understand IEEE Floating-Point Errors http://support.microsoft.com/default.aspx?scid=http://support.microsoft.com:80/support/kb/articles/Q42/9/80.ASP&NoWebContent=1 What Every Computer Scientist Should Know About Floating Point http://docs.sun.com/source/806-3568/ncg_goldberg.html -- Rick (MVP - Excel)