Prev: What to name an object?
Next: LINQ to XML
From: Lothar Behrens on 30 Apr 2010 10:35
Hi,

I have found a problem in my unit tests that my code does not
calculate exact values.
My code produced 2.8000000000000003 but not 2.7000000000000003. In
both cases
I expected 2.8 / 2.7.

And when I change the calculation I get exact values. Why does this
happen?

I am using Visual Studio 2008 Professional 9.0.30729.1 SP and .NET 3.5
SP1.

Here is a sample from my watch window:

28.000000 * (100.000000 / 100.0000000) / (100.0000000 / 10.0000000)
2.8 double
28.000000 * (100.000000 / 100.0000000) * (10.0000000 / 100.0000000)
2.8000000000000003 double
27.000000 * (100.000000 / 100.0000000) * (10.0000000 / 100.0000000)
2.7 double

Thanks

Lothar
From: Martin Honnen on 30 Apr 2010 11:02
Lothar Behrens wrote:

> I have found a problem in my unit tests that my code does not
> calculate exact values.
> My code produced 2.8000000000000003 but not 2.7000000000000003. In
> both cases
> I expected 2.8 / 2.7.
>
> And when I change the calculation I get exact values. Why does this
> happen?

You get the precision that is possible with that number type
(http://en.wikipedia.org/wiki/Double_precision_floating-point_format)
with binary representation of numbers.
Using decimal instead of double would give you better precision. If you
want or need to use double then round as required.



--

Martin Honnen --- MVP Data Platform Development
http://msmvps.com/blogs/martin_honnen/
From: Lothar Behrens on 30 Apr 2010 14:28
On 30 Apr., 17:02, Martin Honnen <mahotr...(a)yahoo.de> wrote:
> Lothar Behrens wrote:
> > I have found a problem in my unit tests that my code does not
> > calculate exact values.
> > My code produced 2.8000000000000003 but not 2.7000000000000003. In
> > both cases
> > I expected 2.8 / 2.7.
>
> > And when I change the calculation I get exact values. Why does this
> > happen?
>
> You get the precision that is possible with that number type
> (http://en.wikipedia.org/wiki/Double_precision_floating-point_format)
> with binary representation of numbers.
> Using decimal instead of double would give you better precision. If you
> want or need to use double then round as required.
>
> --
>
>         Martin Honnen --- MVP Data Platform Development
>        http://msmvps.com/blogs/martin_honnen/

I don't expect numbers like 2.8000000000000003 if I effectively divide
28 through 10.
If I divide 10 through 3, then I expect as much precision as the data
type will provide.

I have compared the code with C++ (not .NET) and there I also have
additionally compared
with using float. That would give me a more expectable result.

Rounding is an option, but only with the result of a calculation to
avoid rounding errors.
Actually I am using rounding in my unittests to get these passing.

I have to play a bit more with floating point values and
calculation :-)
Also I have to play with the datatypes I have choosen in my DB model.
(LinqToSQL / SQLSERVER)

Thanks

Lothar
From: Jeff Johnson on 30 Apr 2010 14:58
"Lothar Behrens" <lothar.behrens(a)lollisoft.de> wrote in message
news:ad69d79e-22a4-4a0a-b442-14ab8c2d0995(a)r34g2000yqj.googlegroups.com...

> I don't expect numbers like 2.8000000000000003 if I effectively divide
> 28 through 10.

Then you clearly don't understand IEEE floating-point numbers. Don't feel
bad, most people don't. Even seemingly innocuous/"obvious" calculations can
lead to these odd values. Dividing by 10 when representing numbers in binary
is no "safer" than dividing by Pi. There's a URL that people post in another
group I read to help explain IEEE numbers to people who are having the same
problems you are. If I can find it I'll post it here.


From: Jeff Johnson on 3 May 2010 09:37
"Jeff Johnson" <i.get(a)enough.spam> wrote in message
news:OhNYmcJ6KHA.1932(a)TK2MSFTNGP05.phx.gbl...

> Then you clearly don't understand IEEE floating-point numbers. Don't feel
> bad, most people don't. Even seemingly innocuous/"obvious" calculations
> can lead to these odd values. Dividing by 10 when representing numbers in
> binary is no "safer" than dividing by Pi. There's a URL that people post
> in another group I read to help explain IEEE numbers to people who are
> having the same problems you are. If I can find it I'll post it here.

Found one: http://support.microsoft.com/kb/42980/EN-US/


 | 
Pages: 1
Prev: What to name an object?
Next: LINQ to XML