Prev: Use a static form of the EventLog class?
Next: ASP Tab Control
From: Todd Heiks on 7 Apr 2010 18:55
Why is 'ans1' true if double and false for single?

Module Module1
Sub Main()

Dim x As Double / Single
Dim y As Double / Single

Dim ans1 As Boolean
Dim ans2 As Boolean

x = 0.3000000000000001
y = Math.Round(x, 2)

ans1 = y = 0.3
ans2 = Math.Round(y, 2) = 0.3

End Sub
End Module

Thanks


From: Herfried K. Wagner [MVP] on 7 Apr 2010 19:03
Todd --

Am 08.04.2010 00:55, schrieb Todd Heiks:
> Why is 'ans1' true if double and false for single?
>[...]
> Dim x As Double / Single
> Dim y As Double / Single
>
> Dim ans1 As Boolean
> Dim ans2 As Boolean
>
> x = 0.3000000000000001
> y = Math.Round(x, 2)
>
> ans1 = y = 0.3
> ans2 = Math.Round(y, 2) = 0.3

The reason is that the result cannot be represented exactly. It's in
general not recommended to compare floating point values of single and
double precision using the '=' operator.

More information:

(Complete) Tutorial to Understand IEEE Floating-Point Errors
<URL:http://support.microsoft.com/?scid=kb;EN-US;42980>

--
M S Herfried K. Wagner
M V P <URL:http://dotnet.mvps.org/>
V B <URL:http://dotnet.mvps.org/dotnet/faqs/>
 | 
Pages: 1
Prev: Use a static form of the EventLog class?
Next: ASP Tab Control