Scientific to Decimal
in [Visual Basic]
|
From: Nobody on 1 Dec 2009 18:20 "Webbiz" <nospam(a)noway.com> wrote in message news:ha6bh5ps7b8qnab20lbqpultrc567oq2lm(a)4ax.com... > Then why are my Single values, showing in Watch as .5938 for example, > showing as a Scientific Notation value after some math and assigned to > another Single variable? > > If I send the original single value to the text box, it will show up > as .5938. But do some subtraction, for example, of two of these > Singles and assign to another Single, or directly to the textbox, and > it displays as Scentific Notation? Because the way numbers are stored in binary form(fraction+Exponent), there can be two binary forms for the same decimal number. Kind of like saying 100/500~101/499. This has implications when testing if two values are equal, which may not be what you expect. This is true for all languages, not just VB. To solve this, use something like "If Abs(x-y)<=0.01 Then" to test for equality, or write that as IsEqual() function for easy reading. See this article and related articles in the References section: INFO: Visual Basic and Arithmetic Precision http://support.microsoft.com/kb/279755 See this sample which prints the byte values so you can see the difference. It's based on the numbers that you have posted in an earlier reply: Option Explicit Private Declare Sub CopyMemory Lib "kernel32" Alias "RtlMoveMemory" ( _ Destination As Any, Source As Any, ByVal Length As Long) Private Sub Form_Load() Dim x As Single Dim y As Single x = 0.5001 y = 0.4963 y = x - y Debug.Print y PrintBinaryValue y x = 0.0038 Debug.Print x PrintBinaryValue x If x = y Then Debug.Print "x = y" Else Debug.Print "x <> y" End If If Abs(x - y) < 0.01 Then Debug.Print "Almost equal" Else Debug.Print "Not equal" End If End Sub ' Print "x" in binary form Private Sub PrintBinaryValue(ByRef x As Single) Dim b(0 To 3) As Byte Dim i As Long CopyMemory b(0), x, 4 For i = 0 To 3 Debug.Print Hex(b(i)); " "; Next Debug.Print End Sub Output: 3.800005E-03 80 9 79 3B 0.0038 6C 9 79 3B x <> y Almost equal
From: Webbiz on 1 Dec 2009 18:49 On Tue, 1 Dec 2009 17:56:08 -0500, "Jim Mack" <jmack(a)mdxi.nospam.com> wrote: >Webbiz wrote: > >> Then why are my Single values, showing in Watch as .5938 for >> example, showing as a Scientific Notation value after some math and >> assigned to another Single variable? >> >> If I send the original single value to the text box, it will show up >> as .5938. But do some subtraction, for example, of two of these >> Singles and assign to another Single, or directly to the textbox, >> and it displays as Scentific Notation? > >A textbox can't display a single, it can only display strings. And >anything you show in Watch or Debug etc is a also string. > >If you want to control how something appears on the screen (i.e. in >the string domain), use Format. Jim, I know this. What I'm saying is that the value is changing from a regular decimal value to Scientific Notation. So when I go to display it in a text box, it shows as Scientific Notation. Yes, format is what will need to be used, and that's still in discussion in this thread. However, my question is WHY is it changing to Sci-Not? Thanks. Webbiz
From: Webbiz on 1 Dec 2009 18:52 Thanks. Reading article now. :) On Tue, 1 Dec 2009 18:20:08 -0500, "Nobody" <nobody(a)nobody.com> wrote: >"Webbiz" <nospam(a)noway.com> wrote in message >news:ha6bh5ps7b8qnab20lbqpultrc567oq2lm(a)4ax.com... >> Then why are my Single values, showing in Watch as .5938 for example, >> showing as a Scientific Notation value after some math and assigned to >> another Single variable? >> >> If I send the original single value to the text box, it will show up >> as .5938. But do some subtraction, for example, of two of these >> Singles and assign to another Single, or directly to the textbox, and >> it displays as Scentific Notation? > >Because the way numbers are stored in binary form(fraction+Exponent), there >can be two binary forms for the same decimal number. Kind of like saying >100/500~101/499. This has implications when testing if two values are equal, >which may not be what you expect. This is true for all languages, not just >VB. To solve this, use something like "If Abs(x-y)<=0.01 Then" to test for >equality, or write that as IsEqual() function for easy reading. See this >article and related articles in the References section: > >INFO: Visual Basic and Arithmetic Precision >http://support.microsoft.com/kb/279755 > >See this sample which prints the byte values so you can see the difference. >It's based on the numbers that you have posted in an earlier reply: > >Option Explicit > >Private Declare Sub CopyMemory Lib "kernel32" Alias "RtlMoveMemory" ( _ > Destination As Any, Source As Any, ByVal Length As Long) > >Private Sub Form_Load() > Dim x As Single > Dim y As Single > > x = 0.5001 > y = 0.4963 > > y = x - y > Debug.Print y > PrintBinaryValue y > > x = 0.0038 > Debug.Print x > PrintBinaryValue x > > If x = y Then > Debug.Print "x = y" > Else > Debug.Print "x <> y" > End If > > If Abs(x - y) < 0.01 Then > Debug.Print "Almost equal" > Else > Debug.Print "Not equal" > End If > >End Sub > >' Print "x" in binary form >Private Sub PrintBinaryValue(ByRef x As Single) > Dim b(0 To 3) As Byte > Dim i As Long > > CopyMemory b(0), x, 4 > For i = 0 To 3 > Debug.Print Hex(b(i)); " "; > Next > Debug.Print >End Sub > > > >Output: > > 3.800005E-03 >80 9 79 3B > 0.0038 >6C 9 79 3B >x <> y >Almost equal > >
From: Nobody on 1 Dec 2009 19:09 "Nobody" <nobody(a)nobody.com> wrote in message news:uR6jXztcKHA.1028(a)TK2MSFTNGP06.phx.gbl... > Output: > > 3.800005E-03 > 80 9 79 3B > 0.0038 > 6C 9 79 3B > x <> y > Almost equal Here is a break down of byte values for the first number, which was a result of a calculation: > 80 9 79 3B In the correct order, this would be: 3B790980 Based on this article: http://en.wikipedia.org/wiki/Single_precision_floating-point_format Breaking to basic components(all decimal): Sign: 0 Exponent: 118 Fraction: 7932288 Equation: (-1)^Sign * 2 ^ (Exponent - 127) * (1 + Fraction / (2^23)) So the number would be(Using Windows Calculator): 0.0038000047206878662109375 And for the other number, which was entered manually in code(x = 0.0038): > 6C 9 79 3B In the correct order, this would be: 3B79096C Breaking to basic components(all decimal): Sign: 0 Exponent: 118 Fraction: 7932268 So the number would be: 0.003800000064074993133544921875 It's very close to first one. Here are the numbers again, in Hex and decimal format to the precision that Windows Calculator can handle: 3B790980 3B79096C 0.0038000047206878662109375 0.003800000064074993133544921875
From: Rick Rothstein on 1 Dec 2009 19:41
That's a problem then... once the numbers are placed in a numeric variable, they lose all their formatting (numbers are just numbers, they have no format). The problem with using a value from the list of numbers is that none of your values might be "full". For example, what if all your numbers are **supposed** to be shown to 3 decimal points, but all your values just happen to, by rare chance, have only one or two decimal places... if you try using one of these numbers in my expression, you will not get 3-decimal places shown. You also can't just select a number from the list because you might not get a "full" number either. Where are these numbers coming from? Perhaps you can go back to the source to get a representative formatted text representation of the number and use that. -- Rick (MVP - Excel) "Webbiz" <nospam(a)noway.com> wrote in message news:kh6bh5hd7f52vhq97kkumhgoerl125kgl1(a)4ax.com... > On Tue, 1 Dec 2009 16:49:17 -0500, "Rick Rothstein" > <rick.newsNO.SPAM(a)NO.SPAMverizon.net> wrote: > >>>>Let Num be **any** number from the list of numbers that you are >>>>processing >>>>and let Avg be the calculated average... >>>> >>>>Num = <<any value from list>> >>>>Avg = <<calculated average>> >>>>' This line formats the calculated value as per a number in the list >>>>Avg = Format(Avg, "0." & String(Len(Num) - InStr(Num, "."), "0")) >>> >>> >>> Ah, tested but didn't work out because the blasted values in the array >>> are in Scientific Notation. VB had to have done this because the >>> actual values when loaded into the array were not in SN. >>> >>> What causes this and how to keep VB from doing it this way? >>> >>> Here is where this is happening: >>> >>> Values(1) = CurrPriceData.High - CurrPriceData.Low >>> >>> Values() is type Single. >>> >>> CurrPriceData.High = 0.5001 >>> CurrPriceData.Low = 0.4963 >>> >>> Values(1) = 3.800005E-03 >> >>Try using one of the CurrPriceData.High or CurrPriceData.Low values for >>Num >>value in my expression as they seem like they might be in the right format >>(String as opposed to Single I'm guessing). If these values are really >>numeric values and not text String representations of those numerical >>values, then can you see the formatted number from the original source? If >>so, this is what you want to assign to the Num variable. Basically, you >>need >>a **formatted** value so you can see how many decimal places there are... >>if >>these are numerical values, then you will never know if the value you >>chose >>would have ended in zeroes when originally formatted (since numbers do not >>have format, they can end with zeroes after the decimal point, only >>formatted String representations of these numbers can do that). > > > The values are stored in DataArray as Singles > > Type DataArray > dDate As Date > Open As Single > High As Single > Low As Single > Close As Single > End Type > > So these are both Singles... > > CurrPriceData.High = 0.5001 > CurrPriceData.Low = 0.4963 > > And if I send these variables directly to the TextBox, they display > accordingly. > > However, subtract one from the other and assign to the textbox or > another Single variable, and it displays as Sci-Notation. I find this > very odd. > > Yes, I can use the original single variable to test for format, > although I'll have to store it in a Global as these values are static > to another function that has long lost scope. > > Thanks. > Webbiz > > |