Scientific to Decimal
in [Visual Basic]
|
From: Nobody on 1 Dec 2009 17:24 "Webbiz" <nospam(a)noway.com> wrote in message news:ae2bh514mrcog5bovvdiuunvteeq6dgui4(a)4ax.com... > Ah, tested but didn't work out because the blasted values in the array > are in Scientific Notation. VB had to have done this because the > actual values when loaded into the array were not in SN. Single/Double are not stored in Scientific Notation, they are always stored in binary form. When you implicitly assign Single/Double to String, VB does the conversion for you, which may not be what you want. You have to use Format() function. Also, Round() function converts a number from one binary form to another, so it could round something from .8999999999 to .900000001, so you still have to use Format() function to display the value to insure that you get the correct number of extra digits. If you want to see what Single looks like in binary form, see here: http://en.wikipedia.org/wiki/Single_precision_floating-point_format And Double: http://en.wikipedia.org/wiki/Double_precision_floating-point_format
From: Webbiz on 1 Dec 2009 17:34 On Tue, 1 Dec 2009 17:24:43 -0500, "Nobody" <nobody(a)nobody.com> wrote: >"Webbiz" <nospam(a)noway.com> wrote in message >news:ae2bh514mrcog5bovvdiuunvteeq6dgui4(a)4ax.com... >> Ah, tested but didn't work out because the blasted values in the array >> are in Scientific Notation. VB had to have done this because the >> actual values when loaded into the array were not in SN. > >Single/Double are not stored in Scientific Notation, they are always stored >in binary form. When you implicitly assign Single/Double to String, VB does >the conversion for you, which may not be what you want. You have to use >Format() function. > >Also, Round() function converts a number from one binary form to another, so >it could round something from .8999999999 to .900000001, so you still have >to use Format() function to display the value to insure that you get the >correct number of extra digits. > >If you want to see what Single looks like in binary form, see here: > >http://en.wikipedia.org/wiki/Single_precision_floating-point_format > >And Double: > >http://en.wikipedia.org/wiki/Double_precision_floating-point_format > Then why are my Single values, showing in Watch as .5938 for example, showing as a Scientific Notation value after some math and assigned to another Single variable? If I send the original single value to the text box, it will show up as .5938. But do some subtraction, for example, of two of these Singles and assign to another Single, or directly to the textbox, and it displays as Scentific Notation? Webbiz
From: Webbiz on 1 Dec 2009 17:42 On Tue, 1 Dec 2009 16:49:17 -0500, "Rick Rothstein" <rick.newsNO.SPAM(a)NO.SPAMverizon.net> wrote: >>>Let Num be **any** number from the list of numbers that you are processing >>>and let Avg be the calculated average... >>> >>>Num = <<any value from list>> >>>Avg = <<calculated average>> >>>' This line formats the calculated value as per a number in the list >>>Avg = Format(Avg, "0." & String(Len(Num) - InStr(Num, "."), "0")) >> >> >> Ah, tested but didn't work out because the blasted values in the array >> are in Scientific Notation. VB had to have done this because the >> actual values when loaded into the array were not in SN. >> >> What causes this and how to keep VB from doing it this way? >> >> Here is where this is happening: >> >> Values(1) = CurrPriceData.High - CurrPriceData.Low >> >> Values() is type Single. >> >> CurrPriceData.High = 0.5001 >> CurrPriceData.Low = 0.4963 >> >> Values(1) = 3.800005E-03 > >Try using one of the CurrPriceData.High or CurrPriceData.Low values for Num >value in my expression as they seem like they might be in the right format >(String as opposed to Single I'm guessing). If these values are really >numeric values and not text String representations of those numerical >values, then can you see the formatted number from the original source? If >so, this is what you want to assign to the Num variable. Basically, you need >a **formatted** value so you can see how many decimal places there are... if >these are numerical values, then you will never know if the value you chose >would have ended in zeroes when originally formatted (since numbers do not >have format, they can end with zeroes after the decimal point, only >formatted String representations of these numbers can do that). The values are stored in DataArray as Singles Type DataArray dDate As Date Open As Single High As Single Low As Single Close As Single End Type So these are both Singles... CurrPriceData.High = 0.5001 CurrPriceData.Low = 0.4963 And if I send these variables directly to the TextBox, they display accordingly. However, subtract one from the other and assign to the textbox or another Single variable, and it displays as Sci-Notation. I find this very odd. Yes, I can use the original single variable to test for format, although I'll have to store it in a Global as these values are static to another function that has long lost scope. Thanks. Webbiz
From: Jim Mack on 1 Dec 2009 17:56 Webbiz wrote: > Then why are my Single values, showing in Watch as .5938 for > example, showing as a Scientific Notation value after some math and > assigned to another Single variable? > > If I send the original single value to the text box, it will show up > as .5938. But do some subtraction, for example, of two of these > Singles and assign to another Single, or directly to the textbox, > and it displays as Scentific Notation? A textbox can't display a single, it can only display strings. And anything you show in Watch or Debug etc is a also string. If you want to control how something appears on the screen (i.e. in the string domain), use Format. -- Jim Mack Twisted tees at http://www.cafepress.com/2050inc "We sew confusion"
From: dpb on 1 Dec 2009 18:10
Webbiz wrote: .... > Then why are my Single values, showing in Watch as .5938 for example, > showing as a Scientific Notation value after some math and assigned to > another Single variable? > > If I send the original single value to the text box, it will show up > as .5938. But do some subtraction, for example, of two of these > Singles and assign to another Single, or directly to the textbox, and > it displays as Scentific Notation? Because unless you control the output format specifically, you're at the mercy of the VB RTL i/o library logic for converting the internal representation to some default character representation. What that will be will be determined by the magnitude and the implementation of the specific control or i/o operation. If you care what the output looks like, use Format$() BTW, this a generic requirement in any programming language that has the equivalent facility of providing default i/o formatting, not just VB. In Fortran, it's called "list-directed" formatting as just one example. -- |