Prev: Question about stored procedures - please advise.
Next: Query question - Separating Columns?
From: Erland Sommarskog on 8 Jun 2010 18:04
Sylvain Lafontaine (sylvainlafontaine2009(a)yahoo.ca) writes:
> I'm working on a similar problem. It now appears to me that even if float
> are stored internally with 16 digits,

Floats are not stored with 16 digits, they are stored with 53 bits
in the mantissa. This may rougly correspond to 16 decimal digits,
but it is not the same.


--
Erland Sommarskog, SQL Server MVP, esquel(a)sommarskog.se

Links for SQL Server Books Online:
SQL 2008: http://msdn.microsoft.com/en-us/sqlserver/cc514207.aspx
SQL 2005: http://msdn.microsoft.com/en-us/sqlserver/bb895970.aspx
SQL 2000: http://www.microsoft.com/sql/prodinfo/previousversions/books.mspx

From: Eric Isaacs on 9 Jun 2010 16:14
> Floats are not stored with 16 digits, they are stored with 53 bits
> in the mantissa. This may rougly correspond to 16 decimal digits,
> but it is not the same.

According to Microsoft Technet, the precision for FLOAT(53) is 15
digits and the precision for FLOAT(24) and REAL is 7 digits.

Source:
http://technet.microsoft.com/en-us/library/ms173773.aspx

-Eric Isaacs
From: Eric Isaacs on 9 Jun 2010 16:43
Good call John.

-Eric Isaacs
First  |  Prev  | 
Pages: 1 2
Prev: Question about stored procedures - please advise.
Next: Query question - Separating Columns?