From: Eric on 17 Jun 2010 10:19 Hi, In a data table I have records numbered from 801 to 809. I also use these numbers as row number then 8.01 to 8.09. To do so I just devide the record number by 100 and put it in a variable of type decimal (also tried double and single). At one point I compare this value with a number to do an evaluation: itemnr = CDec(rw("itemnr")) / 100 "If itemnr = 8.03 Or itemnr = 8.04 Or itemnr = 8.06 Or itemnr = 8.07 Then" Well it should look like that, but it actually showing: " If itemnr = 8.0299999999999994 Or itemnr = 8.0399999999999991 Or itemnr = 8.0600000000000005 Or itemnr = 8.0700000000000003 Then" How can I get rid of the extra decimals? I don't need them and it's confusing. rg, Eric From: J.B. Moreno on 17 Jun 2010 11:41 Eric wrote: > Hi, > > In a data table I have records numbered from 801 to 809. > I also use these numbers as row number then 8.01 to 8.09. > > To do so I just devide the record number by 100 and put it in a variable of > type decimal (also tried double and single). > > At one point I compare this value with a number to do an evaluation: > > itemnr = CDec(rw("itemnr")) / 100 > > "If itemnr = 8.03 Or itemnr = 8.04 Or itemnr = 8.06 Or itemnr = 8.07 Then" > > Well it should look like that, but it actually showing: > " If itemnr = 8.0299999999999994 Or itemnr = 8.0399999999999991 Or itemnr = > 8.0600000000000005 Or itemnr = 8.0700000000000003 Then" Fractional numbers are always approximate --- if you're going to proceed with this, you're going to have to deal with the rounding issue. I don't understand what you're doing well enough to say what you should do, but I'd reconsider it if I was you.... -- J.B. Moreno From: Armin Zingler on 17 Jun 2010 10:52 Am 17.06.2010 16:19, schrieb Eric:> Hi, > > In a data table I have records numbered from 801 to 809. > I also use these numbers as row number then 8.01 to 8.09. > > To do so I just devide the record number by 100 and put it in a variable of > type decimal (also tried double and single). > > At one point I compare this value with a number to do an evaluation: > > itemnr = CDec(rw("itemnr")) / 100 > > "If itemnr = 8.03 Or itemnr = 8.04 Or itemnr = 8.06 Or itemnr = 8.07 Then" > > Well it should look like that, but it actually showing: > " If itemnr = 8.0299999999999994 Or itemnr = 8.0399999999999991 Or itemnr = > 8.0600000000000005 Or itemnr = 8.0700000000000003 Then" > > How can I get rid of the extra decimals? > I don't need them and it's confusing. If 'itemnr' is Decimal or Double, I can't repro the problem: Dim value As Decimal = 803 Dim itemnr As Double 'As Decimal: same behavior itemnr = value / 100 If itemnr = 8.03 Then MsgBox("equal") 'always shown End If -- Armin From: OmegaSquared on 17 Jun 2010 13:19 Hello, Eric, The numbers 8.xx are being interpreted as floating point values. Appending a "D" to them should correct the problem that you are seeing. That is, try: "If itemnr = 8.03D Or itemnr = 8.04D Or ... It is very important to have a clear understanding of the different number types that can be used and what their limitations are. Floating point values particularly seem to cause some programmers much confusion. See: http://docs.sun.com/source/806-3568/ncg_goldberg.html for a good detailed explanation. Cheers, Randy  |  Pages: 1 Prev: Correction needed in Copy File codeNext: XL app wants focus?