Re: [genius-list] error in floor

On Sat, Jun 05, 1999 at 11:18:20PM -0400, Squeak wrote:
> This is the unmodified version.
>
> Basically 74! doesn't fit into a double, so mpf_get_d(74!*1.0) doesn't
> return the correct value. FWIW it returns the correct decimal digits, it's
> off by a pwoer of 10. (I find that strange to say the least).
>
> FIX: have floor return a mpz_t instead of a double.

floor returns a mpw_t (mpwrap.h) with a type if INTEGER,

the thing is:

genius> floor(78!*1.0)-78!
= 0
genius> floor(79!*1.0)-79!
= -289448810798376537378126472479001018368
genius> 79!*1.0
= 8.946182130782975286851441715398316520698082167795719072138680632278379906935e116
genius> 79!
= 894618213078297528685144171539831652069808216779571907213868063227837990693501860533361810841010176000000000000000000
genius> floor(79!*1.0)
= 894618213078297528685144171539831652069808216779571907213868063227837990693501571084551012464472797873527520998981632
genius>

ok so 79!*1.0 gets a lot of the digits cut and is most likely stored in a
base != 10 (which is why you don't see 0's on the end)

increase your precision for floats to make the limit higher

this is all mpwrap.h, it deals with all type conversions, and whenever you
deal with floats, you may lose absolute precision

George

-- 
------------------------------------------------------------------------------
George Lebl <jirka@5z.com> http://www.5z.com/jirka/
------------------------------------------------------------------------------
  The following implements RSA in perl and is illegal to export from the US:
          #!/bin/perl -sp0777i<X+d*lMLa^*lN%0]dsXx++lMlN/dsM0<j]dsj
          $/=unpack('H*',$_);$_=`echo 16dio\U$k"SK$/SM$n\EsN0p[lN*1
          lK[d2%Sa2/d0$^Ixp"|dc`;s/\W//g;$_=pack('H*',/((..)*)$/)