precision of float

Discussion in 'Cadence' started by jren, Sep 6, 2011.

  1. jren

    jren Guest

    Hi,

    I have two questions related to precision of float:

    1.
    In my SKILL code:
    grid1 = 0.5
    fprintf(stdout "grid1 = %.20f\n" grid1)
    grid2 = 0.4
    fprintf(stdout "grid2 = %.20f\n" grid2)
    Output in CIW:
    grid1 = 0.50000000000000000000
    grid2 = 0.40000000000000002220

    How can I get 0.4 = 0.4000000000000000 as 0.5?

    2.
    I tried to use function techSetPrecision(), but I am confused by the
    following test:
    techSetPrecision(2)
    grid3 = 0.5553567
    grid3 = grid3 * 1.0
    fprintf(stdout "grid3 = %.20f\n" grid3)
    In the CIW:
    grid3 = 0.55535670000000003643
    What I assume is grid3 should become 0.5600000 since I set precision
    of number of digits in float as 2. Can anyone help me better
    understand function techSetPrecision?

    Thanks!

    Jren
     
    jren, Sep 6, 2011
    #1
    1. Advertisements

  2. You can't. Floating point is a pain to work with. Just like you can't
    represent 1/3 exactly in decimal with a fixed number of digits, you
    can't represent 0.4 exactly in common floating point format as they are
    using a binary representation (they are able to represent only a subset
    of fractions with a power of 2 as denominator, so 0.5 which is 1/2 can
    be represented exactly, but 0.4 must be represented as an approximation)

    Yours,
     
    Jean-Marc Bourguet, Sep 7, 2011
    #2
    1. Advertisements

  3. jren

    jren Guest

    Thanks a lot for your explanation.

    Regards!

    Jren
     
    jren, Sep 7, 2011
    #3
  4. jren wrote, on 09/07/11 12:06:
    Note also that this not just a SKILL issue. You'd have exactly the same problem
    with most programming languages (certainly any that use IEEE floating point
    representation, including C).

    Andrew.
     
    Andrew Beckett, Sep 26, 2011
    #4
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.