Developers - Best practises for decimal handling

Xenland

legendary

Activity: 980

Merit: 1003

I'm not just any shaman, I'm a Sha256man

Quote from: deepceleron on February 19, 2013, 08:28:44 PM

Your site may need more integer math accuracy than 1 satoshi = 1

example: 1 satoshi - 1.3% fee = .987 satoshis

>>> print int(.987)
0
>>> print int(round(.987))
1

However, even using arbitrary precision, these libraries don't necessary give you the correct answer anywhere near the magnitude of the float precision defined.

>>> getcontext().prec = 6
>>> Decimal (.00000001) * Decimal (.987)
Decimal('9.87000E-9')

>>>getcontext().prec = 2000
>>> Decimal (.00000001) * Decimal (.987)[/b]
Decimal('9.87000000000000009104248083235174268563047313600349515597682205532708579013502123676582211686536 538763903081417083740234375E-9')

It's close enough though - you'd get an extra 9 satoshi if your transaction was 10 billion BTC.

Assuming the total BTC in circulation limit was increased this could ever actually happen yes.

TheButterZone

legendary

Activity: 3038

Merit: 1032

RIP Mommy

Just going to leave this here, since I'm not a dev: https://github.com/zootreeves/blockchain.info/issues/5

deepceleron

legendary

Activity: 1512

Merit: 1036

Your site may need more integer math accuracy than 1 satoshi = 1

example: 1 satoshi - 1.3% fee = .987 satoshis

>>> print int(.987)
0
>>> print int(round(.987))
1

However, even using arbitrary precision, these libraries don't necessary give you the correct answer anywhere near the magnitude of the float precision defined.

>>> getcontext().prec = 6
>>> Decimal (.00000001) * Decimal (.987)
Decimal('9.87000E-9')

>>>getcontext().prec = 2000
>>> Decimal (.00000001) * Decimal (.987)[/b]
Decimal('9.87000000000000009104248083235174268563047313600349515597682205532708579013502123676582211686536 538763903081417083740234375E-9')

It's close enough though - you'd get an extra 9 satoshi if your transaction was 10 billion BTC.

davout

legendary

Activity: 1372

Merit: 1008

1davout

Quote from: Zeilap on February 19, 2013, 02:03:46 PM

They're not built-in types, they're types provided by a library.

I call built-in what's part of the standard library of a language. So to me it's built-in the languages I listed but not C++ or PHP for example, because these require an external library.

But yes, I don't know of any language that will natively interpret a numeric string as a decimal type without explicitly telling it to.

Zeilap

full member

Activity: 154

Merit: 100

Quote from: davout on February 19, 2013, 12:39:52 PM

Quote from: Zeilap on February 19, 2013, 12:07:11 PM

no languages (that I can think of) provide built-in type support for arbitrary precision arithmetic

Quote from: Zeilap on February 19, 2013, 12:07:11 PM

and for good reason - it is expensive in terms of computation.

If I follow you, languages shouldn't provide standard interfaces to do network I/O, because you know, that's computationally expensive and slow. I guess I won't follow you...

They're not built-in types, they're types provided by a library. What the original poster seems (to me) to be confused about is that they can write code similar to

Code:

var x = 1.234

in some language and it will magically make x into a 'Decimal' object, and that only javascript will 'convert' this into a float/double. This is not true - they all will, as I say, because no language has a built-in type representing decimals, 1.234 is already a float before it gets assigned to anything. If you want decimals, then you have to specifically create one like

Code:

var x = new Decimal("1.234")

or however your library requires.

davout

legendary

Activity: 1372

Merit: 1008

1davout

Quote from: Zeilap on February 19, 2013, 12:07:11 PM

no languages (that I can think of) provide built-in type support for arbitrary precision arithmetic

Quote from: Zeilap on February 19, 2013, 12:07:11 PM

and for good reason - it is expensive in terms of computation.

If I follow you, languages shouldn't provide standard interfaces to do network I/O, because you know, that's computationally expensive and slow. I guess I won't follow you...

Zeilap

full member

Activity: 154

Merit: 100

Quote from: Xenland on February 13, 2013, 08:04:45 PM

Quote from: Seal on February 13, 2013, 07:23:13 PM

Quote from: davout on February 13, 2013, 01:44:06 PM

Any decimal number can be exactly represented this way.

Unless you are using javascript which treats decimals as floats Huh

I was waiting for someone else to point that out as well that there was no language being referenced.

No, this is not specific to javascript - no languages (that I can think of) provide built-in type support for arbitrary precision arithmetic, and for good reason - it is expensive in terms of computation.

What people are trying to say is that you should be using a library which provides a Decimal class which does provide this functionality (or a subset), and not use floating point values at all for computation.

davout

legendary

Activity: 1372

Merit: 1008

1davout

Quote from: Seal on February 13, 2013, 07:23:13 PM

Unless you are using javascript which treats decimals as floats Huh

That just means javascript doesn't support decimals, only floats.
Using floats for display is fine in most cases, using floats for accounting is terribly wrong.

Seal

donator

Activity: 848

Merit: 1078

Quote from: Xenland on February 13, 2013, 08:04:45 PM

Quote from: Seal on February 13, 2013, 07:23:13 PM

Quote from: davout on February 13, 2013, 01:44:06 PM

Any decimal number can be exactly represented this way.

Unless you are using javascript which treats decimals as floats Huh

I was waiting for someone else to point that out as well that there was no language being referenced.

Yes, the only way I thought of getting around this is to store multiple values in different formats for different languages. ie. store both integer and decimal or float.

It would be a pain otherwise to have a website which depends on javascript to manage the correct decimal display output. Every single value return or shown would require the use of javascript.

Xenland

legendary

Activity: 980

Merit: 1003

I'm not just any shaman, I'm a Sha256man

Quote from: Seal on February 13, 2013, 07:23:13 PM

Quote from: davout on February 13, 2013, 01:44:06 PM

Any decimal number can be exactly represented this way.

Unless you are using javascript which treats decimals as floats Huh

I was waiting for someone else to point that out as well that there was no language being referenced.

Seal

donator

Activity: 848

Merit: 1078

Quote from: davout on February 13, 2013, 01:44:06 PM

Any decimal number can be exactly represented this way.

Unless you are using javascript which treats decimals as floats Huh

TheKoziTwo

legendary

Activity: 1552

Merit: 1047

Quote from: Xenland on February 13, 2013, 12:17:13 PM

Quote from: TheKoziTwo on February 13, 2013, 10:44:40 AM

Quote from: Xenland on February 12, 2013, 11:46:54 PM

Quote from: TheKoziTwo on February 12, 2013, 10:36:48 AM

Quote from: davout on February 12, 2013, 05:33:01 AM

Use a decimal type of which you explicitly set the precision to whatever required.

Don't confuse floats with decimals, they are completely different, a float is by definition an approximation, a decimal stores an exact value.

+1

I use only decimals, and bcmath in php will ensure correct calculations.

I might note that in my experience shows any decimal math especially bcmath will result in fun explanations to your customers....

Why? It is 100% accurate as long as you have enough decimals.

Probably the way binary is done(hint). Binary doesn’t know a "decimal" and computers(CPU) can only "auto" guess the required precision or their calculations could take as long as infinity to compute(hence why we have a math symbol for repeating decimals...) thus most "Test" cases will look like its working correctly but it will break at some point.

BCMath uses arbitrary precision and will never fail as long as you round and use a big enough scale. If you think otherwise I would love to see proof / example of how it can fail.

davout

legendary

Activity: 1372

Merit: 1008

1davout

Some folks still don't get it apparently.

Floats are represented with a sum of powers of 2, any integer can safely be represented in binary since 2^0 = 1.

So for example 0.75 can be exactly represented in floating point representation as : 2^-1 * 2^-2

But some other numbers can not be represented exactly in floating point format, no matter how much bits you use.

Use a decimal type, it's internally represented by an integer times a negative power of ten.

0.75 in decimal is represented as 75 * 10^-2.

Any decimal number can be exactly represented this way.

Xenland

legendary

Activity: 980

Merit: 1003

I'm not just any shaman, I'm a Sha256man

Quote from: TheKoziTwo on February 13, 2013, 10:44:40 AM

Quote from: Xenland on February 12, 2013, 11:46:54 PM

Quote from: TheKoziTwo on February 12, 2013, 10:36:48 AM

Quote from: davout on February 12, 2013, 05:33:01 AM

Use a decimal type of which you explicitly set the precision to whatever required.

Don't confuse floats with decimals, they are completely different, a float is by definition an approximation, a decimal stores an exact value.

+1

I use only decimals, and bcmath in php will ensure correct calculations.

I might note that in my experience shows any decimal math especially bcmath will result in fun explanations to your customers....

Why? It is 100% accurate as long as you have enough decimals.

Probably the way binary is done(hint). Binary doesn’t know a "decimal" and computers(CPU) can only "auto" guess the required precision or their calculations could take as long as infinity to compute(hence why we have a math symbol for repeating decimals...) thus most "Test" cases will look like its working correctly but it will break at some point.

TheKoziTwo

legendary

Activity: 1552

Merit: 1047

Quote from: Xenland on February 12, 2013, 11:46:54 PM

Quote from: TheKoziTwo on February 12, 2013, 10:36:48 AM

Quote from: davout on February 12, 2013, 05:33:01 AM

Use a decimal type of which you explicitly set the precision to whatever required.

Don't confuse floats with decimals, they are completely different, a float is by definition an approximation, a decimal stores an exact value.

+1

I use only decimals, and bcmath in php will ensure correct calculations.

I might note that in my experience shows any decimal math especially bcmath will result in fun explanations to your customers....

Why? It is 100% accurate as long as you have enough decimals.

Xenland

legendary

Activity: 980

Merit: 1003

I'm not just any shaman, I'm a Sha256man

Quote from: TheKoziTwo on February 12, 2013, 10:36:48 AM

Quote from: davout on February 12, 2013, 05:33:01 AM

Use a decimal type of which you explicitly set the precision to whatever required.

Don't confuse floats with decimals, they are completely different, a float is by definition an approximation, a decimal stores an exact value.

+1

I use only decimals, and bcmath in php will ensure correct calculations.

I might note that in my experience shows any decimal math especially bcmath will result in fun explanations to your customers....

TheKoziTwo

legendary

Activity: 1552

Merit: 1047

Quote from: davout on February 12, 2013, 05:33:01 AM

Use a decimal type of which you explicitly set the precision to whatever required.

Don't confuse floats with decimals, they are completely different, a float is by definition an approximation, a decimal stores an exact value.

+1

I use only decimals, and bcmath in php will ensure correct calculations.

davout

legendary

Activity: 1372

Merit: 1008

1davout

Quote from: Seal on February 12, 2013, 08:28:41 AM

Javascript treats floats and decimals as the same type. When performing number manipulations for visual display then some issues might arise. The tricky aspect is... how do i use an Integer number format and have javascript run calculations whilst returning numbers in correct format? Can any javascript developers out there share their thoughts?

Interesting.

If it's purely for display purposes I'd say a float is an acceptable approximation.
But of course it really depends on the specifics of your problem, what works for me might not work for you.

You'd probably better in-depth advice if you gave a bit more context

Seal

donator

Activity: 848

Merit: 1078

Quote from: davout on February 12, 2013, 05:33:01 AM

Use a decimal type of which you explicitly set the precision to whatever required.

Don't confuse floats with decimals, they are completely different, a float is by definition an approximation, a decimal stores an exact value.

Don't ever listen to anyone suggesting floating point for financial calculations, they don't know what they're doing, no matter how much precautions and care you take floats are and remain approximations that are not suited for monetary amounts.

If you know the difference and know what you're doing there's really no need to use integers as storage type and handling type. Using an integer to store a decimal value in a high level language would show that you don't know exactly what you're doing.

The reason MtGox will return integers in their API responses is because there is no way to discriminate between decimal and float types in JSON, they are represented the same way. Therefore, if your parsing lib parses these as floats instead of proper decimals you're boned. But if it parses them as integers you have an opportunity to post-process these as proper decimals before actually using them.

Understood davout. Most languages these days can handle the different number types, decimal or float won't be an issue. Certainly in my specific cases using MySQL and PHP, these numbers are handled very well and very precisely.

The issue arises with Javascript which is widely used these days.

http://stackoverflow.com/questions/588004/is-javascripts-floating-point-math-broken

Javascript treats floats and decimals as the same type. When performing number manipulations for visual display then some issues might arise. The tricky aspect is... how do i use an Integer number format and have javascript run calculations whilst returning numbers in correct format? Can any javascript developers out there share their thoughts?

davout

legendary

Activity: 1372

Merit: 1008

1davout

Use a decimal type of which you explicitly set the precision to whatever required.

Don't confuse floats with decimals, they are completely different, a float is by definition an approximation, a decimal stores an exact value.

Don't ever listen to anyone suggesting floating point for financial calculations, they don't know what they're doing, no matter how much precautions and care you take floats are and remain approximations that are not suited for monetary amounts.

If you know the difference and know what you're doing there's really no need to use integers as storage type and handling type. Using an integer to store a decimal value in a high level language would show that you don't know exactly what you're doing.

The reason MtGox will return integers in their API responses is because there is no way to discriminate between decimal and float types in JSON, they are represented the same way. Therefore, if your parsing lib parses these as floats instead of proper decimals you're boned. But if it parses them as integers you have an opportunity to post-process these as proper decimals before actually using them.

CIYAM

legendary

Activity: 1890

Merit: 1086

Ian Knowles - CIYAM Lead Developer

Quote from: Seal on February 12, 2013, 05:16:13 AM

Using Integers:

Ensure that all calculations do not accept any decimal value and rounding is performed at integer level
Ensure you have strict controls in your UI layer to process raw figures into the correct decimal representations
* the tricky aspect here is knowing whether a figure read is a currency figure or a btc figure

Using float/decimals:

Ensure that all calculations are subject to the correct decimal rounding (eg if BTC, then always round every calculation to 8dp.
Ensure that every language used represents decimals correctly

Although I would still very much recommend using integers over binary FP if you do the rounding at all the right points (and this is harder with FP if you are going to be doing multiple calculations) then you should of course end up with the same result.

Seal

donator

Activity: 848

Merit: 1078

Quote from: CIYAM on February 09, 2013, 08:59:48 PM

Quote from: Seal on February 09, 2013, 12:21:45 PM

I've worked in my real jobs extensively in retail and investment banking on a range of trading and reporting systems. Not a single one I have come across represents monetary value in integer format. (hence this discussion thread) Rounding within all banking systems (in my experience at least) occurs programatically in functions.

That's interesting - certainly all the banking software that I've seen (am talking *bank end* not UI) is still written in COBOL (developed in the 70's and patched in the 90's for the Y2K bug) where you use PIC variables which let you describe the numeric as a number of digits and number of decimal places.

Am pretty certain that *under the covers* these are actually represented as integers (just not seen that way by the programmer even) as they don't suffer from the "binary floating point problems".

My experience has mostly been with back end systems so raw data in SQL databases and mainframes. Data inputs for their front end systems will only accept 2 decimal places as valid data. Anything else input from either a data feed or directly would be rejected as a bad data input.

I don't believe that it would matter whether you represent values in integer or in floating point as long as you have programmed your controls correctly. For calculations and rounding, there should can be a few issues. Eg:

Using Integers:

Ensure that all calculations do not accept any decimal value and rounding is performed at integer level
Ensure you have strict controls in your UI layer to process raw figures into the correct decimal representations
* the tricky aspect here is knowing whether a figure read is a currency figure or a btc figure

Using float/decimals:

Ensure that all calculations are subject to the correct decimal rounding (eg if BTC, then always round every calculation to 8dp.
Ensure that every language used represents decimals correctly

BitcoinRate.com

member

Activity: 116

Merit: 11

Storing as integers is the most common way I think. Depending on your framework/database it can differ however. For example in Django you could use DecimalField.

CIYAM

legendary

Activity: 1890

Merit: 1086

Ian Knowles - CIYAM Lead Developer

Quote from: Seal on February 09, 2013, 12:21:45 PM

I've worked in my real jobs extensively in retail and investment banking on a range of trading and reporting systems. Not a single one I have come across represents monetary value in integer format. (hence this discussion thread) Rounding within all banking systems (in my experience at least) occurs programatically in functions.

That's interesting - certainly all the banking software that I've seen (am talking *bank end* not UI) is still written in COBOL (developed in the 70's and patched in the 90's for the Y2K bug) where you use PIC variables which let you describe the numeric as a number of digits and number of decimal places.

Am pretty certain that *under the covers* these are actually represented as integers (just not seen that way by the programmer even) as they don't suffer from the "binary floating point problems".

Seal

donator

Activity: 848

Merit: 1078

Quote from: deepceleron on February 09, 2013, 07:08:13 AM

One should always calculate and store in integer base units when dealing with Bitcoin amounts. Pool operators and other sites that have previously stored values in floats have been shamed out of existence.

There is consensus that from all the replies so far that integers are the way forward, and I would totally agree, from a IT/developer's perspective this makes sense.

I've worked in my real jobs extensively in retail and investment banking on a range of trading and reporting systems. Not a single one I have come across represents monetary value in integer format. (hence this discussion thread) Rounding within all banking systems (in my experience at least) occurs programatically in functions.

Quote from: Scrat Acorns on February 09, 2013, 07:14:22 AM

Quote from: Seal on February 09, 2013, 06:36:48 AM

however work will be required in the display logic of a site / program.

I don't see how this is a problem. You're already using extra logic to escape user input, format timestamps, generate urlencoded links, etc.

Agreed, it isn't an issue however the logic would either have to be built into the calculation layer (ie your programming) or the visual presentation layer if you were to store values in decimal or integer format respectively. I'd like to know what everybody's preference is and why.

Second to this, looking at the mtgox api, values are stored in not one, not two, but THREE formats:

Display value
Float/Double/Decimal value
Integer value

Xenland

legendary

Activity: 980

Merit: 1003

I'm not just any shaman, I'm a Sha256man

If you use my library (Bitcoin Dev Kit) forces you to use integers for math/proccessing while displaying is in your choice dynamically between decimal and integers. Aswell as other helpfull features like make bitcoind communication smoother and wont block your website and make it render blank on a minor bitcoin error (like low balance,etc).

CIYAM

legendary

Activity: 1890

Merit: 1086

Ian Knowles - CIYAM Lead Developer

There are multiple ways to *round* and any decent *numeric* implementation will let you choose the one that suits you best.

If you use a 64 bit integer type and create a *numeric* class around that then you will have no problems with the range of values required for Bitcoin (and pretty much all currencies). As has already been pointed out *binary* floating point types are *not* suitable for financial software.

Welcome to take a look at the numeric class that CIYAM uses: https://github.com/ciyam/ciyam/blob/master/src/numeric.cpp

Zeilap

full member

Activity: 154

Merit: 100

One point that hasn't been mentioned that I've seen be a problem is rounding. I've seen posts on the forum where some site has said they have X amount, but it turns out that really they have ever so slightly less than that amount and it's been rounded up, so when they try to send, they get an error saying 'not enough funds!' which is very confusing. So the moral of this story is to round *down* if you aren't going to display the whole balance down to 8 decimal places.

Scrat Acorns

sr. member

Activity: 293

Merit: 250

Quote from: Seal on February 09, 2013, 06:36:48 AM

however work will be required in the display logic of a site / program.

I don't see how this is a problem. You're already using extra logic to escape user input, format timestamps, generate urlencoded links, etc.

Storing satoshi values in integer format is the sensible thing to do. Just like storing unix timestamps instead of the date string.

You should never store BTC values as floats. You cannot guarantee that your favorite language / web framework / database will handle them the way you expect them to be handled.

deepceleron

legendary

Activity: 1512

Merit: 1036

One should always calculate and store in integer base units when dealing with Bitcoin amounts. Pool operators and other sites that have previously stored values in floats have been shamed out of existence.

The only time it would be even close to acceptable would be if you are dividing or multiplying by a non-integer and want to maintain an even higher degree of accuracy internally in accounts (such as my $0.00000 in mtgox), and then you must ensure that your platform and any other that may run your code will store in IEEE 754 double floats or greater.

Another way bitcoin was more thought out than you may understand: There will be a maximum of 0x775F05A074000 bitcoin base units ever in existence, which is a 51 bit number. From wikipedia "'double precision' (64-bit) binary floating-point number has a coefficient of 53 bits (one of which is implied), an exponent of 11 bits, and one sign bit.".

Seal

donator

Activity: 848

Merit: 1078

I'd like to open up a discussion about the different way you as developers have approached the problems of decimal handling when working on bitcoin projects.

Because bitcoin is divisible down to 8 decimal places, calculations on float / decimal numbers can be a pain in a number of languages (as numbers may never appear exact even if displayed correctly).

When working on the MtGox api, they have marked the decimal value representations as legacy. It appears they are moving away from anything decimal related and using integers (which is multiplied by 1e-8 when displayed for bitcoin). This makes calculations simpler in whatever language you are programming in, however work will be required in the display logic of a site / program.

What are your programming practises for handling the many decimals that bitcoin provides?

I'd like to hear from you if you run a bitcoin site or even a wallet program, btcjam / satoshidice / electrum etc... ?

Topic: Developers - Best practises for decimal handling (Read 3222 times)