Author

Topic: Introducing Bookwerx. Bookkeeping for traders. (Read 127 times)

jr. member
Activity: 39
Merit: 5
November 26, 2019, 07:40:27 PM
#6
Howdy Folks,

One particularly nettle-some issue of various exchanges that we have encountered centers around the issue of dealing with numerical precision.

This issue has at least four main conspirators:

1. Ordinary round-off issues.  As you're probably aware, when calculating the actual quantities of the coins involved in a trade you're likely going to encounter the need to do some rounding.  For example, if the price of a trading pair is 3 Coin A / Coin B, and you have 1 Coin  A, you can buy a whopping 0.33333333333333333333.... coin B. 

2. How do you display it? How many decimal points shall we display?  This gives us a 2nd level of fun-with-rounding because whatever rounding we've done prior may need to get rounded again to fit the display.  Or perhaps we've rounded too aggressively prior and now the display shows a lot of trailing zeros after a now merely wrong number.

3. How do you control the propagation of round-off error throughout your system?  You're constantly calculating prices, adding transaction amounts together, and trying to present the information on reports.  There are many places for rounding issues to seep into this.

4. How do you represent these numbers internally?  Are you going to use Integers? Floats? Strings? Tally sticks?  Some other fundamentally better method?

All of this is a giant nuisance and distraction from the main goal of doing whatever else you're doing with software.  Whenever you see bookkeeping numbers displayed in anybody's software, they've had to grapple with these issues, with whatever degree of success they've had.

My purpose in this post is not to gloat over their discomfort.  Instead, I'd like to explain how bookwerx deals with these issues.

The party starts with internal representation.  I'll spare you the gory details about why Ints, Floats, Strings, and Tally sticks are not sufficient for this job.  Instead, I'll assert that bookwerx gets this done by using two integers to store every number.  These integers are called the significand and the exponent and they work very similarly to scientific notation.  For example, the tuple (70000523, -7) really does a completely adequate job of representing exactly 7.0000523.

In this system we still have the issue of MAXINT to be wary of.  But that's a headache for developers and is of no concern to the end-user, who never sees these numbers in this form anyway. In actual practice, integer MAXINT is high enough to accommodate the actual numbers in real use.  For example: a mere 41 bit int would be enough to describe 10000.00000001 of some coin.  If you're dealing with that many BTC for example, you'll probably be able to get bigger Ints.  I leave it as an exercise for the reader to figure out what could be done with 53, 64, and 128 bit Ints.

Internal representation is merely the first step.  Next, one must have a standard for rounding when doing multiplication and division.  Using my prior examples again, If the price of BBBAAA is exactly 3, and I have 7.0000523 AAA, I can buy 2.333350766666666....  BBB.  The bookwerx system can't represent that number directly and so you'll need to round to some power of 10.  How about round it to 8 decimal places as is frequently done?  Hence the result is 2.33335077, which bookwerx can easily represent as (233335077, -8)

Regarding the issue of propagation of error, there is very little of this in bookwerx.  The numbers are stored internally in an exact format.  They can easily be added and subtracted with zero loss and that's all bookwerx cares about.  Problem solved.  Other people might want to perform other numerical operations on these numbers, and they'll need to figure out how to convert between this internal representation and whatever representation they use and how to control their own error propagation.  But that's outside of bookwerx'  job description.

Finally, we return to the issue of display.  Suppose our result is 2/3.  We round this to 8 decimal places and get 0.66666667.  This of course involves a one-time burst of error, but we can tolerate that and we can easily represent this internally as (66666667, -8).  Later, we want to display this rounded to 3 decimal places (because that's the sort of thing that people do.)  Shall we just truncate and display 0.666? Shall we round again giving us (667,-3) internally and then display 0.667 ?  So many things to fret about!  Even after you bang your fist on the table and order the resolutions of these tedious details, you now have a new problem.  Your end-user thinks he has 0.66666667 Quatloos,  but only sees 0.667 in his account summary.  Worse, he sees 0.6666 in some other transaction listing, so your system is internally inconsistent.

Eye-oh!  How about dump the above and try plan B?

How about give the user control over the quantity of decimal places to display and provide some visual feedback regarding hidden-loss-of-precision-due-to-the-rounding-of-the-display?  The underlying data is never touched and the reports round as necessary but let the users know that there are details omitted.  Finally, because all of the reports are drawing from the One Source of Truth in a consistent manner, all of the reports are internally consistent.

I have attached some screenshots to illustrate:

Start with some transactions with the display limited to 5 decimal places:



By clicking on the suitable control, the user can easily round to 4 decimal places.  But now we have some red to indicate the existence of hidden detail:



By continuing this process the user can further limit the display  as desired:






But wait... there's more!  Why stop at zero?  Do billionaires look at the 10^1 or 10^2 power of their finances?  I wouldn't!  Let's keep limiting the display even more...















jr. member
Activity: 39
Merit: 5
Howdy Jerome,

Hey i would like to understand. AFAIK there are some services like 3commas and BitUniverse that off option of tracking portfolio, trade/transaction histories, and daily profits just through use of API keys imported from exchanges. The UIs are very friendly and easy to use. What extra feature will your service offer and perhaps help provide a stronger advantage of the already existing portolio managers?


Those services look pretty good.   How is bookwerx different?  Glad you asked!

1. bookwerx is not a similar service.  We have an online example to look at http://23.253.160.60/, but are not soliciting users for it as a service.

2. bookwerx is open source and it's meant for people who need to have greater control over what it does and how its installed.  It's a foundation for more sophisticated reporting/analysis that somebody might want.

3. bookwerx does not connect the dots of your entire crypto-empire via a single account with any online service.  I don't have any specific issues with the particular services you mentioned, but, charitably speaking, CloudWorld has not exactly demonstrated much trustworthiness or integrity to date.

4. bookwerx maintains your data locally where you can keep a close eye on it.

5. bookwerx let's you enter any transactions you want, not just whatever is offered by the other services you mentioned.  So for example, crypto-trading is part of one's larger financial environment and some folks want financial reporting that incorporates everything they do, not just the crypto-part.

Thanks for your interest.  Hope this clarifies.
legendary
Activity: 2324
Merit: 1258
Heisenberg
Hey i would like to understand. AFAIK there are some services like 3commas and BitUniverse that off option of tracking portfolio, trade/transaction histories, and daily profits just through use of API keys imported from exchanges. The UIs are very friendly and easy to use. What extra feature will your service offer and perhaps help provide a stronger advantage of the already existing portolio managers?
jr. member
Activity: 39
Merit: 5
Tracking the P&L of an open position, especially anything remotely complicated, would be real easy to do with this API.

One particularly nettle-some issue that I frequently see is that customer balance information on exchanges is usually pretty bad. It's frequently divided into several different sections and we have to transfer funds from one section to another.  I've seen markets where these exchanges are just not listed.  Whatever balances are displayed are mere assertions w/o any list of transactions to support their derivation.  Even if the exchange offers a transaction history, it's frequently just that... a list of transactions of all the different currencies w/o any running total.  Or perhaps there _is_ a running total, but it's out of sequence with the actual transactions, said ordering changing indeterministially! All of this makes reconciliation with external records extremely difficult, if not impossible.

Speaking of reconciliation... even if an exchange _really does_ have good records, the trader needs his own internal records to compare against.  So many people have to keep re-inventing this wheel.  Hopefully bookwerx can save 'em all some trouble :-)
sr. member
Activity: 784
Merit: 257
I never did bookkeeping or recorded trading history, because I don't trade much and only a pair of cryptos that I trade, maybe it will be very useful for people who like to do a lot of trading, so it's easy to calculate profit or loss
jr. member
Activity: 39
Merit: 5
Hello Folks,

One element of trading that's particularly nettle-some is doing bookkeeping.  Believe it or not, many people simply ignore the issue and don't even bother.  Many people try to cobble together spreadsheets and enter transactions by hand.  Neither of these choices are particularly scaleable.  I for one would hate to unleash the TraderBot and then try to follow behind entering transactions by hand in its wake. But creating stronger software that does a better job, is, as with most things computing, easier said than done.

But fear not. All is not lost.  That light in the tunnel is not really an on-rushing train.  I'd like to announce the bookwerx multi-currency, client-server, bookkeeping system.  This is a simple and easy to get started with, open source project, complete with functioning public API.  Take it for a spin around the block, and if it seems useful to you, it's easy enough to setup your own private installation.

bookwerx-core-rust (https://github.com/bostontrader/bookwerx-core-rust) Is the meat and potatoes of bookwerx.  This is a server, written in Rust, that implements an API for doing bookkeeping.  With this API you can define the currencies and accounts that you wish to use, create suitable transactions, observe aggregate phenomenon via reporting tools, and hunt down nettle-some bits o' lint with the linter.

bookwerx-ui-elm (https://github.com/bostontrader/bookwerx-ui-elm) Is an example user interface the enables the user to manually do bookkeeping.  An important benefit of this help the user understand the bookwerx-core-rust API.  With this app the user can observe the http traffic between the UI and the server.  In this way perhaps insight can be had.


Public Demonstration Server

We have a public demonstration server available at http://23.253.160.60.  This will serve bookwerx-ui-elm.  So it's easy enough to get your feet wet with manual bookkeeping, learn the server's API by observing the http traffic to and fro, and eventually graduate to an automated connection to the server, using the API.

Jump to: