Topic: [ANN] Bitcoin-Analytics.com - price correction, additional currencies - page 7. (Read 49911 times)

I'm begging you to remove BTC24 from the arbitrage page.... There's no purpose anymore and it's ruining the purpose of being a member of your service. I'm a paid member and I would be able to use the arbitrage page if it was fixed! You should also remove Vircurex as it's useless for arb, and the price displayed is rarely accurate to the actual price... thanks!

To show the absolute scale just don't divide by total volume, though I'm of the opinion that if you have a stacked bar chart for the trade volume then that will be obvious.

I just was thinking about how visually this shart will appear - as I can understand it will got positiv if buys greater than sells and negative otherwise. The value will not exceed +/-1.

But how this an be modified to be ale to visualize not only side of trade but also the absolute siginificance. I mean to tell whether there are only small sells or big. Can it be some how combine in this idea? Of course we always can have existing already absolute trades volume chart but I am just curious.

What is your opinion?

When orderbook is not updated its median price is fixed and during this period actual trades can go up or down, thus they all automatically assigned to the same side.

This would not be a problem if bitcoincharts could collect and resend trades with original bid/ask assigned from exchanges. I contacted bitcoincharts before and they answered that they do not have any plans to support this feature unfortunately.

So we assign trades to bid/ask side on our own based on currently available orderbook data. This was very simple solution.


We get orderbooks directly from exchanges.
mtgox http full orderbook + stream api
btc-e http orderbook is not full - we could not manage discussion with them about possible solution yet, cause they are too busy on their own.

All trades for all exchanges we get thru bitcoincharts. this is the most simple solution.

we have plans to implement hybrid mode for mtgox trades which will get trades from stream api and in case stream api is down it will automatically switch to bitcoincharts stream

something like this

Note however that sells are being updated but buys are not. That rules out connectivity issues since buys and sells are on the same channel at least for mtgox. Also the same issue on three different exchanges at the same time is fishy.



To clarify, do you get your data from bitcoincharts or using each exchange's API directly?

Aha I see now, actually we just wanted something like this when we tried Time last x traded chart.


There can be several causes of this
1) some times exchange just does not return request  - in this case the best option we have to assume the last available data as actual
the problem can be on exchange side - btc-e was ddos'ed or our hosting did not get connectivity.

2) in case of gox - we use quite complex logic for stream api client wich accounts for several cases:
  - stream api can connect us but does not subsribe to any channel in this case we need to detect that stream is dead after some timout and reconnect
  - if we could not reconnect for 15 minutes we force orderbook reinitialization - request full ordrbook via http
  - also gox can disconnect/reconnect us arbitrarily often and each time we connected again and subscribed we need to reinitialize orderbook state requesting full orderbook via http but we cannot request too often otherwise we will be  blocked by gox. thus we maintain hysotry of requests for each currency and wait in timeout if necessary some times for on hour and rarely for 24 hours.

all those timeouts is subject to changes and tweaks cause mtgox every time shows different and new patterns of how their engine can behave.

in all those cases we use last know state of orderbook to be able to assign trades to bid or aks side.

3) bitcoincharts can:
  - be down
  - we just do not connection to bitcoincharts
  - bitcoincharts delays some trades  - in this case after some time bitcoincharts cn send all dealyed trades and if our application was still up it sorts all of them by timestamp to its time slot and assigns bid/ask side, but if our application happend to restart due its own uncought exceptions then delayed trades could not come to us at all.

So if orderbook state is not changing you can see flat line.
If trade time is increasing then trades are not coming.

so you can imagine that thigns are a bit complex here.

thank you for your attention to those details

Perhaps (buys-sells)/(buys+sells) (if buys+sells=0 then 0) in that period then. The idea is to compare the relative volume of buys and sells in a time window to see if the market is bullish or bearish during that period.

Also the charts seem to be glitching out periodically where nothing is updated except sells. Happens at the same time on multiple exchanges so I assume it's not on mtgox's end, what's up with that?

https://i.imgur.com/3J66QgP.png

https://i.imgur.com/5DnrqRy.png

https://i.imgur.com/VmRQ1Fz.png

Here is one question, as I can understand you meant trading volumes here. What if buys is zero in current slot? or more over what if sells volume is zero or very small? In those cases the metric will have no meaning. Or what did you mean?

I'm not sure I understand what extra information is conveyed by the points on the VWAP line that's not already in the volume chart - as far as I can tell the set of points on the sell line show that a bunch of trades happened around 14:33 but only one of them had large volume as indicated by the corresponding point on the volume chart. The fact that the price didn't change seems incidental.


Yeah, I like the separate volumes for buys and sells a lot. Though the problem with using two line plots for them is 1) the line one with the higher volume in a timestep is going to obscure the lower one and 2) it's hard to tell how much total volume was traded for that timestep. A stacked bar graph with the buy volume on top of the sell volume would solve both problems and as a bonus is visually more clear - with a line plot the slope between points suggests the value is changing between points while a bar better represents the "slot" being filled with volume.

If you zoom the VWAP chart you will see something like this:



The markers are not for every point, but for each point (15-second "slot" as we call it) with trading activity. So here you can see that a large trade activity at 14:29:45 shifted VWAP, but a larger trade volume at 14:33:00 had almost no impact on the price. Removal of the markers will obscure this information.

Of course when you zoom out to default zoom level it starts looking ugly, but we didn't knew how we can improve it without information loss.

In addition you can see here why we chose to plot "buy" and "sell" trades separately instead of joining them with a single line as our competitors do. We show that trades are performed at two different steady prices instead of zigzag pattern you often see on last trade price charts without bid-ask separation and volume weighting:

Seems like that would do the trick. By the way by "/" I meant the ratio, editing to clarify.

The charting library we use supports post-processing with moving average. See the input box at the bottom chart at http://dygraphs.com/tests/range-selector.html

If it's ok we can easily add that input box for moving average window size to all the charts.

I'm not sure if there's a term for it, but total bids:total asks seems clear enough. I'm not sure if filtering would improve it and if so which filter, so again it would be nice to capture some data and compare.

Another one that would be good to have is a moving average of the buys volume:sells volume ratio over windows of e.g. 10m, 1h, 3h, etc.

This what I suggested some time ago for discussion. And I think in this case it would be good to use filtered  raw data to have meaningful estimation for the price.

How can we call this parameter  - expected market price or general market expected price? or? Is there any existing parameter for this in financial world already?

Another idea, the raw total volumes could be used for a total bid USD/total ask BTC chart, would be interesting to see how that correlates with price.

Nevermind that request, I made a chart for the data and "fake" orders do indeed have a large impact.


I think picking one is fine, which one to use depends on the purpose of the chart. The exponential function (a=0.1) prioritizes orders that are very close to the current price and hence should be a better indicator for short term market movement. The bandpass function (a=0.15) is closer to the "true" total volume but is more sensitive to volume changes that are very far away from the current price. The linear function is midway between the two but I'm not sure if it can match the benefit of either. I would run all three for a while and see what they look like, and then analyze the strength and weakness of each.

More UI suggestions: 1) Make the height of the chart area scale with window size, if you've got chart height scaling working it would be really nice to hide a few of the charts and have the rest be taller. 2) Move the axis labels to the right side of the charts, that's where new data is coming in so that's where people will be looking. 3) Add an option for a dark colour scheme, I think most people prefer something like http://bitcoinity.org/markets or http://bitcoin.clarkmoody.com/. 4) On the VWAP line there are markers for each point, but not on the other price lines. I think it looks better without markers.

Without a detailed rundown of your system my ability to evaluate your performance capabilities is limited but I'll try.

1) Server-side performance shouldn't be a problem for these filters, they only take O(n) additional arithmetic operations per update (n = number of orders in the orderbook) and the same amount of memory as whatever you're using to store the volume data right now.

2) When sending data to the client during an update, do you send all data or just the update delta? I'm not sure what sort of data the client keeps track of, but assuming you have all the calculations done server side the update delta should be very small, for each graph you should only need to send one additional datapoint per line per update.

3) It seems the client is receiving data for all currencies and exchanges, even the ones that aren't shown. You can avoid some data transfer by sending it selectively. For example, I only care about mtgox data in USD,  there are 13 other tabs containing data I don't need. What I would do is have an option in the user's preferences for the default exchange and currency, and only load and display that data by default. Request and subscribe to the other tabs only when the user requests it.

4) Similar to 3), you don't have to show all charts by default and the client doesn't need data for hidden charts. As I suggested before, add a control to hide/show each chart, and have an option for a default set of charts in user preferences. Only request and subscribe to data for a chart if and when it's shown.

For each series you'll have 24*60*4 = 6240 datapoints. Each datapoint is two floats, i.e. 8 bytes, yielding about 49kB of raw floats per series (not sure how the data is structured, e.g. if you're using JSONs there will be more overhead). I'm not sure what kind of compression ratios you're getting, a cursory search suggests gzip doesn't do a very good job of compressing floats. Still, it shouldn't be a problem for most people and you may be able to reduce it further by using a compression algorithm that's optimized for floats.

With regards to the filtering I suggested, I think there's no real point sending the raw data to the client; as it is the data is only useful for showing relative change in volume for either bids or asks. Seeing as orders that are very far from where the trades are happening don't really affect the prices, I don't think you lose anything by reducing the weights assigned to them. In fact by discounting spam orders and far-out orders it would better link relative change in volume with change in price since the orders that matter are given more importance.

Hi, Masqurin,

this is exactly a kind of thoughts we really like and appreciate. Thank you for your time and efforts.

We must admit that the density of information you gave us is quite high and we need to digest different parts of it to be able to understand what is easy and what is not to implement in our current application architrecture.

It would be really helpful if we can exchange further to see what limitations we currently have and what can be implemented and in what order.

As for the math of the filtering it not a problem to implement filtering as is, but the next question is:

currently while preparing data for display client we need to make some steps to solve several problems

1) our server should be able to gather and recalculate all parameters once per 15 seconds and not overload the processor

2) we then need to store limited part (not all) calculated paramateres in memory in 15 second buckets in such a form to not exceed the memory limit of our machine

3) then we need to prepare data to transfer it to the client browser - and again to have resonable amount of data to transfer (keeing in mind that we already are using float data type, gzipping) it takes some times up to 10 seconds to load the whole data to browser

So adding additional parameters to calculate is not the main problem, but to decide what to transfer to client and and how to display it could be a difficult task.

For example: as for the orderbook volume we have two series per ticker, and those data currently represent actual sum of all USD and BTC in orderbook even if orders are spam orders, and if we decide to filter out spam orders in order to visualize bids and asks series in the same units then we will loose our original genuine data which is also make sence in its own context (real not filtered amount of USD and BTC in orderbook)

What do you think about this particular problem what would be good solution from the user point of view to maintain both series together on the client to be able to compare them or just to leave filtered and to abandon the original full volume series?


I mean you prefer to have both pair series or only one pair?

also as each filtering method will produce differernt series would it be practical to maintain all of them (thus multiplying serires) and toggle between them or just one. If one which one would be the most practical for analyzes on your opinion?

One more thing instead of using a line plot for depth and trade volumes it might be better to use a stacked bar chart. Right now it's really hard to hover over one exact datapoint to see its values, and it doesn't really take advantage of horizontal scaling.

Ok, I made some charts to demonstrate. The total discounted volume for bids and asks in USD is shown beneath each.

Here's the raw data I was working with
https://i.imgur.com/yPjLtle.png
Bids: 12170767.4644583
Asks: 27476862169015.9

And here's the depth chart up to twice the top bid
https://i.imgur.com/BRxe2b6.png

Here is the exponential discount factor I described before, it prioritizes prices close to the spread.

The formula for the discount on bids is e^(-a(highestBid-bid)), and e^(-a(lowestAsk-ask)) for asks.

Here's the discounted volume:
https://i.imgur.com/CSZjOCi.png
Bids: 105023.402931941
Asks: 23900.6565377794

That was a little too restrictive, relax the scaling parameter a bit to a=0.1:
https://i.imgur.com/K0SOpL1.png
Bids: 1522439.36415067
Asks: 1236490.55085024

It can be relaxed further to include points further down the line:
https://i.imgur.com/b8VT37W.png
Bids: 3262099.42799612
Asks: 2777887.40515259

Alternatively to give full weight to the majority of the orders, use 1-e^(-a*bid) for bids, and max(0, 1-e^(a*(ask-lowestAsk)-highestBid))) to create a symmetrical bandpass filter about the spread.

Here's what it looks like with a = 1:
https://i.imgur.com/1aToRt2.png
Bids: 11970104.7601015
Asks: 9177170.77805910

Restrict the further out values a little with a = 0.15:
https://i.imgur.com/PowevBb.png
Bids: 11091817.7195716
Asks: 8799423.98610072

A simple linear discount function, use 1-(highestBid-bid)/highestBid for bids and max(0, 1-(ask-lowestAsk)/lowestAsk) for asks
https://i.imgur.com/9b6neS5.png
Bids: 6158532.21541063
Asks: 5225588.99409854

Depending on what kind of analysis you want to make I'd say either the bandpass function with a=0.15, exponential function with a=0.1, or linear function would all work fine.