The coming change at mtGox and its implications - page 2.

damnek

hero member

Activity: 614

Merit: 500

Quote from: Hawkix on March 29, 2013, 05:01:17 AM

Quote from: molecular on March 29, 2013, 04:37:53 AM

Last night I finally understood what's with the gox engine lag:

Quote from: #mtgox

...
molec: we need to change the way users place orders to solve this
MagicalTux, in what way?
disallow users to place orders they can't fill
MagicalTux, +1. oh...
MagicalTux, it's dawning on me
and disallow users to withdraw balance required for open orders
MagicalTux, you have to trickle down the whole order list to set invalid/open status
MagicalTux, go do it!
right now each time a user balance change (trade, etc) we need to recheck all his open orders
...

I think that with proper data structures, one can link together the bids and asks of single user and immediately update "reserve" for the bid/ask after sell or buy. Funding and withdrawing are infrequent compared to trades. The structure can be held in memory, with commits to permanent storage. This should handle hundreds of trades per second, not a few, like it is now.

Of course, this requires to avoid using normal relational databases, queries upon them, PHP code in server, etc.

Also, the socket with trade data is sometimes overloaded, too. I am not a TCP/IP expert, but I cant see problem with cloning the data between multiple (in order of thousands) sockets open. This would solve all those disconnects on ClarkMoody, MtGoxLive, etc.

An exchange, which cannot provide reliable and non-lagged data for its users, is .. unusable. Like that graph at the top of mtgox.com. Isn't it ironic that to trade at MtGox, or just watch it, you need to use third party tools?

Absolutely agree with you. I've been long enough to know that bitcoin and exchanges are not the same thing but I can't help to think that this market got too big for what the infrastructure really is. It's just not there yet.

Hawkix

hero member

Activity: 531

Merit: 505

Quote from: molecular on March 29, 2013, 04:37:53 AM

Last night I finally understood what's with the gox engine lag:

Quote from: #mtgox

...
molec: we need to change the way users place orders to solve this
MagicalTux, in what way?
disallow users to place orders they can't fill
MagicalTux, +1. oh...
MagicalTux, it's dawning on me
and disallow users to withdraw balance required for open orders
MagicalTux, you have to trickle down the whole order list to set invalid/open status
MagicalTux, go do it!
right now each time a user balance change (trade, etc) we need to recheck all his open orders
...

I think that with proper data structures, one can link together the bids and asks of single user and immediately update "reserve" for the bid/ask after sell or buy. Funding and withdrawing are infrequent compared to trades. The structure can be held in memory, with commits to permanent storage. This should handle hundreds of trades per second, not a few, like it is now.

Of course, this requires to avoid using normal relational databases, queries upon them, PHP code in server, etc.

Also, the socket with trade data is sometimes overloaded, too. I am not a TCP/IP expert, but I cant see problem with cloning the data between multiple (in order of thousands) sockets open. This would solve all those disconnects on ClarkMoody, MtGoxLive, etc.

An exchange, which cannot provide reliable and non-lagged data for its users, is .. unusable. Like that graph at the top of mtgox.com. Isn't it ironic that to trade at MtGox, or just watch it, you need to use third party tools?

oakpacific

hero member

Activity: 784

Merit: 1000

We need to pay attention especially to a rapid price movement scenario, when people will be trying to issue orders desperately, and may never bother to check on their balances, and Gox's UI irresponsiveness only adds salt to the injury, as people will try to issue more orders when they can't see their orders showing on the list. The more orders people issue, the greater the lag, which would make people more anxious and making more vain efforts, which in turn causes even more lag. This vicious closed-feedback loop causes the ridiculous 10-minutes lag we saw on a weekly basis.

damnek

hero member

Activity: 614

Merit: 500

Quote from: molecular on March 29, 2013, 04:53:20 AM

Quote from: genuise on March 29, 2013, 04:47:02 AM

Does it also mean that ask side is also working the same way? So this is also bearish?

The situation I described had bearish result (rising market, no more auto-opening of bids => perceptively less bids). The other way around has bullish result: in a crash or falling market no more auto-opening of asks => ask side looks less impressive.

So if I judge this right, it could dampen the moves (up and down).

But keep in mind: this is mainly by perception. If people judge this correctly, it should not have much impact at all.

EDIT: just thought again: of course the order-book will be thinner, maybe this could mean that rebounds will be even quicker.

I sometimes use the current system to actually dampen some moves. For instance last night I set a sell at 92 USD and a bid, while not having enough USD funds, at 88.

molecular

donator

Activity: 2772

Merit: 1019

Quote from: Zomdifros on March 29, 2013, 04:56:38 AM

With the transition of North American customers to Coinlab, are they still going to use the same trading engine (in Tokyo?)? If all these orders will be done on a new engine in the US, this could drastically reduce the pressure on the current system. If so, when will this transition happen?

I can't imagine them running 2 markets. That doesn't make sense.

Zomdifros

full member

Activity: 210

Merit: 100

With the transition of North American customers to Coinlab, are they still going to use the same trading engine (in Tokyo?)? If all these orders will be done on a new engine in the US, this could drastically reduce the pressure on the current system. If so, when will this transition happen?

molecular

donator

Activity: 2772

Merit: 1019

Quote from: genuise on March 29, 2013, 04:47:02 AM

Does it also mean that ask side is also working the same way? So this is also bearish?

The situation I described had bearish result (rising market, no more auto-opening of bids => perceptively less bids). The other way around has bullish result: in a crash or falling market no more auto-opening of asks => ask side looks less impressive.

So if I judge this right, it could dampen the moves (up and down).

But keep in mind: this is mainly by perception. If people judge this correctly, it should not have much impact at all.

EDIT: just thought again: of course the order-book will be thinner, maybe this could mean that rebounds will be even quicker.

oakpacific

hero member

Activity: 784

Merit: 1000

Does this have anything to do with the original requirement of a card trading exchange?

genuise

sr. member

Activity: 379

Merit: 250

Does it also mean that ask side is also working the same way? So this is also bearish?

oakpacific

hero member

Activity: 784

Merit: 1000

If I just did a calculation with my ass, it's gonna make the complexity roughly O(m*log(n)) instead of O(log(n)), and m could be easily larger than log(n).

I suspect there are many badly-programmed bots relying on this out there.

molecular

donator

Activity: 2772

Merit: 1019

Last night I finally understood what's with the gox engine lag:

Quote from: #mtgox

sturles: if you need realtime data, use socket.io
MagicalTux, "realtime" Huh

;-)
molec: socket.io feed is actually realtime
MagicalTux, so it's the engine that can't cope?
right now, yep
because it needs to do too many checks
we are working on changing the way all this works
but it needs a *lot* of changes
MagicalTux, I think you're doing something wrong. I can't see why it's hard... what's the reason?
MagicalTux, might be my ignorance, but I did quite a few dbs. I might be overlooking some problem,... maybe security-related?
molec: we need to change the way users place orders to solve this
MagicalTux, in what way?
disallow users to place orders they can't fill
MagicalTux, +1. oh...
MagicalTux, it's dawning on me
and disallow users to withdraw balance required for open orders
MagicalTux, you have to trickle down the whole order list to set invalid/open status
MagicalTux, go do it!
right now each time a user balance change (trade, etc) we need to recheck all his open orders
MagicalTux, I see. and that gives you concurrency issues like hell
we're working on solving this, but changing the way the site works need to be done right
so we're doing a lot of tests first
MagicalTux, I agree. don't haste
well
we need to haste, because current volumes cannot be handled anymore by current engine
we made it faster (no more 30min delays), but it still has troubles with large moves

What this means is: you can't have any order in the book that aren't backed by funds. Currently "un-backed" orders will have status "invalid" and wont show in the public order book, but they can exist and once funds become available to back the order, it will change its status to "open".

What are the implications of this?

Currently, in a uptrend market like we have had for a long time, when price rises and people who have invalid bids get their asks filled, fund become available and (some of) the bids of that user appear automatically in the orderbook. As I see it, this mechanism is rather bullish, because it keeps the bid-side of the book nicely filled up.

Once these invalid order cannot exist any more due to the change, the bid-side will not fill up like that any more. This might be interpreted by the market participant as a weakness of the bulls, especiall if the participants are not yet fully aware of the change.

On the bright side: we can expect the trading engine to perform much better.

Topic: The coming change at mtGox and its implications - page 2. (Read 5064 times)