Author

Topic: Can someone with more hard and software knowledge than me explain why MtGox.... (Read 1139 times)

hero member
Activity: 756
Merit: 522
But the moment you start doing this sort of thing on commodity hardware you start running into limits like this. Somebody measured 37 transactions per second as their limit.  That almost sounds like there's a conventional HDD in their trade engine's processing system somehow.

There's some numbers on random writes per second at http://en.wikipedia.org/wiki/IOPS - there's a very large spread of performance ranging from HDD and low end SSD through the really expensive stuff.  The range is from ~100 operations per second for HDD through nearly 10 million/sec on a fusionio card.  That's a spread of 100000x between slowest and fastest.

This is why qualified people despise wikipedia researchers.
sr. member
Activity: 351
Merit: 250
I'm always grumpy in the morning.
Mt. Gox has the unique problem that they can't reverse transactions, which means their security and stability requirements for their trading engine are significantly higher than faced by conventional exchanges. Especially with high-speed trading engines it's common to cut a lot of corners with regard to security and stability because if the system does screw up the problem can be fixed after the fact. If Mt. Gox's system screws up and someone withdraws the Bitcoins they shouldn't have obtained Mt. Gox can do nothing to reverse that transaction.

Theoretically they could fix that by putting newly obtained funds in a quarantine, ie. if you just bought BTC then that money can't be withdrawn for a given time period. Not sure how popular that would be though.
legendary
Activity: 1120
Merit: 1152
FWIW I have heard from third-parties that Mt. Gox runs their trading engine on the slow, but exceptionally secure, hardware security modules used by banks for things like ATM machines.

Mt. Gox has the unique problem that they can't reverse transactions, which means their security and stability requirements for their trading engine are significantly higher than faced by conventional exchanges. Especially with high-speed trading engines it's common to cut a lot of corners with regard to security and stability because if the system does screw up the problem can be fixed after the fact. If Mt. Gox's system screws up and someone withdraws the Bitcoins they shouldn't have obtained Mt. Gox can do nothing to reverse that transaction.
sr. member
Activity: 351
Merit: 250
I'm always grumpy in the morning.
They're using SQL transactions in their trade engine.  That brings an awful lot of speed limitations.

There's parts of me itching to write a custom trade engine in C++ when I read that.
member
Activity: 77
Merit: 10
For amusement purposes:  http://leveldb.googlecode.com/svn/trunk/doc/benchmark.html
Quote
Note about Ext4 Filesystems

The preceding numbers are for an ext3 file system. Synchronous writes are much slower under ext4 (LevelDB drops to ~31 writes / second and TreeDB drops to ~5 writes / second; SQLite3's synchronous writes do not noticeably drop) due to ext4's different handling of fsync / msync calls.

One could imagine scenarios where they could be running their SQL database in an ext4 and getting a similar bottleneck.

Sadly, they give very so all that's left is guessing and speculation like this.
full member
Activity: 140
Merit: 100
Well they certainly didn't give any proof of stellar technical competence so far. As a dev, yes, I do think they are amateurs aswell.
member
Activity: 77
Merit: 10
They're using SQL transactions in their trade engine.  That brings an awful lot of speed limitations.
transactions are important, unless you wants to lose data.

Oh I know.  There's also some famous questions on stackexchange from (?) bitcoin-24 people: http://stackoverflow.com/questions/15026825/php-mysql-how-to-prevent-two-requests-update and we know how that all turned out.

The financial / big-iron folks spend a lot of money on doing this fast.  The sky's the limit.  eg: Battery backed ram (search: nvdimm) is just one more iteration. IBM still make wheelbarrow loads money on this.

But the moment you start doing this sort of thing on commodity hardware you start running into limits like this. Somebody measured 37 transactions per second as their limit.  That almost sounds like there's a conventional HDD in their trade engine's processing system somehow.

There's some numbers on random writes per second at http://en.wikipedia.org/wiki/IOPS - there's a very large spread of performance ranging from HDD and low end SSD through the really expensive stuff.  The range is from ~100 operations per second for HDD through nearly 10 million/sec on a fusionio card.  That's a spread of 100000x between slowest and fastest.

Personally, I've had great luck with the stec zeus ram / iops devices.  The differences in database throughput that involve transactions relative to generic hardware are profound.

I should know better than to drop a 1-line quip on a topic that's not so simple.  It would have ben better to say that the transaction throughput is heavily dependent on their back-end hardware if their trade engine is operating at the SQL transaction level.

There's other ways to do this sort of thing but that's an entirely different topic.  And orders of magnitude more work and harder to get right.
donator
Activity: 1218
Merit: 1079
Gerald Davis
They're using SQL transactions in their trade engine.  That brings an awful lot of speed limitations.

So does just about any financial service and they handle transaction volume in the thousands of transactions per second (with peaks in the tens of thousands).
legendary
Activity: 1050
Merit: 1000
You are WRONG!
They're using SQL transactions in their trade engine.  That brings an awful lot of speed limitations.
transactions are important, unless you wants to lose data.
member
Activity: 77
Merit: 10
They're using SQL transactions in their trade engine.  That brings an awful lot of speed limitations.
sr. member
Activity: 280
Merit: 250
... try to avoid abbreviations.

Mt.GOX = Magic The Gathering Online Exchange (really!)

It doesn't require much hard or software knowledge to figure it out from here.  Grin
hero member
Activity: 672
Merit: 500
Is still lagging and messing up.

The facts that i know is that they are TINY compared to the big trading sites. So don't tell me that is the real problem. They really aren't doing that many trades.
They have the money to buy/rent the best hardware and software and pay the best server gurus.
They have had months now to fix their shit.

My conclusions is they are a bunch of clueless amateurs unless i'm missing something.

Can someone point me out if i'm wrong in my thinking somewhere?
Jump to: