Topic: mtgox servers (Read 2835 times)
Actually what I meant to say is that their main line into their server room(s) has to be a 1000gbps capable line.. and they need a minimum of 10gbps bandwidth to survive . their servers behind whatever protection they come up with can be 2gbps .. no problem.
If your systems are behind Nic cards that are 2gbps and you are being hit by a DDos of 10gbps.. how are you supposed to protect from that? That size DDos will just halt all traffic just from the bottleneck effect.
The database on the main server should be entirely on SSD. It's small enough.
RAID1 is appropriate, as disks are cheap, and they really aren't using much space. Downtime is more expensive than disks.
There should be a secondary server with a copy of the database replicated in real time just for handling read only transactions, like clarkmoody.com, bitcoin itty.org, etc. And a backup for it, too. The main server is obviously still overloaded.
They are underconfigured andneed to stop thinking smalltime!
But most important, they need to stop th spambots from choking the system with .001 BTC orders. An easy way to do that is set a minimum fee per trade of .006 BTC. This would only impact the nuisance orders, since any reasonable person is trading at least 1 BTC.
RAID1 is appropriate, as disks are cheap, and they really aren't using much space. Downtime is more expensive than disks.
There should be a secondary server with a copy of the database replicated in real time just for handling read only transactions, like clarkmoody.com, bitcoin itty.org, etc. And a backup for it, too. The main server is obviously still overloaded.
They are underconfigured andneed to stop thinking smalltime!
But most important, they need to stop th spambots from choking the system with .001 BTC orders. An easy way to do that is set a minimum fee per trade of .006 BTC. This would only impact the nuisance orders, since any reasonable person is trading at least 1 BTC.
hmm looked under kill to me by far
You can't run an exchange as a single-stack application. No way it will scale.
The fact that they are still trying to throw hardware at this means they simply do not understand scaling.
They will not solve these problems because they don't have a clue how to solve them
RUN AWAY, don't just walk away.
The fact that they are still trying to throw hardware at this means they simply do not understand scaling.
They will not solve these problems because they don't have a clue how to solve them
RUN AWAY, don't just walk away.
So... does it seem to anyone that their hardware is extremely underutilized and their software is inefficient?
4x 8 core CPUs and shit tons of RAM should be able to handle more than 1 transaction per second.
4x 8 core CPUs and shit tons of RAM should be able to handle more than 1 transaction per second.
Yes it likely should be sufficient to handle 100x that, assuming the single box was used as dedicated trading engine with other servers used to offload other tasks (API access, webserver, support, charting, etc).
LOL Raid 1 on their backup data server? WTF? Raid 1??? Even Raid 5 is no good... Spend less on equipment, and more on someone that knows what they are doing.
So... does it seem to anyone that their hardware is extremely underutilized and their software is inefficient?
4x 8 core CPUs and shit tons of RAM should be able to handle more than 1 transaction per second.
4x 8 core CPUs and shit tons of RAM should be able to handle more than 1 transaction per second.
is it really for the DDOS protection?
I tough that to be a backend database server, inaccessible from the outside, behind a pool of frontend servers... Still, 2Gbps of dedicated bandwidth to handle DDOS? It's not gonna scale, after all...
But who am i to criticize their setup?
Edit, the 2Gbps uplink might be internal; they do have an external DDOS protection service.
But who am i to criticize their setup?
Edit, the 2Gbps uplink might be internal; they do have an external DDOS protection service.
Quote
traceroute to mtgox.com (72.52.5.67), 64 hops max, 52 byte packets
1 10.0.0.10 (10.0.0.10) 0.468 ms 0.383 ms 0.248 ms
2 * 15.757 ms 15.461 ms 15.779 ms
3 wf2.mi.sw-gw1.seflow.it (158.58.168.1) 14.648 ms 19.250 ms 16.862 ms
4 caldera.dc2.hsr1a.seflow.it (95.141.47.254) 45.370 ms 172.757 ms 223.565 ms
5 ge4-12.mil01-1.eu.as5580.net (78.152.32.201) 34.848 ms 17.582 ms 21.405 ms
6 tge1-3.par02-1.fr.as5580.net (78.152.34.109) 36.162 ms 41.075 ms 34.076 ms
7 tge1-1.lon01-1.uk.as5580.net (78.152.34.74) 50.626 ms 42.321 ms 47.633 ms
8 blackhole.prolexic.com (195.66.224.31) 43.664 ms 44.109 ms 43.942 ms
9 unknown.prolexic.com (209.200.156.34) 43.583 ms 43.484 ms 43.788 ms
10 unknown.prolexic.com (72.52.5.67) 43.222 ms 43.045 ms 43.080 ms
1 10.0.0.10 (10.0.0.10) 0.468 ms 0.383 ms 0.248 ms
2 * 15.757 ms 15.461 ms 15.779 ms
3 wf2.mi.sw-gw1.seflow.it (158.58.168.1) 14.648 ms 19.250 ms 16.862 ms
4 caldera.dc2.hsr1a.seflow.it (95.141.47.254) 45.370 ms 172.757 ms 223.565 ms
5 ge4-12.mil01-1.eu.as5580.net (78.152.32.201) 34.848 ms 17.582 ms 21.405 ms
6 tge1-3.par02-1.fr.as5580.net (78.152.34.109) 36.162 ms 41.075 ms 34.076 ms
7 tge1-1.lon01-1.uk.as5580.net (78.152.34.74) 50.626 ms 42.321 ms 47.633 ms
8 blackhole.prolexic.com (195.66.224.31) 43.664 ms 44.109 ms 43.942 ms
9 unknown.prolexic.com (209.200.156.34) 43.583 ms 43.484 ms 43.788 ms
10 unknown.prolexic.com (72.52.5.67) 43.222 ms 43.045 ms 43.080 ms
5000$ for 2 servers with less than 100GB of ram each? Who is their provider, rackspace, softlayer?
Suggest them to buy and colocate, Jesus!
Suggest them to buy and colocate, Jesus!
It's probably the double gbit links that are pricey. In some locations 100mbit dedicated can run you up 1k per month.
I agree. 2GB/s of WAN data is spendy, no matter where you are. It could very well be the bulk of the cost.
5000$ for 2 servers with less than 100GB of ram each? Who is their provider, rackspace, softlayer?
Suggest them to buy and colocate, Jesus!
Suggest them to buy and colocate, Jesus!
It's probably the double gbit links that are pricey. In some locations 100mbit dedicated can run you up 1k per month.
isn't bitcoin itself rather cpu intensive?
Not really. I mean not 32 high end server cores intensive. We aren't talking mining. Bitcoind's (i.e. running a node for the backend) bottlenecks are I/O & memory. You got me curious so I checked. Peak CPU time on our bitcoind process is ~2% (excluding when it was bootstrapping which was more like 4%) and that is on a Single CPU (quad core xeon).
Even if bitcoind was CPU intensive the current codebase is single threaded so you aren't going to use more than 1 core anyways (so the 2% is more like 8% of a single core). If you are running on a atom based netbook or a oversubscribed VPS instance you might have problems but any dedicated hardware won't blink.
isn't bitcoin itself rather cpu intensive?
5000$ for 2 servers with less than 100GB of ram each? Who is their provider, rackspace, softlayer?
Suggest them to buy and colocate, Jesus!
Suggest them to buy and colocate, Jesus!
RAM prices have really fallen in the last 18 months (more than the usually 50% drop every 2 years).
It is strange they wouldn't buy and colocate though. Especially for that high end. Paying $5K to $10K for the hardware and then cutting your monthly cost to <$1K per month would for a private cabinet would have paid off a long time ago. Then again maybe Japan has some insanely expensive datacenter costs no matter which way you go?
The specs don't look overkill except maybe the 32 cores. Nothing they do is that CPU intensive. Lots of memory and some fast disks makes sense given the db centric nature. I am surprised they didn't seperate it out into db server and web/front end server though. Still I doubt they are getting much utilization of CPU side. If they are it would be cheaper to go with some high end NICs (TCP/IP offloading) and a HSA then trying to bang it all out with expensive CPU cores.
5000$ for 2 servers with less than 100GB of ram each? Who is their provider, rackspace, softlayer?
Suggest them to buy and colocate, Jesus!
Suggest them to buy and colocate, Jesus!
If you are not looking for HW on ebay and for the cheapest possible reseller of server housing services, then the price seems to be ok.
i was just looking at this:
https://mtgox.com/img/pdf/20120831/Transparency.008.jpg
does anyone else think that this server setup is way overkill for gox?
maybe i underestimate its traffic.
https://mtgox.com/img/pdf/20120831/Transparency.008.jpg
does anyone else think that this server setup is way overkill for gox?
maybe i underestimate its traffic.
It's not.
Yeah if anything MtGox needs even better servers, or at least better ddos protection. They seem to get knocked off far too often
Well, figure they should be using at least two layers of encryption (database and Bitcoin keys), so it's probably more taxing then you think.
Jump to: