Topic: [ANN Mt.Gox] It’s been an epic few days: What happened? (Read 4042 times)

wow, what a bitch!

I'm guessing the PR in your name is there for irony?

Digging up a slightly old thread...

I was curious if those 57,000+ accounts created recently are legitimite.  Obviously this information would inspire the price to go higher (and I believe it did help up to $266).  I could see it as another way to attack Gox by creating a bunch of false accounts and swamp their customer service.

Just a thought..

And it has been actually designed (correctly) and actually implemented (correctly), by people who actually matter and actually do things (unlike you + friends). Read it, because you must read it. Stop with the posturing, you are not part of the cool kids, you're a dork "working on a soft drink distribution system" with "some friends". It's called Coca-Cola, dummy. It's listed under KO. You're embarassing yourself.

I'd also suggest Tor hidden services, and using private hidden service URLs that are different for each partner. The URL's don't give any information about the network topology, and if any individual URL is compromised and DoS attacked the individual URL can easily be taken down. Even URL's for individual high-volume traders would be feasible, although, keep in mind I'm no Tor expert so someone who knows more should weigh in before doing this. In particular what the Tor developers think of the attention and attacks it may attract.

In some cases using Amazon's infrastructure could work too. Amazon Simple Notification Service is essentially a DoS-resistant broadcast medium. Of course, it's central infrastructure, so not appropriate for every use, but  there is a lot of public information like price tickers that could use it. In any case information should be distributed though multiple methods.


Regardless of what is done, an important first step is to sign and timestamp all public information broadcasts so that regardless of where you got the data, you can verify it as being genuine; the current API does not appear to authenticate pricing information other than via https.

+1 to the private UDP broadcast idea.

Further, there has been talk off and on (by myself, and others) about a "backbone network" where Big Players privately and directly interconnect, for reasons similar to this.

Yup as working as in ITer in a DC, if they want to hit you with DDoS a lot of tricks work, till they either flood the lines or bring down the router of the DC, where you are at. Once they hit that point, there is no stopping it. And even though the real big DC's have a dozen of 10gbit lines, at some point it just stops. (A month back we saw an DDoS we saw a DDoS peak of 250gbit hit... what do you do about it... you null the IP at your carriers. (making the site unavailable))
Though when hackers/botnets are attacking in this type of volume, there is nothing that can really save you, the only way to minimize, is host global, all with loads of overcapacity, high performance routers and heavy duty DDoS firewalls. This is the only way to truly mitigate most of them. (if they attack from US, only the US DC will get hit, and EMEA and Asia will hopefully keep running smooth, same goes for both other regions when under attack)

DDoS is one of the most difficult things to combat, purely because its nature is to flood the network, hence extra capacity would be needed to combat this. (you can't buy 100gbit of network speed when doing 1gbit of traffic just because you might get hit with a DDoS up to 100gbit... and what if they DDoS you with 101gbit?) Seeing what DDoS botnetworks have been capable of, these days you are technically not safe with anything under 2tbit.... (highest I've seen was 1.3tbit) well if your in the hosting business, you know what a 10gbit uplink costs, multiply by 200, subtract 30% a month... while it would only take a botnet of half a million with 20mbit connection to push that same DDoS out. (known botnets have been found with more then 300k bots) And then I'm not even talking about some infected webserver, on a gbit connection to the internet, that can start to DDoS as part of a botnet... (only takes a 1000 of these to get a 1tbit)

Still I think Mt.Gox can do better, DDoS can't always be completely prevent, but bot lagg time in triple digit seconds as shown earlier in the thread... that should've been impossible imo. Get that security ramped up and make sure everything is running efficient. Know you get hit by DDoS on regular basis, and apply as much pre-emptive mitigation methods as possible.

You can run java very lean if you know what your doing and not adding a hundred libraries to it. But this is one of those flame wars that will start cause you like those languages better, in any case what ever language would be better.



Again this is what language you like better, and will probably start a flame war. Also C/C++ if they don't know what they are doing can have massive memory leaks with all that data.

These are my thoughts to an extent.
Down with Walmart. *ren and stimpy stinky face*
Anarchy! 

A high performance trade engine should be written in C/C++, simple as that.

This may work, but you will have to make arrangements with the DC hosting, which is something they may or may not be able to do (from the looks of it, more like not).


Ahaha can I have that engraved on a medallion? 

Why?

I would like the trade engine to be written in Haskell, Erlang or some flavor of Lisp. Just because a functional language makes it easier to keep the code clean of unintended consequences. Also, Java is very bloaty, not a good thing for high performance code.

However, I understand sometimes compromise is necessary.

Yes, UDP stands for Useless Deprecated Protocol, it's that protocol that automatically detects and cunningly removes any information and redundancy you add to transmissions to help an application using it detect and recover from losses of datagrams, thereby making it useless as an upstream data source for scripts.  It is also a protocol that is specialized in conveying data that, by virtue of having traveled via UDP, becomes impossible to republish on a TCP stream for the benefit of being consumed by scripts, to help increase the workload of script writers who of course will want to implement a UDP listener in their scripts directly.  It is commonly known as an unreliable protocol, and it gets this reputation by sneakily altering important data in such a manner where it cannot be made reliable through other methods.  In fact, it even warps the minds of people considering using it, such that they cease to even know what UDP is!

Do you know what UDP is? If they switched to UDP now all PHP/Python/java scripts would be useless and take now even more code to just connect. Also trading bots need reliable connections which UDP doesn't support by any means. UDP is great for the same information broadcast over and over, like said above a radio. Trading shouldn't use UDP it makes it very unreliable. They just need either a better network architecture clearly.

I am not sure that trading bots and DDoS are considered the same problem from the view of the consensus here.

UDP isn't going to solve this problem, instead it make it harder for bots to trade. If they really wanted to solve this issue it is so simple. The trading engine should be ran completely offline, and use a database, like redis to store all information so the rest api can still have access to the information. Also the trade engine should be written in java or python.

Before it comes within miles of the host sending it.  After not informing the public who the UDP is coming from.

The UDP sending address doesn't have to be public knowledge, since not anyone can necessarily subscribe to it.  It would be a private UDP feed only offered to specific known sites.  The UDP feed would be used to drive the services of other sites who currently get it via websocket now, who in turn could provide that data to other downstream TCP websocket clients.

They say reading enriches the imagination.


You are rubbing sticks together trying to solve already solved problems. The many ways available for talking to MPEx.


Of course the clueless knowitalls haven't yet answered the

I guess there simply would not be be anything to broadcast during the times when the engine is down or when nobody can make any trades. It would broadcast only silence. Maybe it could broadcast "Help!!! Help!!! We're under attack!".

Thank you for the explanation and terrific post.


This is absolutely true. Some attacks are more application level (synflood, real HTTP requests), but others are of such a volume where the pipe is saturated. You'd pretty much have to have anycasted datacenters and massive pipes to the Internet to absorb large enough attacks. I've seen 3 Gbit/s attacks to minor sites for no apparent reason. I can't imagine what MTGox gets on a regular basis.

Re, the UDP suggestion. That might not be a bad idea at first glance. It'd work if MTGox advertised the price from some mostly unknown IPs and out different routers, out to a list of subscribers. Another option would be to put this data in DNS, maybe in a TXT or SRV record with a TTL of 60. Then the DNS servers might be attacked, which could be a new problem.

In my opinion, MTGox runs a great site. It's a bit tricky to get onto and the interface isn't as sleek as some sites, but ultimately, MTGox has single handedly encouraged a massive growth of adoption. I think ideally trades should be distributed by nature, but MTGox is still (and probably always will be) the benchmark site for Bitcoin trading, especially in bulk.

My hat is off to these guys for how thorough they are, dealing with the past through days, and 57,000 signups in one month. Those are some real challenges.

Why the lag? It's really not that many transactions. It has to be 90% the procedure / architecture of your engine.

Also +1 to UDP

At what level do you propose blocking the incoming traffic?

The point is that when merely using UDP, unlike TCP, the source can block ALL incoming traffic which makes it immune to DDoS. As casascius points out, UDP is like a radio broadcast signal. TCP is like the postal service with delivery confirmation.

I propose two ideas, i dont know if good ideas:

1-since the attackers are operating in order to get benefits by buy/sell, couldn´t gox proceed to identify them or putting under suspect some accounts?

2-stop the site while ddos occurs?

I don't think you understand how DDoS works on this level. Your UDP stream would have to have a source, which would have to have an IP, which then would get flooded to crap. It's the routers that pop, not the machines.

the attack can also happen in order to STABILIZE bitcoin. the more people are unsatisfied with mtgox, the more they will flock to other exchanges and STABILIZE the bitcoin ecosystem. we do not need one huge centralized exchange. remember this!

Cheers for the update. Ignore the haters 

Looky what I found:



Anyone recall?

Also MtGox could take a position on my UDP streams idea, which could be any of the following without commitment:

a) Great idea, we haven't thought of it, and you're right, it would totally get information out immune to DDoS, we'll consider it but like anything else will take time
b) Great idea, but we don't agree it would work as well as you think it will, or for (specific technical reason) won't work on our platform
c) We haven't got a clue as to what this means
d) I don't have a clue what this means because I'm not a developer or tech guy myself, but I have relayed your suggestion to someone more technical, and he says (response).  (Hopefully this suggestion is more valuable than to merely forward it blindly like the latest facebook meme, since MtGox's reputation is suffering and this will actually solve the claimed issue at hand)

Just to be clear, using UDP to broadcast ticker data would be, for all intents and purposes, IMMUNE from DDoS attacks, because such a stream consists solely of outbound traffic which is not influenced by inbound traffic.  Unlike a normal stream, there is no inbound overhead for packets to acknowledge or to keep the connection in sync, packets which can be drowned out in a DDoS attack.  UDP is much more like a point-to-point radio broadcast: the signal gets sent from point A to B even if nobody's listening

I didn't see anything in there about adjusting the trade fee schedule.

It used to be < $160,000 to get to 0.3% (and probably a lot lower), now it's upwards of $1,370,000, a factor of ten. When will you be adjusting it to make it in line with current prices?

One thing Mt.Gox could do is release what information they have on the attack(s). We all know it's a botnet of infected computers, but I would not count out the bitcoin community out from doing detective work and more.

Have to admit, I too am a sucker for folks who fess up to their failings, so props to you for doing that head-on without any crap; it affords you a level of credibility that ought to be a sober example for others who lack it in lumps, like the bullshitting BFL brigade.

If this is excellent I bet nobody eats what you cook.

If you were to make the list of "events which might have killed Bitcoin", MtGox's hack is indisputably at #1, above the Pirate heist, above the recent unexpected fork, above everything else. This is MtGox's legacy: they're a historical threat to Bitcoin' continued existence and a permanent nuisance in the day to day life of it. To further illustrate this point:


Let's skip to recent times:


It's so bad we have a special command to query it! And it's so funny people have taken to embellishing it:


The notion that you're running an exchange with multisecond lag is ridiculous on its face. I'm not even sure why this has to be spelled out, it's beyond ridiculous. It's like Monty Python's cheese shop, it's like Monty Python's "self defense classes", it's like a comedy routine. This is what we use you for, MtGox, comedic relief. You're not an exchange, okay? You're Bitcoin's very own Comedy Central.

So now, armed with this basic understanding of what's what and where we're standing, let us dissect MtGox's most recent load of bullcrap (continued).

Thanks for updating the community 

I agree with you, but let's say that the FinCEN announcement delayed a few things.

I really hope so because to date both services have fallen well short of the mark in keeping users informed.

We were using Black Lotus and runaway from them... And 10Gbps is pretty much nothing for us, we have this on weekly basis and EC2 doesn't have enough CPU/memory to handle our db on a single instance

We, Coinlab & Mt.Gox, will announce something on this matter soon.

Prolexic sucks, they are resellers. Go with Black Lotus and make them agree to guaranteed uptime protection they'll do it. You could also clone a backup elastic cloud image on Amazon that ssh's to your secure db and switch to it whenever you get hammered beyond 100Gbps

You are right (last 24hrs), I was stuck on the past 12hrs data

Thought the top was $147?

Alex, you mentioned the verification issue.

It's really unclear to me what is going to happen with this when US/Canadian users get transferred to CoinLab.  Will the accounts which are currently awaiting verification still be verified by MtGox or will US/Canadian customers be verified by CoinLab following the transition (ie, will those US/Canadian customers currently awaiting verification by MtGox need to start the verification process again with CoinLab)?  

The lack of recent information about the transition is worrying as customers were initially told that they needed to agree to CoinLab's ToS if they wanted to use MtGox following the transition but the information on the CoinLab site says that the funds of US/Canadian customers will automatically be transferred to the US.  

This implies that people will need to withdraw their funds from MtGox before the transition if they don't want those funds transferred to the US bank (which may present difficulties for unverified customers given the backlog on verification) or they'll have to register as a CoinLab user whether they want to or not in order to withdraw their funds.

It was previously stated that MtGox user data would not be transferred to CoinLab without user agreement (ie, MtGox users accepting CoinLab's ToS), but if the funds are being transferred automatically then how will users be able to access them without using CoinLab?

I know you have a lot going on, but the transition to CoinLab has to the potential to be disastrous if you don't keep your users fully informed and - quite frankly - you don't need any more things happening right now which undermine people's confidence in you.

I applaud you for this update and must admit that I was one of the people to go a little overboard.

This kind of communication goes a long way and is something you should do a lot more.

I however maintain that you had a *lot* of time over the past months to get a more efficient trading engine up and - given that it should be *the* top priority - the progress on that front is simply not satisfactory.

I, for one, will be moving to another exchange until Mt.Gox meets the standards that I expect from the leader of the industry.

P.S. Just to have some basis to my claims: I have a strong background in computer engineering, especially when it comes to big & performance intensive systems. I can assure you, I know the challenges that come with hosting infrastructure, a good trading engine, massively parallel architectures. Still, if prioritized the right way, results should have been there a long time ago.

Thanks for the excellent feedback.

There is one thing you can do that's relatively easy.  You guys should consider doing a UDP blast of ticker information to perhaps a dozen volunteer sites who can use this information to relay it into an independently-provided web socket feed so that it doesn't have to go through you.  By UDP, just in case this isn't clear, I mean User Datagram Protocol, the sessionless protocol supported by TCP/IP, the kind of one-way outbound packets you can send from your facilities that shouldn't be affected by an inbound DDoS flood.

Do you have enough reserves to suspend trading and meet payroll/vendor obligations until the new trade engine is ready for prime time (and maybe the verification queue is beaten to within an inch of its life, as well)? Not sure how many strikes against Gox there are going to have to be, before most smart people say "you're out!" and look somewhere else.

Personally, Namecheap is being denied 4 years of domain transfer/renewals of mine because of this insane high-low. I was just about to execute my bitcoin deposit to them before the latest of seemingly countless DDOSs. Bit-Pay loses out on its fee as well.

This thread is a great example of the communication that goes a long way towards defusing those kinds of reactions.

When your customers don't know what's going on they tend to assume the worst.

Thanks for the update

Dear Mt.Gox users and Bitcoiners,

It’s been an epic few days on Bitcoin, with prices going up as high as $142 per BTC. We all hope that this is just the beginning!

However, there are many who will try to take advantage of the system. The past few days were a reminder of this sad truth.

Mt.Gox has been suffering from its worst trading lag ever, 502 errors, and at one point some users were not able to log in their account. The culprit is a major DDoS attack against Mt.Gox.

Since yesterday, we are continuing to experience a DDoS attack like we have never seen. While we are being protected by companies like Prolexic, the sheer volume of this DDoS left us scrambling to fine-tune the system every few hours to make sure that things don’t go beyond a few 502 error pages and trading lag.

Why has Mt.Gox become the target of a DDoS attack?
It is not yet clear who is behind this DDoS and we may never know, but these actions seem to have two major purposes:

1.   Destabilize Bitcoin in general.
It is not a secret Mt.Gox is the largest Bitcoin exchange with more than 80% of all USD trades and more than 70% of all currencies. Mt.Gox is an easy target for anyone that wants to hurt Bitcoin in general.

2.   Abuse the system for profit.
Attackers wait until the price of Bitcoins reaches a certain value, sell, destabilize the exchange, wait for everybody to panic-sell their Bitcoins, wait for the price to drop to a certain amount, then stop the attack and start buying as much as they can. Repeat this two or three times like we saw over the past few days and they profit.

What can be done?
Believe it or not, there is pretty much nothing that can be done. Large companies are frequently victims of these kinds of attacks. Even though we are using one of the best companies to help us fight against these DDoS attacks, we are still being affected.

There are a few things that we can implement to help fight the attacks, such as disconnecting the trade engine backend from the Internet. By separating the data center from the Mt.Gox website, we will continue to be able to trade.

What can you do?
Like our favorite author here at Tibanne says… Don’t Panic!

“Panic-selling is a wide-scale selling of an investment which causes a sharp decline in prices. Specifically, an investor wants to get out of an investment with little regard of the price obtained. The selling activity is problematic because the investor is selling in reaction to emotion and fear, rather than evaluating the fundamentals.” (Source: Wikipedia)

I understand that many of you have a lot at stake here, but remember that Bitcoin, despite being designed to have its value increase over time, will always be the victim of people trying to abuse the system, or even the value of Bitcoin decreasing occasionally. These are not new phenomena and have been present since the beginning of time when humans first started trading.

Trade Engine Lags
Lag affects everyone, not only us, but also major, world-renowned exchanges like the NASDAQ and NYSE. We can fix lag, but we cannot eradicate lag. Only small exchanges with low volume and liquidity are immune to lag.

Does this mean that we are giving up fighting lag? Hell, no. We are working on it by creating a new trade engine that will solve many problems, but it’s not a magic bullet. We can always try to scale our servers, but we cannot predict what happens from external sources: DDoS, panic selling, immediate increase of buyers, etc. Lag will always be there, but our mission is to make lag as small as possible.

Account Verification
As if a major DDoS attack was not enough, we at Mt.Gox are victim of our own success!

Last year, Mt.Gox saw an average of 9,000 to 10,000 new accounts created every month. This number doubled in January, tripled in February, and sextupled in March. In this month alone, March, over 57,000 new accounts were created!

Our support and account verification team went from four people in January 2012 to twenty-two people working every day of the week. We are now hiring even more people to solve this problem by finalizing some deals with external companies.

Remember that even if you are waiting for your account to be verified, you can still deposit or withdraw funds via our Japanese account and make your trades! (Only accounts that we pro-actively required to be verified are limited to deposits and trade only.)

Finally
We have seen a significant amount of comments on the web (various forums, Reddit, etc.) that portray Mt.Gox as a company held by “idiots” and other rather rude words, complaining about inability to deal with lag and other system issues, without understanding the magnitude of work and attacks we are facing every day.

I understand the frustration many of you feel. We hate this situation as well. Since we took over Mt.Gox, we have been through Hell and back and we are still here. We are still the largest exchange with over 420,000 trades per month and  USD $121 million monthly trade volume. We have worked our way through all the requirements needed to run our exchange legally.

Now, there are some things we can improve, but so far we are doing an incredible job that no other exchange has been able to do so far. While I understand a certain amount of frustration, realize what we have accomplished. I appreciate all the work you are doing everyday to push things forward and to help secure the future of Bitcoin

And to all of you who are supporting us on a daily basis, thank you! We could not have done any of this without your help!