Pages:
Author

Topic: [ANN Mt.Gox] It’s been an epic few days: What happened? (Read 4039 times)

hero member
Activity: 896
Merit: 532
Former curator of The Bitcoin Museum
+1 to the private UDP broadcast idea.

Further, there has been talk off and on (by myself, and others) about a "backbone network" where Big Players privately and directly interconnect, for reasons similar to this.

And it has been actually designed (correctly) and actually implemented (correctly), by people who actually matter and actually do things (unlike you + friends). Read it, because you must read it. Stop with the posturing, you are not part of the cool kids, you're a dork "working on a soft drink distribution system" with "some friends". It's called Coca-Cola, dummy. It's listed under KO. You're embarassing yourself.

wow, what a bitch!

I'm guessing the PR in your name is there for irony?
hero member
Activity: 546
Merit: 500
Digging up a slightly old thread...

I was curious if those 57,000+ accounts created recently are legitimite.  Obviously this information would inspire the price to go higher (and I believe it did help up to $266).  I could see it as another way to attack Gox by creating a bunch of false accounts and swamp their customer service.

Just a thought..
hero member
Activity: 756
Merit: 522
+1 to the private UDP broadcast idea.

Further, there has been talk off and on (by myself, and others) about a "backbone network" where Big Players privately and directly interconnect, for reasons similar to this.

And it has been actually designed (correctly) and actually implemented (correctly), by people who actually matter and actually do things (unlike you + friends). Read it, because you must read it. Stop with the posturing, you are not part of the cool kids, you're a dork "working on a soft drink distribution system" with "some friends". It's called Coca-Cola, dummy. It's listed under KO. You're embarassing yourself.
legendary
Activity: 1120
Merit: 1152
+1 to the private UDP broadcast idea.

Further, there has been talk off and on (by myself, and others) about a "backbone network" where Big Players privately and directly interconnect, for reasons similar to this.

I'd also suggest Tor hidden services, and using private hidden service URLs that are different for each partner. The URL's don't give any information about the network topology, and if any individual URL is compromised and DoS attacked the individual URL can easily be taken down. Even URL's for individual high-volume traders would be feasible, although, keep in mind I'm no Tor expert so someone who knows more should weigh in before doing this. In particular what the Tor developers think of the attention and attacks it may attract.

In some cases using Amazon's infrastructure could work too. Amazon Simple Notification Service is essentially a DoS-resistant broadcast medium. Of course, it's central infrastructure, so not appropriate for every use, but  there is a lot of public information like price tickers that could use it. In any case information should be distributed though multiple methods.


Regardless of what is done, an important first step is to sign and timestamp all public information broadcasts so that regardless of where you got the data, you can verify it as being genuine; the current API does not appear to authenticate pricing information other than via https.
legendary
Activity: 1596
Merit: 1100
+1 to the private UDP broadcast idea.

Further, there has been talk off and on (by myself, and others) about a "backbone network" where Big Players privately and directly interconnect, for reasons similar to this.

full member
Activity: 140
Merit: 100
Mining FTW
The point is that when merely using UDP, unlike TCP, the source can block ALL incoming traffic which makes it immune to DDoS. As casascius points out, UDP is like a radio broadcast signal. TCP is like the postal service with delivery confirmation.

At what level do you propose blocking the incoming traffic?

Before it comes within miles of the host sending it.  After not informing the public who the UDP is coming from.

The UDP sending address doesn't have to be public knowledge, since not anyone can necessarily subscribe to it.  It would be a private UDP feed only offered to specific known sites.  The UDP feed would be used to drive the services of other sites who currently get it via websocket now, who in turn could provide that data to other downstream TCP websocket clients.
Also MtGox could take a position on my UDP streams idea, which could be any of the following without commitment:

a) Great idea, we haven't thought of it, and you're right, it would totally get information out immune to DDoS, we'll consider it but like anything else will take time
b) Great idea, but we don't agree it would work as well as you think it will, or for (specific technical reason) won't work on our platform
c) We haven't got a clue as to what this means
d) I don't have a clue what this means because I'm not a developer or tech guy myself, but I have relayed your suggestion to someone more technical, and he says (response).  (Hopefully this suggestion is more valuable than to merely forward it blindly like the latest facebook meme, since MtGox's reputation is suffering and this will actually solve the claimed issue at hand)

Just to be clear, using UDP to broadcast ticker data would be, for all intents and purposes, IMMUNE from DDoS attacks, because such a stream consists solely of outbound traffic which is not influenced by inbound traffic.  Unlike a normal stream, there is no inbound overhead for packets to acknowledge or to keep the connection in sync, packets which can be drowned out in a DDoS attack.  UDP is much more like a point-to-point radio broadcast: the signal gets sent from point A to B even if nobody's listening

I don't think you understand how DDoS works on this level. Your UDP stream would have to have a source, which would have to have an IP, which then would get flooded to crap. It's the routers that pop, not the machines.
Yup as working as in ITer in a DC, if they want to hit you with DDoS a lot of tricks work, till they either flood the lines or bring down the router of the DC, where you are at. Once they hit that point, there is no stopping it. And even though the real big DC's have a dozen of 10gbit lines, at some point it just stops. (A month back we saw an DDoS we saw a DDoS peak of 250gbit hit... what do you do about it... you null the IP at your carriers. (making the site unavailable))
Though when hackers/botnets are attacking in this type of volume, there is nothing that can really save you, the only way to minimize, is host global, all with loads of overcapacity, high performance routers and heavy duty DDoS firewalls. This is the only way to truly mitigate most of them. (if they attack from US, only the US DC will get hit, and EMEA and Asia will hopefully keep running smooth, same goes for both other regions when under attack)

DDoS is one of the most difficult things to combat, purely because its nature is to flood the network, hence extra capacity would be needed to combat this. (you can't buy 100gbit of network speed when doing 1gbit of traffic just because you might get hit with a DDoS up to 100gbit... and what if they DDoS you with 101gbit?) Seeing what DDoS botnetworks have been capable of, these days you are technically not safe with anything under 2tbit.... (highest I've seen was 1.3tbit) well if your in the hosting business, you know what a 10gbit uplink costs, multiply by 200, subtract 30% a month... while it would only take a botnet of half a million with 20mbit connection to push that same DDoS out. (known botnets have been found with more then 300k bots) And then I'm not even talking about some infected webserver, on a gbit connection to the internet, that can start to DDoS as part of a botnet... (only takes a 1000 of these to get a 1tbit)

Still I think Mt.Gox can do better, DDoS can't always be completely prevent, but bot lagg time in triple digit seconds as shown earlier in the thread... that should've been impossible imo. Get that security ramped up and make sure everything is running efficient. Know you get hit by DDoS on regular basis, and apply as much pre-emptive mitigation methods as possible.
legendary
Activity: 1498
Merit: 1000
Also the trade engine should be written in java or python.

Why?

I would like the trade engine to be written in Haskell, Erlang or some flavor of Lisp. Just because a functional language makes it easier to keep the code clean of unintended consequences. Also, Java is very bloaty, not a good thing for high performance code.

However, I understand sometimes compromise is necessary.

You can run java very lean if you know what your doing and not adding a hundred libraries to it. But this is one of those flame wars that will start cause you like those languages better, in any case what ever language would be better.


Also the trade engine should be written in java or python.

Why?

I would like the trade engine to be written in Haskell, Erlang or some flavor of Lisp. Just because a functional language makes it easier to keep the code clean of unintended consequences. Also, Java is very bloaty, not a good thing for high performance code.

However, I understand sometimes compromise is necessary.

A high performance trade engine should be written in C/C++, simple as that.

Again this is what language you like better, and will probably start a flame war. Also C/C++ if they don't know what they are doing can have massive memory leaks with all that data.
member
Activity: 80
Merit: 10
the attack can also happen in order to STABILIZE bitcoin. the more people are unsatisfied with mtgox, the more they will flock to other exchanges and STABILIZE the bitcoin ecosystem. we do not need one huge centralized exchange. remember this!

These are my thoughts to an extent.
Down with Walmart. *ren and stimpy stinky face*
Anarchy!  Tongue
member
Activity: 71
Merit: 10
Also the trade engine should be written in java or python.

Why?

I would like the trade engine to be written in Haskell, Erlang or some flavor of Lisp. Just because a functional language makes it easier to keep the code clean of unintended consequences. Also, Java is very bloaty, not a good thing for high performance code.

However, I understand sometimes compromise is necessary.

A high performance trade engine should be written in C/C++, simple as that.
hero member
Activity: 756
Merit: 522
Before it comes within miles of the host sending it.  After not informing the public who the UDP is coming from.

The UDP sending address doesn't have to be public knowledge, since not anyone can necessarily subscribe to it.  It would be a private UDP feed only offered to specific known sites.  The UDP feed would be used to drive the services of other sites who currently get it via websocket now, who in turn could provide that data to other downstream TCP websocket clients.

This may work, but you will have to make arrangements with the DC hosting, which is something they may or may not be able to do (from the looks of it, more like not).

Yes, UDP stands for Useless Deprecated Protocol, it's that protocol that automatically detects and cunningly removes any information and redundancy you add to transmissions to help an application using it detect and recover from losses of datagrams, thereby making it useless as an upstream data source for scripts.  It is also a protocol that is specialized in conveying data that, by virtue of having traveled via UDP, becomes impossible to republish on a TCP stream for the benefit of being consumed by scripts, to help increase the workload of script writers who of course will want to implement a UDP listener in their scripts directly.  It is commonly known as an unreliable protocol, and it gets this reputation by sneakily altering important data in such a manner where it cannot be made reliable through other methods.  In fact, it even warps the minds of people considering using it, such that they cease to even know what UDP is!

Ahaha can I have that engraved on a medallion?  Grin
hero member
Activity: 501
Merit: 500
Also the trade engine should be written in java or python.

Why?

I would like the trade engine to be written in Haskell, Erlang or some flavor of Lisp. Just because a functional language makes it easier to keep the code clean of unintended consequences. Also, Java is very bloaty, not a good thing for high performance code.

However, I understand sometimes compromise is necessary.
vip
Activity: 1386
Merit: 1140
The Casascius 1oz 10BTC Silver Round (w/ Gold B)

Do you know what UDP is? If they switched to UDP now all PHP/Python/java scripts would be useless and take now even more code to just connect. Also trading bots need reliable connections which UDP doesn't support by any means. UDP is great for the same information broadcast over and over, like said above a radio. Trading shouldn't use UDP it makes it very unreliable. They just need either a better network architecture clearly.

Yes, UDP stands for Useless Deprecated Protocol, it's that protocol that automatically detects and cunningly removes any information and redundancy you add to transmissions to help an application using it detect and recover from losses of datagrams, thereby making it useless as an upstream data source for scripts.  It is also a protocol that is specialized in conveying data that, by virtue of having traveled via UDP, becomes impossible to republish on a TCP stream for the benefit of being consumed by scripts, to help increase the workload of script writers who of course will want to implement a UDP listener in their scripts directly.  It is commonly known as an unreliable protocol, and it gets this reputation by sneakily altering important data in such a manner where it cannot be made reliable through other methods.  In fact, it even warps the minds of people considering using it, such that they cease to even know what UDP is!
legendary
Activity: 1498
Merit: 1000
UDP isn't going to solve this problem, instead it make it harder for bots to trade.

I am not sure that trading bots and DDoS are considered the same problem from the view of the consensus here.

Do you know what UDP is? If they switched to UDP now all PHP/Python/java scripts would be useless and take now even more code to just connect. Also trading bots need reliable connections which UDP doesn't support by any means. UDP is great for the same information broadcast over and over, like said above a radio. Trading shouldn't use UDP it makes it very unreliable. They just need either a better network architecture clearly.
vip
Activity: 1386
Merit: 1140
The Casascius 1oz 10BTC Silver Round (w/ Gold B)
UDP isn't going to solve this problem, instead it make it harder for bots to trade.

I am not sure that trading bots and DDoS are considered the same problem from the view of the consensus here.
legendary
Activity: 1498
Merit: 1000
The point is that when merely using UDP, unlike TCP, the source can block ALL incoming traffic which makes it immune to DDoS. As casascius points out, UDP is like a radio broadcast signal. TCP is like the postal service with delivery confirmation.

At what level do you propose blocking the incoming traffic?

Before it comes within miles of the host sending it.  After not informing the public who the UDP is coming from.

The UDP sending address doesn't have to be public knowledge, since not anyone can necessarily subscribe to it.  It would be a private UDP feed only offered to specific known sites.  The UDP feed would be used to drive the services of other sites who currently get it via websocket now, who in turn could provide that data to other downstream TCP websocket clients.

UDP isn't going to solve this problem, instead it make it harder for bots to trade. If they really wanted to solve this issue it is so simple. The trading engine should be ran completely offline, and use a database, like redis to store all information so the rest api can still have access to the information. Also the trade engine should be written in java or python.
vip
Activity: 1386
Merit: 1140
The Casascius 1oz 10BTC Silver Round (w/ Gold B)
The point is that when merely using UDP, unlike TCP, the source can block ALL incoming traffic which makes it immune to DDoS. As casascius points out, UDP is like a radio broadcast signal. TCP is like the postal service with delivery confirmation.

At what level do you propose blocking the incoming traffic?

Before it comes within miles of the host sending it.  After not informing the public who the UDP is coming from.

The UDP sending address doesn't have to be public knowledge, since not anyone can necessarily subscribe to it.  It would be a private UDP feed only offered to specific known sites.  The UDP feed would be used to drive the services of other sites who currently get it via websocket now, who in turn could provide that data to other downstream TCP websocket clients.
hero member
Activity: 756
Merit: 522
I've seen 3 Gbit/s attacks to minor sites for no apparent reason. I can't imagine what MTGox gets on a regular basis.

They say reading enriches the imagination.

Re, the UDP suggestion. That might not be a bad idea at first glance. It'd work if MTGox advertised the price from some mostly unknown IPs and out different routers, out to a list of subscribers. Another option would be to put this data in DNS, maybe in a TXT or SRV record with a TTL of 60. Then the DNS servers might be attacked, which could be a new problem.

You are rubbing sticks together trying to solve already solved problems. The many ways available for talking to MPEx.

I guess there simply would not be be anything to broadcast during the times when the engine is down or when nobody can make any trades. It would broadcast only silence. Maybe it could broadcast "Help!!! Help!!! We're under attack!".

Of course the clueless knowitalls haven't yet answered the

At what level do you propose blocking the incoming traffic?
hero member
Activity: 938
Merit: 500
https://youengine.io/
Just to be clear, using UDP to broadcast ticker data would be, for all intents and purposes, IMMUNE from DDoS attacks

I guess there simply would not be be anything to broadcast during the times when the engine is down or when nobody can make any trades. It would broadcast only silence. Maybe it could broadcast "Help!!! Help!!! We're under attack!".
sr. member
Activity: 391
Merit: 333
Thank you for the explanation and terrific post.

What can be done?
Believe it or not, there is pretty much nothing that can be done. Large companies are frequently victims of these kinds of attacks. Even though we are using one of the best companies to help us fight against these DDoS attacks, we are still being affected.

This is absolutely true. Some attacks are more application level (synflood, real HTTP requests), but others are of such a volume where the pipe is saturated. You'd pretty much have to have anycasted datacenters and massive pipes to the Internet to absorb large enough attacks. I've seen 3 Gbit/s attacks to minor sites for no apparent reason. I can't imagine what MTGox gets on a regular basis.

Re, the UDP suggestion. That might not be a bad idea at first glance. It'd work if MTGox advertised the price from some mostly unknown IPs and out different routers, out to a list of subscribers. Another option would be to put this data in DNS, maybe in a TXT or SRV record with a TTL of 60. Then the DNS servers might be attacked, which could be a new problem.

In my opinion, MTGox runs a great site. It's a bit tricky to get onto and the interface isn't as sleek as some sites, but ultimately, MTGox has single handedly encouraged a massive growth of adoption. I think ideally trades should be distributed by nature, but MTGox is still (and probably always will be) the benchmark site for Bitcoin trading, especially in bulk.

My hat is off to these guys for how thorough they are, dealing with the past through days, and 57,000 signups in one month. Those are some real challenges.
legendary
Activity: 1708
Merit: 1020
Why the lag? It's really not that many transactions. It has to be 90% the procedure / architecture of your engine.

Also +1 to UDP
Pages:
Jump to: