Topic: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind - page 12. (Read 31408 times)

Same URL? Will try it out in a moment.


Thats actually better, BUT it would require modifications to both bitcoind and pushpoold.
This was the previous main target, but i couldn't find anyone able to implement it.

It would be really. really. awesome. Offering another 10 BTC for such an implementation.

You guys might find the comments at the bottom of this pushpool commit interesting as well: https://github.com/shanew/pushpool/commit/ef4f7261a839e628bfa24a988055a23fe442e5d2  He mentions changing from TCP sockets to unix domain sockets for the communications between pushpoold and bitcoind.

Okay, new build is up. It passes a pretty aggressive stress test, but I'll keep stressing it, just in case. Thanks for trying it.

Thanks, please do!
If we solve this permanently, I'm sending 20 BTC your way...

tcp_fin_timeout is already set to 3s, forgot to mention that in my previous post.
These settings are currently used (since a week back), and we still have the problems described before

Sorry to hear that. I'll audit my changes for the kinds of things that typically cause seg faults.

Update: Found the problem. Fixing it now.

Jine,

I've google around a little, tcp_max_tw_bucket puts a limit (default 180000) on the maximum number of sockets in TIME_WAIT, if this limit is exceeded sockets entering TIME_WAIT state are closed without waiting and a warning issued... but first you have to reach the limit.

tcp_fin_timeout, or net.ipv4.tcp_fin_timeout, on the other hand, seems to be what you need.

Lowering it (it could be 60 seconds right now) you should be able to limit  TIME_WAIT sockets to a lower number without/before reaching the bucket limit.

I mean, if you set it to 3 second, you'll end up having as many TIME_WAIT sockets as your incoming connections in a 3 seconds period, so your 25K sockets should go down to 1500-2000.

spiccioli.

I think it's possible by patching the kernel.
This will however not fix the issue completly.

I'm using tcp_max_tw_bucket (or something similar) atm - that limits TW-sockets to 10k max.
It works, but it does not solve the problem

At first sight i was like this:   
A bit later... 

Seg-faulted after a few seconds under the load. Not sure why, can't find any logs about it... :/

Jine,

maybe I'm saying something that is known to everybody here... but, I'll say anyway since I had a similar problem long ago (not bitcoin related, though).

Every socket spends some time in TIME_WAIT to correctly handle the closing of itself.

You can change the time it spends in TIME_WAIT which is set too long for a localhost only socket.

Per RFC every client waits 2*MSL (maximum segment life) before finally closing a socket, if you lower this time (default should be at least 120 seconds) to 10 seconds (for example) you should see that you have a lot less sockets waiting, and this should help you avoid reaching this 25K waiting sockets limit.

I don't know how to change MSL in Linux, though

Best regards.

spiccioli

I modified bitcoind both to implement keepalive properly and to fully multithread its RPC so that it can handle multiple RPC connections.

You can download a modified rpc.cpp here:
http://davids.webmaster.com/~davids/rpc.cpp

RPC performance is much snappier and it seems to work. But I am by no means a boost expert, so this is at your own risk. This is definitely 'quick and dirty, should clean up later' type code. Appreciation may be shown at this address: 1H3STBxuzEHZQQD4hkjVE22TWTazcZzeBw

You can try it by connecting to port 8332 with telnet: after one request has been serverd, bitcoind will close the connection. Even worse, as long as that connection is open it won't accept an other one. Everything is done on a single thread, you can see it in rpc.cpp,  at ThreadRPCServer2()

pushpoold does NOT create a new socket explicitly for each connection, nor closing the previous one.
I got a PM with a "temporary" fix for this - make it explicitly close & open new connections.

We'll see how that goes, I'll try it out later on.

I don't know if this problem is confirmed to be bitcoind-only, but all larger pools based upon bitcoind+pushpoold are suffering of it AFAIK.
I'm gladly accepting comments, suggestions or other tips on this matter.

Has this been confirmed as a bitcoind problem?  Maybe it is pushpool that is misbehaving?  What about explicitly terminating the connection?  Adding keepalive isn't really a magical fix for that because pushpoold could decide it wants to open a new connection anyway.

I am, and this is what I read out from the first post too

On the other hand let's hope that libcurl can properly reuse the connections kept open, otherwise the new problem will be the overhead of creating/destroying the threads for each connection.

Not that if my bitcoind patch worked

Lets get things straight.

It's an issue on localhost, for each and every getwork request pushpoold gets - it opens up a NEW socket against bitcoind, and issues getwork over that socket.
When it's finished and pushpoold got his work, the socket is left untouched and still opened.

We (and I know BTC Guild have) tweaked Linux a bit to reuse TIME_WAIT connections (using tcp_tw_recycle in /proc/...) which has helped - but it's not scalable.
As the load increases, the amount of new connections is overwhelming.

This was taken yesterday by me:

At that moment, we had 21.000+ TIME_WAIT sockets ready to be reused and/or closed due to timeout.
When that figure reaches 25k+ - the server starts to get hard to keep up, we're also hitting limits with num openfiles (nofiles in ulimit) which we increased from 1024 to 128.000+.

We're looking for a solution to get rid of "all" those TIME_WAIT connections (which 99.99999% comes from 127.0.0.1:8332 (bitcoind) against pushpoold at 127.0.0.01)
The best way I've come up with is implementing keep-alive support in bitcoind - hence this thread.

This will mean that bitcoind and pushpoold are only using a couple (multi threaded) keep-alive sockets for getwork and sending responses.
This will drastically lower the amount of open sockets (nofiles dropping to the bottom) and free both ports (limit of 65554 ports in TCP/IP afaik) and also make it more scalable.

Are everyone following now?

Ahh, okay. So forget the DDoS resistance stuff. But we still don't know why this is a problem.

If all the pool server needs is the 'getwork' output, it seems like a much better solution is to avoid polling altogether and have 'bitcoind' put this information in a file or write it continuously to a queue or something. (Assuming I'm understanding the problem correctly. I'm not 100% sure that I am.)

OP said that the connections are between the mining pool server and bitcoind. I suppose the rpc port of bitcoind isn't exposed to the internet, therefore can't be flooded. Also, most of the time it makes no sense to leave it open for anyone.

The problem description doesn't explain what the actual problem is. It says there are a large number of connections in TIME_WAIT state, but doesn't explain why that's a problem. I'd hate for someone to add keep alive and have it not actually solve the problem.

Since a DDoS attack can create arbitrarily large numbers of connection in the TIME_WAIT state, it's a much better solution to fix whatever is causing the large number of connections to be a problem. What is going wrong because of this? Fix that, and you'll get improved resistance to DDoS attacks for free.

I just took a look. The Boost asio library is used, so to avoid replacing everything (and introduce a different style of coding), one should probably stick with that. The whole thing is running on a single thread, and as soon as a request is answered, closes the connection. Doesn't seem too efficient, indeed. I tried starting a new thread for each request, but there is some glitch somewhere. Unfortunately I am not too familiar with boost