Topic: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind - page 10. (Read 31438 times)

That code expects that if it asks for HTTP 1.0 behavior, it will get correct HTTP 1.0 behavior. How unreasonable! My latest fix should do it.

Okay, all builds have now been updated. (rpc.diff.txt, bitcoin-3diff.txt, rpc.cpp)

It now does keepalives correct if the client specified HTTP/1.1 and didn't ask for the connection to be closed or the client specified HTTP/1.0 and asked for the connection to be kept alive. If this doesn't fix your connection hanging issue, make sure you are either sending "HTTP/1.0" or "Connection: close".

My mate MIGHT have a solution -- he's dealt with having to have millions (literally) of connections to single servers/clusters before in his programming. He might of already built an API to deal with this.

I'm using the standard php bitcoin interface from Mike Gogulski: http://www.nostate.com/3962/bitcoin-php-a-php-library-for-bitcoin/

It could be my bug or it could be your bug. I think it's my bug, and it only affects clients that either send HTTP/1.0 requests or send 'connection: close' headers. I'll fix it.

Having problems doing rpc calls from php now. any page using them just stalls out.... I'm guessing it's waiting on a connection close (that doesn't exist)...

Just sent the first of possibly many donations.  Great efforts so far, and I'll be testing these on one of my servers tonight.

A preliminary combined diff is now available.
http://davids.webmaster.com/~davids/bitcoin-3diff.txt

Features:

1) RPC enhancements to support JSON parsing in a thread, concurrent handling of multiple RPC connections, and working HTTP keepalives.

2) Native long-polling support. New option "-pollpidfile=/path/to/pidfile". Pass in the path to your pool manager's PID file. (By default with pushpoold, it's "/var/run/pushpoold.pid".) This is faster than blkmond and eliminates the need to run it. (In fact, DO NOT use blkmond with this option as you'll wind up telling your miners to abort valid runs.)

3) More aggressive connection management. Outbound connections are made in their own thread and the client tries more aggressively to get more of them. This should mean both that you find out about new blocks faster and you get your own blocks out faster. This is most noticeable when you just restart bitcoind, as it tries very aggressively on startup to re-establish connectivity. But the benefits should accrue all the time -- you'll be working on a longer chain faster and others will be working on extending your blocks faster.

The changes to 'net.cpp' should not be used on machines with limited net connectivity as more connections will mean more traffic. Currently, these changes are not very tunable. (I plan to add a '-hub' option that will enable these features to make you more of a 'hub' of the BitCoin network.)

Test reports, suggestions, and donations are all welcome.

hmm, I was... tried with -p0, retrying.

Update: copied to src/src and renamed to rpc.diff. patch < diff worked fine. Thanks!

compiling now.

I made it against .23, and I just checked it against a clean .23 and it applied fine. Make sure you're in the src/src directory. "patch < rpc.diff" should work.

running the diff against .23 I get 10 out of 10 hunks failed.....

does this only work against .22?

Donations will go to prioritizing my next BitCoin-related project, optimizing bitcoind for pooled mining in a few other ways.

First, native long-polling support in bitcoind, eliminating the need for blkmond and speeding up switching to a new block. You'll be able to pass a pidfile to bitcoind, and it will automatically send a SIGUSR1 to the pool manager when a new block is found.

Second, aggressively acquiring and maintaining more connections. The current client actually goes out of its way to keep fewer connections. But if you have more connections, you win two ways. First, you find out about new blocks faster, meaning you won't waste time working on a shorter chain. Second, your found blocks will get out the fastest, increasing the odds that your block will stay in the chain.

Third, optimizing the code paths that miners hit heavily. This includes, for example, more rapidly getting a found block out to all connected clients. (This actually may already be close to optimal. I'm not sure there's much room for improvement here, but it won't hurt to look.)

Checking in after getting linked back here this morning.  Is the patch now working in a high volume production environment?  If so I'll have to throw some BTC at JoelKatz as well.

Thanks. It is much appreciated.

Sent you a little something Joel for your efforts

Even better!

Or modify it back to 32-65k and apply your patch

Ahh, you need to blacklist critical ports then!

See starting from line 604 here:
http://www.mjmwired.net/kernel/Documentation/networking/ip-sysctl.txt

We have tweaked ip_local_port_range


EDIT: This to allow more sockets to be open at once, due to the amount of connections we're getting

By default, temporary ports are allocated from 32768 - 61000, so 8332 shouldn't ever conflict (unless you changed that to increase the range). This is controlled by /proc/sys/net/ipv4/ip_local_port_range.

If you increased the range to try to prevent running out of ports, you can blacklist 8332. Use
/proc/sys/net/ipv4/ip_local_reserved_ports.

It may help to add a 'brake' to pushpoold to stop it from doing whatever it's doing that's causing the issue. (Perhaps if a connection to bitcoind fails, do not attempt to reconnect for two seconds. That's pretty easy to hack in.) But I doubt it's that it's using a needed port.