Pages:
Author

Topic: [20 BTC] Multithreaded Keep-alive Implementation in Bitcoind - page 6. (Read 31453 times)

legendary
Activity: 1260
Merit: 1000
Locked up again...

Somebadger: Your git is definitely not stable, sadly.  I'm trying now without any -hub command line, we'll see what happens there.

legendary
Activity: 1260
Merit: 1000
I'm also seeing this in the console log now and again:

[2011-07-14 23:53:5.926549] JSON-RPC call failed: (null)
[2011-07-14 23:53:5.926566] submit_work json_rpc_call failed
legendary
Activity: 1260
Merit: 1000
Is anyone else having problems with somebadgers git pull of this?

I had a bunch of socket recv error 104 errors in the debug.log and bitcoind was not serving any work this morning.
legendary
Activity: 1596
Merit: 1012
Democracy is vulnerable to a 51% attack.
Something else occurred to me: If we make these changes, we can allow the miner to decide how much he wants to pool mine and how much he wants to solo mine. Instead of defining each share submitted as 1, we can define each share submitted as the fraction of the revenue for the block sent to the pool's account. So a miner could split the money 50/50 and get half shares from the pool but make 25BTC if he mines the block himself, or anywhere in between.

In fact, he could even mine for more than one pool at a time, accumulating fractional shares from each pool.
full member
Activity: 171
Merit: 127

I'm not sure how the proof of work flow would go though. If the pool only gets the address and the branch, it cannot submit the block to the public chain. Are you expecting the miner to do that alone? Or are you suggesting different paths for solved block versus share found?

This will force pool managers to rely on the miners to work on useful branches, and it will mean that if a miner has a poor connection, he may be submitting useless shares for which he will get full credit.

This is true even now - miners can fail to submit the precious one true solution due to many reasons. Even on purpose. And yes, I expect the miner to submit his blocks.

Quote
... And avoiding recomputing the entire merkle tree on each 'getwork' call will eliminate the biggest CPU sucker left in 'getwork' after my changes.

Feel free to pull this in your code. My patch is tested against pushpoold and seems to work fine.
legendary
Activity: 1596
Merit: 1012
Democracy is vulnerable to a 51% attack.
Also, pools should be changed to allow miners to just prove they included pool's coin base in the block they solve. This is possible by sending the transaction with pool's address in it and the next Merkle branch. Then miners will have complete control over which transactions to include and which block chain to build on.
This is a great idea, however, it requires changes to *everything*. Bitcoin will need changes, pool managers will need changes, and mining programs will need changes.

I'm not sure how the proof of work flow would go though. If the pool only gets the address and the branch, it cannot submit the block to the public chain. Are you expecting the miner to do that alone? Or are you suggesting different paths for solved block versus share found?

This will force pool managers to rely on the miners to work on useful branches, and it will mean that if a miner has a poor connection, he may be submitting useless shares for which he will get full credit.

I think this needs some subtle changes to address those issues. But I think the idea of letting the miners decide which transactions to include is a great one. And avoiding recomputing the entire merkle tree on each 'getwork' call will eliminate the biggest CPU sucker left in 'getwork' after my changes.
full member
Activity: 171
Merit: 127
Even if too late in this thread, I'd like to make some comments. Partly because I feel some guilt over the notorious 'getwork' RPC call.

It was made with only one purpose - to experiment with mining outside of Satoshi's code. I never imagined it will feed heavy loaded servers in the way it is doing it now. Sadly, it made pooled mining possible and at the same time allowed de-democratization of the mining process. I believe this can be fixed.

Right now, classic 'getwork' is re-processing all transactions whenever there are new ones and 60 seconds have passed or when there is new block(1). Worse, because each worker needs its own hash space, the Merkle tree is recalculated entirely with each request(2). When the size of transaction pool (unconfirmed transactions) gets really large, this becomes unfeasible. There were episodes with significant numbers of spam transactions which proved this.

Some months ago I made https://github.com/m0mchil/bitcoin/tree/poolmode

About (1), transaction processing was moved out of the RPC thread (to main.cpp, ProcessTransactions) to make 'getwork' always return as fast as possible. For (2) UpdateMerkleTree was introduced to allow only specific branch to be rebuilt (specifically the first one). Bitcoind was creating new thread handle for each connection accepted (I guess it was needed for connection timeout guard) - this was removed (see rpc.cpp, around 'boost::thread api_caller') because with pools the server is always used locally, by trusted process(es). Even still single threaded, 'getwork' performance improved drastically.

But it is time for a new scheme. I see Gavin's monitorX patch as a good candidate. We need something like 'monitorTransactionPool' to push whenever there is change in the set of transactions currently ready to be included in a block.

Also, pools should be changed to allow miners to just prove they included pool's coin base in the block they solve. This is possible by sending the transaction with pool's address in it and the next Merkle branch. Then miners will have complete control over which transactions to include and which block chain to build on.

I am intending to have this implemented soon.
hero member
Activity: 546
Merit: 500
Looks good here too, cheers Smiley
member
Activity: 170
Merit: 10

What build is this exactly?


It's a git clone of bitcoin4pools as posted a while back:
https://github.com/somebadger/bitcoin4pools

As of commit https://github.com/somebadger/bitcoin4pools/commit/fbfd3ecbd322edd2c478135fb43e63b8ad4edeab

Which I notice happens to mention CReserveKey, which is also in the backtrace..

I have updated the repo, reverted back to .23 with patches and updates, plus --help additions.
.24 had some issues will get back on those once i get some time.
https://github.com/somebadger/bitcoin4pools/commits/v0.3.23-pool

ps. running now for 22hrs without a glitch Smiley
full member
Activity: 302
Merit: 100
Presale is live!
Is it any other way to do it? Possibly by not requesting work at all? Generating it in pushpoold perhaps?
I don't have any other ideas.

Both me and Eleuthria is talking about suicide on IRC (not serious tho... yet ;P) because of all the issues.. (Not only bitcoind-related, but server/ddos/stability/stupid users etc)

That's an idea i've had myself and i'm trying to implement it in my own backend however i still haven't quit understood what exactly the data is calculated from (besides that the previous block is involved somehow) but once i get it, it should be pretty trival to implement considering the code for handling the block chain is already ported to C#: http://code.google.com/p/bitcoinsharp/

So if anyone got a simple explenation or links that explains the getwork process please share Smiley
hero member
Activity: 546
Merit: 500

What build is this exactly?


It's a git clone of bitcoin4pools as posted a while back:
https://github.com/somebadger/bitcoin4pools

As of commit https://github.com/somebadger/bitcoin4pools/commit/fbfd3ecbd322edd2c478135fb43e63b8ad4edeab

Which I notice happens to mention CReserveKey, which is also in the backtrace..
legendary
Activity: 1596
Merit: 1012
Democracy is vulnerable to a 51% attack.
I've been there with the DDoS attacks, I know how awful that is.

Giel de Nijs suggested a binary protocol between bitcoind and pushpool. This would eliminate all the HTTP/JSON/hex/base64 nonsense entirely. It would also permit pushpool to request work units in blocks of, say, 10, reducing the handoffs between the two programs. This would mean long polling could be implemented in a saner way than a signal as well.

On the bright side, my current secret project will make all mining pools obsolete anyway and drastically reduce the feasibility of the 51% attack so there's some hope. Wink
sr. member
Activity: 403
Merit: 250
Is it any other way to do it? Possibly by not requesting work at all? Generating it in pushpoold perhaps?
I don't have any other ideas.

Both me and Eleuthria is talking about suicide on IRC (not serious tho... yet ;P) because of all the issues.. (Not only bitcoind-related, but server/ddos/stability/stupid users etc)
legendary
Activity: 1596
Merit: 1012
Democracy is vulnerable to a 51% attack.
Getting the locking right on this patch (the one to precompute the skeleton block) is proving to be a pain.
sr. member
Activity: 403
Merit: 250
... and now it froze.

Reverted back to old bitcoind... Sad

EDIT:
Quote
received: addr (31 bytes)
received: inv (37 bytes)
  got inventory: tx f9e2f6b050644cc94341  have
sending: addr (31 bytes)
received: addr (31 bytes)
sending: inv (37 bytes)
sending: addr (31 bytes)
sending: inv (37 bytes)
sending: inv (37 bytes)
received: addr (31 bytes)
received: addr (61 bytes)
sending: addr (31 bytes)
sending: inv (37 bytes)
sending: inv (37 bytes)
BitcoinMiner:
proof-of-work found
  hash: 00000000000004f00d620b3a2fd5e4ba19043e2053e9f4060a3795d2d00c5155
target: 0000000000000abbcf0000000000000000000000000000000000000000000000
CBlock(hash=00000000000004f00d62, ver=1, hashPrevBlock=0000000000000a6772c7, hashMerkleRoot=7e05d00288, nTime=1310055675, nBits=1a0abbcf, nNonce=568783248, vtx=37)
  CTransaction(hash=14b69a7cab, ver=1, vin.size=1, vout.size=1, nLockTime=0)
    CTxIn(COutPoint(0000000000, -1), coinbase 04cfbb0a1a035fc601)
    CTxOut(nValue=50.07110000, scriptPubKey=044dcd4ee79a23daccd12df3450a23)
  CTransaction(hash=da5496d248, ver=1, vin.size=16, vout.size=2, nLockTime=0)
    CTxIn(COutPoint(9c55619ba7, 1), scriptSig=3045022100f6c19dcd06577d)
    CTxIn(COutPoint(8d7f3d9c1d, 1), scriptSig=304602210095f7f99816769e)
    CTxIn(COutPoint(3255f7943d, 0), scriptSig=304602210089bbaa23e5be3c)
    CTxIn(COutPoint(a8be29ad98, 1), scriptSig=3045022100bab4ddaaf7da9c)
    CTxIn(COutPoint(abdc5e1a70, 1), scriptSig=30450220442f40e8b65ef407)

... new block?

    CTxOut(nValue=0.01000000, scriptPubKey=OP_DUP OP_HASH160 91d45c1f79fa)
  CTransaction(hash=a51043e11b, ver=1, vin.size=2, vout.size=2, nLockTime=0)
    CTxIn(COutPoint(6701c1b7a5, 0), scriptSig=30450220224680e1db9d648d)
    CTxIn(COutPoint(60cc6580e0, 0), scriptSig=304402207cb98fb1870604cd)
    CTxOut(nValue=0.00775252, scriptPubKey=OP_DUP OP_HASH160 c1e8b98c8c3d)
    CTxOut(nValue=0.02000000, scriptPubKey=OP_DUP OP_HASH160 401ce258491c)
  vMerkleTree: 14b69a7cab da5496d248 035078318f ea17127588 c0c344418b eef1e852f6 649507f180 ac905c95d4 0842b073a4 305372759d 9a6e764565 6794e2e09c d76296138f fb2fbfb31b 827e418d7f b142831de3 cd499e996d 5b43802900 68971974ef 29a1b4054b 1ca1ee01f1 f7abc8ab40 9f5768fc54 47394$
07/07/11 16:21 generated 50.0711
keypool keep 10145
AddToWallet 14b69a7cab  new
AddToWallet 68971974ef  update
ThreadRPCServer method=getinfo
ThreadRPCServer method=getinfo
ThreadRPCServer method=getinfo
ThreadRPCServer method=getinfo
accepted connection 178.47.108.93:60078
ThreadRPCServer method=getinfo
ThreadRPCServer method=getinfo
ThreadRPCServer method=getinfo
ThreadRPCServer method=getinfo
ThreadRPCServer method=getinfo

* was unresponsive, so i runned ./bitcoind stop*


ThreadRPCServer method=stop
DBFlush(false)
addr.dat refcount=0
addr.dat flush
ThreadSocketHandler exiting
ThreadOpenConnections exiting
ThreadRPCServer exiting
blkindex.dat refcount=1
wallet.dat refcount=25
StopNode()







* restarted *





Bitcoin version 0.3.23-beta
Default data directory /root/.bitcoin
sr. member
Activity: 403
Merit: 250

I have some updates available at:
http://davids.webmaster.com/~davids/updates.diff.txt

This includes a performance improvement for ToHex, some style cleanups, some tuning changes to hub mode to reduce outbound connections, and a huge change to 'getwork'.

The change to 'getwork' is one I attempted before with not so good results, but I've rethought the way it's implemented, and this is much simpler. The idea is to update the 'skeleton block' used to build 'getwork' requests when a new transaction or block is received rather than waiting for the next 'getwork' request to come along to do it. The reason this makes a difference is that it means the 'getwork' request won't have to wait for the code handling the new block/transaction to release the 'cs_main' lock. Testing shows this really smooths out the 'spikes' in the 'getwork' response time.

However, this is a risky change. It is invasive to not just the code to get a work unit but also the code to confirm a valid block. Some people may wish to apply just the ToHex change and the cleanups.



Quote
jine@bitcoins:~$ uptime
 18:20:27 up 9 days,  6:45,  3 users,  load average: 0.00, 0.19, 0.22

I say no more.

We had some serious issues a while ago with a lot of disconnects, so i took a shot and applied this....
0.00 load.. seriously? Tongue Cheesy
sr. member
Activity: 403
Merit: 250
wallet.cpp is in the github version - no the stable one.
legendary
Activity: 1596
Merit: 1012
Democracy is vulnerable to a 51% attack.

What build is this exactly?
hero member
Activity: 546
Merit: 500
JoelKatz,

Managing to get segfaults when I send getwork to bitcoind with this set of patches now, backtrace below:

Code:
(gdb) bt
#0  0x00007f710a6e8394 in pthread_mutex_lock ()
   from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x0000000000447103 in boost::interprocess::interprocess_recursive_mutex::lock (this=)
    at /usr/include/boost/interprocess/sync/posix/interprocess_recursive_mutex.hpp:53
#2  0x0000000000477d98 in Enter (this=0x0, nIndex=@0x9c1968, keypool=...)
    at util.h:245
#3  CCriticalBlock (this=0x0, nIndex=@0x9c1968, keypool=...) at util.h:260
#4  CWallet::ReserveKeyFromKeyPool (this=0x0, nIndex=@0x9c1968, keypool=...)
    at wallet.cpp:1041
#5  0x000000000047889c in CReserveKey::GetReservedKey (this=0x9c1960)
    at wallet.cpp:1117
#6  0x0000000000458562 in CreateNewBlock (reservekey=...) at main.cpp:2713
#7  0x000000000048f630 in GetWorkBlock (
    pmidstate=0x7f70feff9ef0 "С\377\376p\177", pdata=0x7f70feff9e30 "#",
    phash1=0x7f70feff9eb0 "\250\223\f\003", hashTarget=...) at rpc.cpp:1359
#8  0x000000000048fca2 in FastGetWork (id=...) at rpc.cpp:1461
#9  0x000000000049d03b in ThreadRPCServer3 (parg=0x7f71002503d0)
    at rpc.cpp:2031
#10 0x000000000049d328 in ThreadRPCServer3 (parg=0x7f71002503d0)
    at rpc.cpp:1981
#11 0x000000000049d328 in ThreadRPCServer3 (parg=0x7f71002503d0)
---Type to continue, or q to quit---
    at rpc.cpp:1981
#12 0x000000000049d328 in ThreadRPCServer3 (parg=0x7f71002503d0)
    at rpc.cpp:1981
#13 0x000000000049d328 in ThreadRPCServer3 (parg=0x7f71002503d0)
    at rpc.cpp:1981
#14 0x000000000049d328 in ThreadRPCServer3 (parg=0x7f71002503d0)
    at rpc.cpp:1981
#15 0x000000000049d328 in ThreadRPCServer3 (parg=0x7f71002503d0)
    at rpc.cpp:1981
#16 0x000000000049d328 in ThreadRPCServer3 (parg=0x7f71002503d0)
    at rpc.cpp:1981
#17 0x000000000049d328 in ThreadRPCServer3 (parg=0x7f71002503d0)
    at rpc.cpp:1981
#18 0x000000000049d328 in ThreadRPCServer3 (parg=0x7f71002503d0)
    at rpc.cpp:1981
#19 0x000000000049d328 in ThreadRPCServer3 (parg=0x7f71002503d0)
    at rpc.cpp:1981
#20 0x00007f710a6e5d8c in start_thread ()
   from /lib/x86_64-linux-gnu/libpthread.so.0
#21 0x00007f7109c9004d in clone () from /lib/x86_64-linux-gnu/libc.so.6
#22 0x0000000000000000 in ?? ()
(gdb)

This happening both with a fresh blockchain downloaded and a new wallet.

Let me know what I can give you to help debug Smiley
newbie
Activity: 42
Merit: 0
fyi: renamed the repository to bitcoin4pools, new url is:

https://github.com/somebadger/bitcoin4pools

Any addition suggestions are welcome, will start a new thread later today and reference this thread and give credit to JoelKatz Smiley

Thanks for that!

Please make a post of said new through in this post, as I am watching this post for some time now as I assume many other pool creators are.
Pages:
Jump to: