Topic: Thoughts and questions on BTC Pools and merged mining (Read 5624 times)

I would really interested in a JK 4diff patch for vinced mm bitcoind version in order to test the poolserverj implementation. Is there any solution out there?

Having played with pushpool and understanding the best simple optimisations to it, I'll have to have a look at poolserverJ now for my home test network.

Java - one of the best oo languages around ... In My Opinion.

Used Java on and off for many years.
Memory limitations are the only issues I've had that are in any way a true issue.
All other issues are simply just bad programmers.
Now if memory is an issue ... try compiling your Java code instead of 'bytecoding' it

My favourite argument in favour of Java is based on most people's lack of understanding of programming
Java code is expected to be faster than most other code.
Why? Simple to understand reason: optimisation.
You write a program today, compile it then release it.
Result: today's optimisation is the fastest it will get.
Java: release is the bytecode that is compiled at runtime, thus as optimisation gets better over time so does you Java bytecode's performance.

Code stability is the programmers fault and no more.
Running out of memory? Well I gotta say, if you have absolutely no idea of how much memory your program requires then you need to understand your code better ...
Debugging is probably the easiest language of all due to tracebacks being available with way more information than most common languages and easy to see/read

So he's "changing over to it" and has not actually changed over to it?  See my next comment in further regards to this:


In my case, and I'm not speaking for other pool authors, I do a lot of work on the getwork server that most other pools do on their backend.  This is mostly because it's exceptionally inefficient to try to do a scored pool on the back end.  BTC Guild is a proportional pool, so it's not really a surprise that it's substantially easier to switch a proportional pushpool over to another bit of software, since there's literally no changes *needed* in pushpool.  I know Eleuthria has done some modifications in Pushpool, but from speaking to him (granted, this was a couple months ago) his changes are not anywhere near as extensive as mine.


My java "hate" comes from countless experiences with Java based products in high capacity enterprise environments (Specifically in the wireless and banking industries).  I have yet to run across a single piece of software that is java based that is a) stable, b) not a resource hog (while still offer sub-par performance) and c) expandable.

I will admit that I have not tried anything with 64 bit Java, since most of the enterprise stuff that's running java based junk is 32 bit, but I'm not talking about 128m of memory limits.. I'm talking about high capacity applications that needs > 8 GB.  This likely doesn't apply to psj, though, so it's probably a moot point. I am just illustrating the limitations of java and why it's not a good choice for applications that are expected to grow or need stable, low latency operation.

Again, I'm not saying psj suffers from any of this, but I've yet to see a java application that doesn't suck and/or have some idiotic limitation because of the exceptionally craptactular JVM.  When a program crashes, I just love having to suss out whether it was the JVM that had a problem or the actual program itself... yay, double my debugging work!

Regardless, I'll take a look at it and see what would be required to convert my setup over to it.  I just have visions of going through that work and then finding myself (once again) up against an nigh impenetrable wall of asinine java design architecture.

Of course I'm not using poolserverj, and I like native code too, but you are a bit wrong in this part :)

It's the merkle root of all the aux chains blockheaders + number of aux chains.

Really?  Is that why the 2nd biggest pool (and the largest that uses pushpool) is changing over to it?  If you have concerns about it's capacity ask Eleuthria how many pushpool instances he replaced with a single poolserverj instance in a production stress test before it choked.


Again suggest you ask Eleuthria how long it took him to make the necessary mods.  You don't even have to rebuild the binary, it's designed with plugin architecture so you can just override the commonly customised parts (like worker auth, db interaction etc) into a separate jar...


Leaving aside the obvious java hate here (I've learned it's not worth having that argument).  poolserverj was designed from the ground up for large pools.  pushpool was not.  That's not an indictment on the pushpool author, it's simply a consequence of the fact that the network was a lot smaller when pushpool was designed than when poolserverj was.   


umm... it's ONE command line switch... -Xmx128m or whatever you want to limit the heap size to.    psj does not use memcached, it keeps all it's caches internal so of course the process is going to use more than pushpool.  It also uses the general principle of preferring performance over limiting memory footprint where there's a choice.  That's a design choice because it's meant to handle high loads.

pooserverj will only have one binary.  wether it uses merged mining or not will be configurable via properties.  There will be no additional overhead if not using merged mining.

Yes a JK 4diff for the mm version of bitcoind is needed however I don't think it's really necessary for namecoind or other aux daemons.  They do not deal with a torrent of rpc requests like the bitcoind does.  Just some aux block updates every few seconds and submission of valid shares which is also rare.

To clarify, poolserverj does not support merged mining natively yet.  It will work behind a merged-mining-proxy instance.  I've been getting my head around it all this last week and plan to make a native implementation tomorrow.  It should be ready by the changeover block unless there's a sudden upsurge in hashrate but it will be alpha.

There was no legitimate complaints, only trolls.

Which is what I said after I realised how the two chains where being mined with the same hash - and I said that coz it is the only way to actually do it - and said that coz I KNOW it is the only way to actually do that.
I didn't think back in my first post in the cgminer thread that they were putting extra data in the bitcoin block chain ... why?
COZ EVERYONE KEEPS TELLING ME THAT THEY DON'T DO THAT.
However, they ARE putting extra data in the bitcoin block chain.
Thus the web page is wrong.
Again this is all a simple understanding of how bitcoin works - nothing to do with reading how namcoin merged mining works.
If namecoin is generating bitcoin blocks, those bitcoin blocks MUST adhere to the bitcoin block chain rules.

Any additional bytes per blocks are an issue since the size of the bitcoin block chain is already an issue.
... and if every block has these extra bytes (as the namecoin people would hope that everyone does merged mining) then it is even more of an issue.
Saying it is 33 bytes doesn't help. I'll look at the code and determine what it really is.

you may be right with that (i just rechecked the thread but did not found anything regard size). do you have a link for that?

anyway: i do not think 33 additional bytes per block is a problem.

and you just confirmed that its possible to set custom data (eg namecoin blockdata) inside the bitcoin block without breaking things for the default bitcoin client.

luke-jr did increase the block size.
You didn't understand that?
That was part of the complaint about what he was doing.
He is current just putting "Eligius" in there (and stopped with the prayers) and thus increasing each block size by 3 bytes
The normal coinbase is 8 bytes.

My blocks (that say Kano) in the other 4 chains have increase each single block that I got by 6 bytes.
Yes I changed the bitcoin code for my solo mining - how on earth else do you think that is possible?
It has nothing to do with the mining software.
(and also has nothing to do with normal pool software - only the server that creates the answers for the getwork requests)

it works with default bitcoin client and the default chain (its not an alternate chain). only reason you need a bitcoin patch is if you want to mine solo (or if you are a pool op)

it's the pool owner's bitcoin who decides what to place in the coinbase.

luke-jr (owner of eligius) did a test and placed prayers in it - without a block size change; without a modified client. is this impossible too?

You CANNOT say "gets 33bytes namecoin data in his coinbase" and "without a bitcoin block size change"

Seriously? You don't see the fallacy of that statement?

Are you telling me that the real bitcoin block chain is going to reject all merged mining bitcoin blocks and only namecoin will accept the namecoin blocks?

If you modify anything in the block (including the coinbase) after calculating the hash, but before putting the block into the real bitcoin block chain, the block will be rejected.

I know this not only by reading but by fact, due to having modified most of the different alternate coin software myself and adding a block into each block chain that says "Kano": I0coin block 29930, SolidCoin V1 block 33533, GeistG block 3676, IXcoin block 21128.
Did it for fun by reading the code and working out how the coinbase works, but more for the sake of understanding how things work.

You see, it has to work with the real bitcoin block chain - doesn't matter if you run a hacked bitcoin, the real bitcoin must accept the bitcoin blocks.

Thus your statement must be wrong (and again, the front page of that site must be wrong)

And Yes I will be reading the code when I have time to go through it (as I said) since no one knows or at least wants to admit what the code actually does (or makes statements that cannot be true)

please do that before posting about merged mining again.

the bitcoin block gets 33bytes namecoin data in his coinbase (without a bitcoin block size change).

there are several people who did run this on testnet (and you can do this right now).

To put this discussion in the right place ...

I didn't link to that document, I referred to where I could gain the source.
That document is wrong.

As I said you completely missed what "create hash" is.
"create hash" is hashing the contents of the block that goes into the block chain and then checking if it is under the difficulty requirement.
If both are true, you then have a block.

Since you are hashing the contents of the block, you cannot even compare that to some other different block.
It's not possible.

What is possible, is to add extra data to the block, that some other block chain can check against and decide it will allow it to be a block.

The hash of the bitcoin block hashes the following information:
version
previous_hash
time
difficulty
nonce
the miner's extranonce
the transactions in the block (this includes the generation block which has the coinbase)

i.e. the whole block

The above information is mandatory for the block chain to be secure.
It ensures that no one can change the block chain somewhere in the middle.

Thus the bitcoin hash is the hash of the complete bitcoin block contents.
It is not the hash of ANYTHING more or less.

For a hash for namecoin to be securely usable in both bitcoin and namecoin, the bitcoin block would have to contain something extra added that namecoin can use.
If this were not true, then namecoin would have to rely completely on the contents of the bitcoin block chain to secure itself i.e. namecoin would become dependent on having a copy of the whole bitcoin block chain to be able work.

P.S. I still haven't read the namecoin code yet

For the big pools, poolserverj is not really an option.  We (not that I'm including myself in the big pools) have our getwork servers, either custom or pushpool highly modified to conform to our database and methods.  Switching over to poolserverj would be a major undertaking and rewrite of the poolserverj software.

Not something that couldn't be done, but it's not something that's going to happen very quickly.  It needs to work with current implementations to gain traction, not foisted off on yet another getwork server.  I don't relish the idea of :

a) Running ANYTHING java based, having to use Java based junk in the enterprise, I can tell you I have never once seen a java based program that was worth anything in high availability/heavy load environments.  Couple that with the fact that without some severe tweaks, the memory limits of the java VM leave a LOT to be desired.  I'll stick with native code, thanks.  That said, it's usually the interface that is complete FAIL with java crap, so maybe Poolserverj doesn't suffer from this, since there's no need for an interface to speak of.  But there are so many other problems with java scalability wise that it's just too unstable and problematic to build a business off of.
b) Rewriting yet MORE code to conform to my standards

Agreed.

Although, I would run the pool on a separate machine until I was 1000% sure it wouldn't interfere with the normal pool.

More important than any of that is DOCUMENTATION. On the protocol, not the patches assuming I want to risk the main pool operation.

As this nears completion, it would be benificial for pool operators if there was:
A bitcoind diff for the merged client (since most of us run custom clients)
A merged-mining poolserverj binary
An easily modifiable namecoind source to apply joel catz fixes (or a premodified one).

Yes


its standard namecoind


have a look at PoolServJ which replaces pushpool, lp and proxy

btw.
thats the reason i've started this thread. i want all pool owners to know how merged mining works - so its up to them to get it to work. and i hope some miners do read this and will prefer merged mining pools (like i do)

All the wiki is/has is a lot of description about what Merged Mining is, what test pools there are and other useless (as far as implementation goes) information.  There's six sentences about where to find the source code for namecoind and bitcoind from a repository with little to no documentation on what you need to implement, where and/or how.  

The wiki is almost completely useless for a pool opeartor, with the exception of links to the repository, in relation to getting merged mining going.  Hence why I said you have to comb through lots of posts and other useless information to even begin to get an idea of what exactly is needing to be done.  From combing through posts, I came to the conclusion that it's unstable, untested and not anywhere near ready for prime time.  Given that, I've not bothered to try to decipher exactly what needs to be done with the files in the repository.

Do I need to recompile bitcoind?  Looks like probably yes.
Do I need to compile namecoind?  Is it a standard namecoind?  Yes and yes?  I dunno
The proxy script, what needs to be done with it, other than to make it executable and run it in place of bitcoind and namecoind?  
Can the proxy script even handle a high load or is it going to fail miserably when you dump 100 or 200 GHs at it?  

There's lots and lots of other implementation questions that are either nowhere to be found or buried deep within threads that have so much noise that it requires lots of time and dedication to wade through.

Quite correct, the thing is that usually you submit solutions for shares that meet "difficulty 1". Some of these also fit to a higher difficulty and every ~1.7 millionth one also fits the current bitcoin difficulty, so it gets used to solve a block.

The trick now is that you put all namecoin and bitcoin transactions together (it is structured as a tree, so all namecoin transactions just take up a single hash value in the end in bitcoin blocks) so a solution that would fit namecoin difficulty (much lower than bitcoin atm) can solve a namecoin block too.

Thanks,
Sam

merged mining means that while you are mining bitcoins you ALSO mine namecoins (or other chains in the future) with the same hashrate.

means: if you have 1GH you can mine namecoins with 1GH AND bitcoin with 1GH at the same time.

if you dont mine solo you dont need a modified bitcoin for this: only pools need a modified bitcoind which writes 33bytes of namecoin data in the coinbase of bitcoin blocks (the blocks size does NOT differ). you do need a new namecoin which accepts "dual-mined"-blocks (the official namecoin daemon will accept merged-mining-blocks after namecoin block 19200)

^^ please correct me if i am wrong.

Would someone mind defining the term "merged mining"?
Sam

The PPS pool will absolutely be the first testing server I use for merged mining since I don't have to do any special accounting for NMC rounds vs BTC rounds.  I'll have to talk to shadders about it.

Has anybody tried applying the JoelKatz 4diff patch or integrating it into the merged mining bitcoind?  If they aren't compatible, it's going to be a major handicap getting a reasonably sized pool on board.

as far as i know PoolServJ is working on merged mining (thanks davinci!!!). bt- guild-pps seems to use it.. wouldn't that be a good test bed?

You could do it hidden, just convert the NMC to BTC and pay these out as bonus or make some promo stuff or so - there shouldn't be any difference for miners anyways if they mine merged or not.

I've been looking into merged mining, but right now I don't see it integrating itself into the main BTC Guild pool anytime soon.  It will be a bookkeeping nightmare unless I hide the NMC part and do it as a side calculation whenever a round ends.  It's difficulty to implement in a production environment, especially one where the database is not easy to make modifications to without slowing/halting the live servers.

I may put it onto the PPS pool for BTC Guild, since the code and DB schema is much more flexible on the PPS pool [no need to keep track of 'rounds'].

I fully intend to support merged mining as an option.

My concerns:

Need to be able to run alongside standard btc server, to give the option to pool users.
New Namecoin client needs to support multi-threading & long-polling implementations.
None of the implementation can affect my other coin clients or bring the pool server to a grinding halt.

Ill parrot part what some of the other ops have said as well.  Mainframe is looking into this but not jumping in with both feet for the following reasons:

1) unsure of demand for this from miners
2) same reasons Inaba mentions (lack of coherent and centralized information)
3) same reasons Luke-Jr mentions.  Im also not willing to jeopardize a stable BTC pool with unproven merged mining code (unproven in a live full scale production environment)

If some of these issues are addressed im happy to look at implementing it as well.

Could you please tell me which information about merged mining you are missing? I really did my best to gather all information on dot-bit Wiki (http://dot-bit.org/Merged_Mining). IMHO merged mining is not an arcane art. Just patch bitcoind, install latest namecoind and connect them with merged mine proxy. Point an existing pushpoold to merged mine proxy and you are done. For a pool you have to adapt your frontend and backend scripts of course, but this has to be done by each pool operator. Perhaps I became blinkered in my work. Thus please let me know which information is needed, I'll do my best to add it to the Wiki. Furthermore I'm going to review it in order to reflect the latest status of the project.

There is a major lack of coherent communication in regards to merged mining.  I'm willing to implement it if there were some central location that information were able to be found... but so far you have to comb through various posts and forums that are disjointed and inaccurate.

But as of right now, it seems that the only information that I have found is that it's unstable, untested and almost unusable without a lot of trial and error.

From what I can tell. Merged mining indeed is working. https://alpha.masterpool.eu mined several blocks on testnet. There were some perfomance issues. However I didn't have the time to test vinceds last patch (https://github.com/vinced/namecoin/commit/3c704cd6abdb0f836ea4ea9e5323a7875af10cc9#contrib/merged-mine-proxy). It seems he fixed the problem why we found way too less blocks on AUX chain. Thus before I tell something wrong I'd like to test the new version. For that reason I recommissioned http://alpha.masterpool.eu. The poor server is a bit messed up because of its hibernation (some stats are missing others like contributors are running in "history" mode) but mining works and you should see the generated blocks being verified at: https://alpha.masterpool.eu/statistics under Unconfirmed Blocks

Please give me the time to mine a few blocks with the new version before I say anything wrong  

Furthermore there are already other pools implementing merged mining (including big BTC ones) but IMHO its up to them to announce it. From what I know officially DavinciJ15 from http://nmcbit.com is trying to implement merged mining right now. And the guys from p2pool announced they'll integrate merged mining as well IIRC.

Edit:

Yes, I'm willing to help every person who is able to run a pool on his own and has the technical knowledge about how merged mining should work. But please I won't waste my time on giving "Mining pool for Dummies"-Workshops. My family already forgot about my face because of being on NMC projects all day  Furthermore If you have questions regarding merged mining I'd like to kindly ask to join on http://dot-bit.org forum. I know, it is PITA to switch bulletin boards, and I don't like it as well. But most knowledge about merged mining is concentrated on NMC forum. It seems on BTC forum I'm regarded as the ultimate merged mining guy. But that's not true. There are other much smarter people over on NMC forum  

nmcbit statement:
https://bitcointalksearch.org/topic/m.532815

it just seems that merged mining is a closed thing at the moment....so i suggest you all to move to masterpool right now...

nodemaster (if you read this): would you mind to help other pools with this? i am pretty sure they will honor it.

The test stuff never worked for me at all. It rejected all the blocks. I'm not willing to sacrifice Bitcoin mining stability, so I won't be using their proxy or bitcoind modifications. I've tested my own implementation of the tie-ins over the past few months (that's what the prayers were testing).

abcpool said they will consider it
but i am not sure if they know math

deepbit is out of the game (at least for now)...

eligius would be interesting

i am just curious: as far as i see they did a test on testnet and have one pool ready. i thought they have released their proxy?

what is slow? does the proxy not deliever work fast enough for a pool with your size or i am wrong with my thought that they've released their stuff?

or: the release was too late to get your custom patches in namecoin (which would be understandable - dont wont to offend you)?

Eligius has planned to support merged mining for a while, but actual implementation has been extremely slow due to lack of anything remotely usable from the Namecoin side.

Hi @all,

atm there are only 197 more blocks needed to start merged mining.

are there any pools (except masterpool) which will take the challange?
did you (as a pool op) at least run a test? how is it working for you?

please tell us your thoughts!

btw. i am looking for a nice *PPS pool which offers Merged Minging (if you know one just tell)

i do have the strong impression that every btc pool which do not support merged mining will:
 - loose miners as they can get more anywhere else
 - would be (verbal) attacked by people that thinks the pool is stealing the nmc's

brw masterpool and cgminer gave me < 0.02% stales (after 10k shares). but i dont want to use a proportional pool.