Topic: [1050 TH] BitMinter.com [1% PPLNS,Pays TxFees +MergedMining,Stratum,GBT,vardiff] - page 272. (Read 837101 times)

Read what I wrote again, please. This is an idea for how I would implement work generation with GBT and Stratum. It's not something I am already doing. I also said that I won't do it if you can tell me why reusing work with ntime rolled in sync with the clock is harmful.

So grab work every minute and get a 60 times speedup instead of 600. That's still pretty good. Or how often you feel is necessary to not avoid transactions.


Please go back and read what I actually wrote. I don't know how you get from "update ntime in sync with the clock" to "ntime is 1 hour out of sync with the clock".

I tell you how you can save CPU time by producing fewer merkletrees. You reply trying to teach me how merkletrees are calculated. I tell you that you can roll ntime in sync with the clock. You reply that this is bad because ntime will be out of sync with the clock.

You're just trolling, right?

... fail ...
I sincerely hope you are not ignoring network transactions as you just suggested you do ...

It's not a slow implementation on a miner ... unless you program it badly ...

On a miner it uses more CPU of an idle CPU.
.
I run 2 cgminers on a rig that the CPU sits on 1.2GHz coz it doesn't ever need to run full clock speed.
.
One of the cgminers is running a BFL Single - and that reports CPU less than 1% ... using Stratum
The other one is running 2x6950 + 2xIcarus - and that reports CPU at 2% ... using Stratum



The hack in roll-n-time has the obvious point that it makes the block ntime field next to meaningless.

... and guess what ... difficulty is calculated (incorrectly) based on that field ...

If the block used at the beginning of the difficulty calculation has ntime out by 1hr forward (and the block used at the end of the difficulty calculation has ntime correct) - it will mean the result will be 0.66% too high.
If the reverse, the result will be 0.54% too low.

If you hit close to 2hrs in the first block (and zero in the last block) it's 1.3% too high ...

(It is already 0.06% too high every time at the moment due to a bug in bitcoind - that requires a hard fork to fix)

I understand what you guys are saying. Of course I know what an extranonce is. How do you think I managed to write a pool server and miner?

I was telling you there is a fast way and a slow way to do this. And no, of course you don't have to deviate from actual time to roll ntime. How does a clock work? It doesn't stand still. It rolls forward one second per second.

You can generate work for the first second after a block change. For all the other seconds up till the next block change you just roll ntime 1 step forward and reuse all that work data.

Assuming 10 minute block changes you generate work for 1 second and reuse it for 599 seconds. You just saved 99.83% of the CPU power needed for work generation.

Since you are sending in multiple proofs of work with the exact same first sha-256 chunk it also makes midstate optimizations possible on the server side.

All this while delivering a more accurate ntime than ever before.


This information is more interesting. So you'll probably be ok whichever way you do this. I'm sure most mining rigs will also be able to deal, even if they use 600 times the necessary effort to generate work. Maybe not a raspberry pi driving several 1.5 TH/s rigs, but most will be ok.

But all that aside, when I already have rollntime working and doing what I outlined above really doesn't cost me any extra effort to implement, why shouldn't I do so? Everything else being equal, when you can choose between a fast and a slow implementation, do you really choose the slow one?

If you can tell me why this is harmful in some way, then I won't do it. But "just do like me" or "you have to use the extranonce" doesn't cut it, sorry. I also never understood why other pools don't want to roll ntime on the server side. This is why we barely noticed a 1.5 TH/s GPUMAX lease, while that would be a struggle for many pools. A holier-than-thou attitude about ntime doesn't help when your server is burning.

Just note that "harder" is a very relative term.  With 800 connected miners my Stratum pool server utilizes between 0 and 3% of 1 CPU core (I believe that server is running a E5345 2.33 GHz Quad Core Xeon).  It's multithreaded, so that 0-3% is split between 2 or 3 cores depending on how much is happening at once.  The 3% spikes come from parsing the bitcoind JSON response to getblocktemplate and preparing the full merkle tree plus sending the work to miners.  When not doing that, it's generally floating at 1%.  Less CPU than bitcoind.  Share validation is virtually invisible in terms of CPU load even with 800 miners.

Actually, no it's not smart to do.

Rolling ntime is a hack and I have said that many times on the forum before, it's just that while people are still using it, it will be able to handle the first round of ASICs
(Though, as eleuthria pointed out, there may be issues if the boitcoind design handling large roll-n-time is flawed ...)

Stratum and GBT just require smart code to not need to use roll-n-time in a miner ...

The only advantage of using roll-n-time for it would be one of:
1) You CPU mine (who on earth would do that)
2) You have difficulty writing asynchronous code and timing when the work is needed (i.e. not a smart coder)

Preparing work for the devices before they need it, is of course necessary to avoid the very small overhead of producing the coinbase txn and one side of the merkle tree (no you don't have to generate the full merkle tree with Stratum)
Though even with GBT where you do have to generate the full tree, again you simply time it to be done in advance of device requirement.

Smart coding is implementing Stratum or GBT without negatively affecting the miner (as is quite obvious how)

Yes share validation by the server is harder, but work production is many times easier.
You generate work that is effectively usable by everyone, not just a single worker - there is no merkle/hash generation work different for each connected worker.
The work generated can easily be used for all miners until new txn's appear that are chosen to be included in new work.

For the server, in getwork you must also do the same regeneration when new txn's appear, but for each worker you must also regenerate the whole merkle tree correction: one side of the merkle tree

With stratum you already roll nonce2 which has a variable number of bytes... currently it has 4 bytes which gives you a heck of lot more range than just 7200 rolls... There is no need to roll the time on top of that, and in fact the stratum spec says you can only modify the time to be consistent with the real time (i.e. increment by one every second). That still gives you 2^32 rolls per second with stratum. So 2^32 unique work units per second, each with 2^32 nonce like previously, and there is scope to actually increase the number of bits of nonce2 as well. It will be nice if the block times are more accurate as we move away from rolling the time as a hack to make more work, and stratum makes it possible.

With Stratum and GBT rolling ntime is up to you. I'll do it in my miner, because it's smart to do.

Stratum and GBT don't use roll-n-time since they don't need to.
They can technically generate unlimited work without touching the ntime field.
They should however, be setting the ntime field correctly each time they start a 2^32 nonce range check.

Both generate the coinbase transaction and thus both can add/change the coinbase field to resolve having to roll the ntime.
The obvious idea here is to have a so called 'secondary nonce' at some chosen point in the coinbase field that is incremented each time the true nonce has been checked exhaustively.
This 'secondary nonce' makes rolling ntime unnecessary due to firstly being able to roll the 'secondary nonce' a full 2^32 times and secondly, if the protocol/pool allows it, it can be larger - 64 or even 96 bits.
For a 1EH/s device (a device that could do 1 million TH/s) the 'secondary nonce' would only need 35 bits to last 120 seconds before running out of work.
With 64 bits, a 660YH/s device would last a fraction more than 2 minutes
(1YH/s is 1 trillion TH/s)

Edit: of course the correct fix for all this is to increase the size of the block header nonce ... but that's a whole 'nother discussion:
https://bitcointalksearch.org/topic/handle-much-larger-mhs-rigs-simply-increase-the-nonce-size-89278

just switched over the minirigs about 20 mins ago.. looks ok so far.

While rollntime is of course a hack it does have some nice properties. You can generate more work using the same merkletree and the server can verify more proofs of work using the same midstate. So even for a miner generating its own work (using GBT or Stratum), if it uses rollntime it can improve the performance of both client and server.

By the way, for those testing on port 9000, an easy way to see that everything works despite not all the hashrate displays on the website including the test port: look at the "shifts" display in the lower right of the livestats. If you move some hashpower from port 8332 to port 9000 and the shifts still show the same hashrate (and probably score unless the pool hashrate changes much), then everything is OK.

Only smallish miners on 9000 right now (getting diff 1 and 2). We'll need some bigger ones to test too, please.

eleuthria I do agree, however:

At the moment the forward time limit in bitcoind is 7200 seconds.
Cgminer limits it to 7000 to avoid getting end condition issues (being too close to it)

Yes in my opinion the limit shouldn't be anywhere near that - but someone came up with roll-n-time (a true hack in the clear meaning of the word) and getwork pools use that hack to their advantage.

I would be curious to know the trace of that change (to 7200) - how long it has existed and who has changed it over time ... and know who to blame for it
(git will no doubt answer all that ... but my curiosity is currently less than the effort involved )

Once pools have switched away from GetWork and are using reliable replacements (or unreliable even - but anything but getwork) maybe it would be time to reduce that limit and thus force pools to not allow it to be so large.
There really is no issue with running a reliable clock on any platform.
If people do have that problem, then they need to fix the hardware, not hope that the bitcoin network will work around it.

roll-n-time, however, on an ASIC, will of course, as you said, move closer to these end case conditions and the problems associated with them will become more likely.
However, the reality of that is that it means the implementation in bitcoind is not too well thought out ... and I guess they need to do something about it ... very soon ... though that's unlikely

kano:  Don't forget that allowing miners to roll the nTime as far as you're suggesting can be dangerous for the network.  If enough blocks are found too far in the future (to where the median timestamp over the last 11(?) blocks is in the future), valid blocks can be rejected by the network because they contain timestamps "in the past" (even if they're not).

When I was working with ArtForz on a new protocol (prior to Stratum, but basically the same design), this was something we talked about in depth.  It can happen from a variety of reasons, malicious [pools providing base times in the future], inept [pool server running a bad clock], or just a run of luck.  With the expected speed of chips coming out, and future advances, all 3 scenarios are possible.

Ah you've fallen into the trap of paying too much attention to Luke-Jr's FUD and not spotting the things he glosses over

The issue of how long you can hash on work also has an upper limit of course.
The miner should not be working on the same work for more than 2 minutes (I'd suggest even 1 minute is reasonable) since that means a clear negative effect on the whole point of mining - confirming transactions - and an increase in average transaction confirm time

If the a coin has a 5 minute block change, then allowing a miner to work for the full 5 minutes means it wont be accepting any new transactions for 5 minutes ......

Most people look down on blocks that contain no transactions since the actual reason for mining and being paid the block reward is to give miners the incentive to confirm transactions.
However, mining without including new transactions often enough, is almost as bad IMO

I wonder if there shouldn't actually be an option in bitcoind to select the preferred first target to send outgoing transactions to.
Thus people would choose pools that don't ignore as many transactions, and minimise delaying passing their transactions to their miners, as the best target to send them to first ...

So if this 1.5 TH/s rig is mining on a pool with merged mining that has block changes on average every 5 minutes, it would need 10 requests per block change to get work with getwork, but only 1 request with GBT or Stratum. But since GBT transfers more data, the 10 getwork requests may be less bandwidth than the 1 GBT request.

But with 15 TH/s the 100 getwork requests are probably less efficient than 1 GBT request.


I'm not sure this will reduce stales. I also think some miners with crappy home routers will have problems with this. Some of those routers do NAT but silently stop forwarding packets if the connection was idle for 5 minutes or in some cases even less. Maybe Stratum has a ping type of command to keep things moving, though?

Of course getwork has its problems, but I'm not sure if HTTP is one of them.


Yeah, GBT uses more bandwidth than Stratum. It also has more possibilities than Stratum, but I guess it is a moot point so far since no miners take advantage of them.

Actually - at the moment GBT is worse than GetWork

GetWork with Roll-N-Time will provide enough work for a 375GH/s rig to mine for up to 2 minutes.
GBT will provide the same but transfers a LOT more data to do it.

Once people start using multiple TH/s rigs it will "start" to be an issue, since a 1.5TH/s rig will ... only ... last for 30s on a GetWork
Yeah it will be a while before GetWork isn't usable ...

Stratum has a specific advantage - it works with a permanent connection and this results in losing less work on an LP

Stratum doesn't send hundreds of transactions with each item of work
GBT does (unless the GBT pool is ignoring hundreds of transactions)

Stratum and GBT will be better at getting work than getwork with rollntime. Rollntime was already a big improvement, Stratum and GBT will improve this a little further. But that is all.

The server already spends most of its time verifying proofs of work from miners. Unfortunately Stratum and GBT will have zero impact on this.

So we will need variable difficulty whether it is over getwork, getblocktemplate (GBT) or Stratum.

For big ASIC miners you need two things:

• More efficient work generation: rollntime (OK), GBT or Stratum (better)
• More effiicient proofs of work: higher difficulty

Next up after var diff now is GBT and Stratum. I'll probably add GBT first. I know a lot of you are using cgminer which only supports Stratum. But GBT is so much easier to add when you already have a well-tuned engine for getwork with JSON-RPC over HTTP. It makes sense to grab the low hanging fruit first.

I like this idea. Just get Stratum enabled and the big miners won't be a problem.

The problem with X-Mining-Hashrate is that you can make your computer(s) send whatever value you wish.

There's a conflict of interest here. Lower difficulty means less daily variance for the miner, which most miners prefer. But lower difficulty also means higher load on the server. If all miners could choose their own difficulty then most would choose diff 1 and the pool would need many more servers, which would be rather expensive. We are doing fine on 1 server right now with difficulty 1 for everyone. But that will no longer work once ASICs are out.

I think the HHTT pool is on to something. At that pool the lower difficulty you choose the more you must pay to the pool. That said, they do look a bit expensive and it makes no sense for small miners (choose between huge fees or huge variance).

I am considering adding a perk that gives you half the difficulty you would otherwise have. It could be a very cheap one. I also thought about a perk that would allow you to choose the difficulty freely, but then a single big miner could perhaps take down a server by choosing diff 1.

Organofcorti has some nice numbers on work difficulty and the accompanying variance: http://organofcorti.blogspot.com/2012/10/71-variable-pool-difficulty.html

I guess it comes down to; how much load can the server handle and how much variance is acceptable for the miners? Any opinions on the variance part?

And yes, you can reduce difficulty by using multiple worker accounts. And if I make change it to be per user account rather than worker account, then you can just create many user accounts. I would prefer to avoid this. So I would want the "normal usage" to give a variance that is good enough. I would also prefer for small miners to be able to mine without choosing between huge fees or huge variance.

Is there something preventing you from connecting multiple rigs to the same worker ?

Is there any way to keep using X-Mining-Hashrate after the initial go? This would allow larger miners to have 1 worker but not suffer from higher share variance for the entire farm.