Topic: [OLD] Eligius: ASIC, no registration, no fee CPPSRB BTC + 105% PPS NMC, 877 # - page 108. (Read 458255 times)

My stats have not been updating the last few hours. A couple of blocks have been found and no change. I see post like this and it almost always works  itself out so I'm not that worried about it but I thought I should point it out.

the round of death 

Sent 1 BTC for all your hard work and to help with this situation. Thanks for running a great pool.
https://blockchain.info/tx/f434da12572c82f97c22c8d2a56bb2f7bd16e34c5afd31c5441747f975011671

 

Greetings Eligius miners!

So, let me go over a few things. 

First, I've caught up the payout queue completely with txn 3b529c9505fedbe31323f98963ae7bb38b88bb66b95bee7a1c8ce2ad2c58ae86.

Pool has been doing pretty well!  CPPSRB has been serving everyone quite well.  100% PPS has been paid for all work for over the past month!

Unfortunately, there was a minor mishap last evening.  In the interest of continued openness with the community, and to make a long story short, last night the pool reward system code did not pick up on a block the pool found, 245236.  So, the reward system did not calculate anything based on the payouts in that block, and then block 245239 was found without it having knowledge of 245236, thus mostly doubling up on the payouts in the block.  (See 245236 and 245239.)

Quick summary: The pool overpaid some miners.  Fortunately, over 90% of the overpayment was recovered.  The remaining ~2.3 BTC I have covered out of pocket.  The error causing this has been corrected.  Overall, there is no cause for panic and business continues as usual. 

Here are the details.

The cause was an extremely rare race condition between the pool server->database->reward system made possible by our recent growth and the need for multiple eloipool instances.  Code has been put in place to prevent this from happening in the future, as well as a few other potential problem conditions.  (Technical details:  Eligius uses a postgresql back end.  One table is used to store every incoming share in a row, using a automatically incremented sequence number.  Eligius is running 15+ instances of the eloipool open source pool software, all of which have a connection to the database and are inserting rows in a BEGIN INSERT COMMIT style.  We average somewhere around 1000 inserts per second.  As it turns out, there can be an initial collision of sequence numbers.  This is always corrected.  However, there is a rare case where two inserts try to use the same sequence number, they both fail because of it, and get issued new ones, skipping the original.  This causes a brief delay as everything recovers from the collision, and the original insert gets processed several hundred rows after it was supposed to.  In our case, the reward system had already read back the chunk of rows where the block-winning-share was eventually inserted, but before it was completed, resulting in data returned omitting this particular row in the eyes of the real time reward system. I could find very little documentation on this "feature", but, it is expected that there will be gaps in postgres serial sequence numbers  For the correction, the reward system code now checks to make sure that any blocks that contain Eligius payouts that appear on the network are flagged and verified as being reconciled with the reward system.  It also attempts to wait on any gaps in the postgresql sequence numbers and rechecks data near any missing ones.  The combination of the two fixes will prevent this situation in the future.  As an added bonus, the pool software now provides it's current best previous block hash to the reward system and the payouts are verified to match before being valid.).

The reward system missing the block/payout resulted in an over payment of most of the miners included in block 245236's payouts.  When this was noticed, I shutdown the reward system to investigate.  With the payout system disabled, this allowed time for some of the miners who were included in this over payment to continue to mine and slowly correct the over payment.  Luckily, we found five blocks in that time frame.

As of the time the reward system was brought back online and all was reconciled, the total over payment amount came to 6.995 TBC (2.30075557 BTC).  I have personally absorbed this final over payment amount out of my own funds.  Last night, however, I had braced for the worst (all miners affected leaving the pool and not offsetting the over payment) and semi-expected a full 25 BTC shortage, which I would have needed to pay to cover miner payments.

Here is the final remaining list of miners that were overpaid.  Some portion of the over payment was able to be applied to these miners' shelved shares so that the over payment did not result in a huge over payment with regard to work-completed.  However, the payments of those shelved shares is not part of the normal block reward and thus is included in the total over payment amount above.


If any of these miners are kind enough to come forward and return any portion of this over payment, please do so, and it would be greatly appreciated.  I There are no negative balances on Eligius as a result of this.  If anyone is kind enough to return any of these over payments, or is otherwise willing to donate to the cause, again, it would be greatly appreciated and you may send said funds here: 1oopsyR7UB3kA1cr6ka7bDLnYbFRBnHmN (yes, oopsy...)

I apologize to everyone for this issue.  It is fortunate that the majority of the miners at Eligius are honest and continued to mine even after the over payment, and I thank you all for that.  To those who did benefit from this mistake I do welcome you back and will not penalize you if you decide to continue mining at Eligius.  While the mistake was obvious, and I would hope that no one would intentionally take advantage of the pool, it also was not the fault of any miner, so, I can not hold anyone else accountable for the error besides myself.  It is also fortunate that the error was caught quickly.  It is also very fortunate that the form of Eligius's payouts (payments directly from the coinbase) limits the potential impact of these over payments to at most, one block reward, where other reward systems at other pools can (and have) lost hundreds of BTC to errors.

To summarize again:  Business as usual now.  I have fronted the coins for the over-payment out of my own funds (not the pool's).  The error was the result of a rare race condition that was not taken into account in the reward system code.  The code has been corrected.  There was no compromise of any pool systems in any way.  No miners were negatively affected by this.

Thank you all for your continued support of the pool!  I think there are great times ahead for us!

-wk

You're gonna have to update your miner, I think... Give MacMiner a try (https://bitcointalksearch.org/topic/the-official-macminer-thread-btcltc-cpugpufpgaasic-197110)

So i see I wish I understood mining a little better. Using diablo on a mac pro. This might be why I have so many issues. Most pools I get nowhere with. Guild....slush etc. Only one I have any joy with is eligius' getwork, 50btc and bitminter. Bitminter was a no-no after the endless DDOSing.

I just get error messages and nothing happening when I try to connect to others.

Sorry I realise this should be somewhere else....anyone wanna send me somewhere I can get some help?

No connection issues on this end that I can see. Thousands of workers connected.

Can't connect to GBT or Stratum. Getwork worked for me for the last three weeks and now I can't connect to that

stats are not updating for me.

Thank you *very much* for this very detailed explanation. Now I'm convinced you are right

Ironically, a couple of hours after your reply, I see some shelved shares taken to estimated during lucky phase i.e. early in the latest round, for the first time in >2 weeks. Looks like I was impatient (and unlucky, as you pointed out) after all indeed.

Donations are never needed but always appreciated, I guess. You can count on the 0.2 BTC donation as promised, I will post txid in a few days, hopefully from my shelved shares

Thanks again for the time taken responding to this, I appreciate it.

Thanks for the explanations, guys!

To put things in perspective, Eligius used about 30 TB of bandwidth in the first 30 days of using the new server with GBT not redirected to stratum.  Talking about around 100 megabit commit, which is, in general, prohibitively expensive. We had peaks to over 0.5 gigabit outbound during longpoll at peak times, which is ridiculous for something that should be pretty simple.

Since the recent changes we're down to under 1/3rd of previous bandwidth usage.  Still quite a few people using GBT by disabling stratum redirection in their miners, and more power to them.  I think GBT is definitely going to end up as the preferred protocol, but, in it's current bandwidth hungry state, it is pretty wasteful when every miner, big and small, uses it.

*cheers for GBT 2.0*

-wk

There are two reasons stratum has become the preferred protocol:

• GBT uses a fixed amount of bandwidth regardless of hashrate, and was designed for ASICs, not CPUs. For ASICs, fixed bandwidth is a pretty good thing: much less compared to getwork, but for CPUs, it's absurdly high! Unfortunately, the stupid botnets blindly adopted GBT and were creating major bandwidth issues for Eligius.
• GBT is not yet fully implemented. The server is currently expected to send every client a full set of transactions to put in blocks. This adds to the bandwidth needed for it. A full GBT implementation gets this transaction data from your local bitcoind, not the pool. I am working on getting full GBT support into Eloipool (or a rewrite) and BFGMiner, but it takes time to get everything just right.

I expect that when GBT is finished, it can once again become the preferred protocol.

First, let me point out that no donations are needed for this.  I'd much rather you understand the system regardless.

I'm not sure what you mean by "strobes", but I will attempt to explain your personal situation in as much detail as possible.

Lets first take a look at a historical graph of your mining hashrate and balance changes:



And now lets take a closer look at the time after you stopped mining with the pool:



Before I go any further, let me point out that based on these graphs, approximately 0.2 BTC in shelved shares were in fact rewarded to you in the period between the time you stopped mining with us and now.

I believe you somewhat understand the spikes in the estimated balance are the shelved shares that could be paid at the start of a round (lucky) and tapering off as a) no new shares from active mining are added, and b) less of your shelved shares are in the top 25 BTC of the share log as they are pushed further down by active miners' shares.  This is exactly what is happening here in your graph.

So, to set the stage, your most recent shelved share was mined on 2013-06-21.  Then, after that, 0.2 BTC worth were paid/rewarded, making it so your most recent shelved share is even older than that since your newest ones were paid first.  I'll spare you the math in figuring it out (can be estimate with the graphs alone), but as of today your newest shelved share is from approximately 2013-06-17.

So, in order for your shelved shares to be in the top 25 BTC of the share log, and thus have an estimated payout (spike in the estimate balance graph) possible that means that every share from right this moment all the way back to 2013-06-17 would need to be either already paid or total less than 25 BTC.  Consider that is nearly three weeks of time, and there have been many unlucky rounds in that time frame, neither is likely.

It looks like, according to your graph, your shelved shares stopped being in the top of the share log around 2013-06-25.  Specifically, the last block that caused your estimated balance graph to change was 243,363 (compare the times). Lets take a look at the block history from that time frame forward:



Here is the meat of this whole exercise: Notice that block 243,363 was unlucky, 44,469,230 shares at difficulty 19,339,258.  That means this block alone put 25,129,972 newer shares than yours on top of the share log as new shelved.  When block 243,429 was found, the top of the share log (25 BTC, or approximately 19,339,258 shares) was taken as estimated payouts for this new round.  But wait, 25,129,972 in shelved shares were just added on top of your shelved.  25,129,972 > 19,339,258, thus your shelved shares are no longer in the top 25 BTC of the share log.  Looking further up the block list, block 243,429 was also unlucky, adding approximately 23,342,644 more shares on top of your shelved.  So, just the next two blocks alone have buried your shelved shares under over 48 million active miner shares.  Take into consideration that there was a decent run of unlucky blocks to follow, and your shelved shares are so far under currently active miners' shares that there is no way they would be near the top of the share log any time soon.

So, the system is working perfectly, you just stopped mining at a particularly unlucky time, not that you had any way of knowing that.

CPPSRB variance averages out, generally over approximately 60+ days, to level off in the upper 90s% PPS, even 100% PPS at times. (See variance graph on the stats page)  You had only mined with Eligius for about 4 weeks, which, taking into account the bad luck rounds in that time and to follow, is definitely not enough time for variance to average out nicely to a high % PPS.  Even so, according to your stats page you were paid over 90% PPS on average for your work.  This definitely would have been higher had you continued to mine with us, or had the pool been more lucky.

In any case, the system is working perfectly.  Under CPPSRB active miners always have an advantage, since they will always have shares in the top 25 BTC of the share log, and thus are always earning.  When you stop mining you are purely at the mercy of the luck that follows, which in your case was not the best.

I hope this helps you understand the setup better and the details of your particular situation.

-wk

Fair enough. I've been out of contact with bitcoin for a few months, so I wasn't keeping up on all that. I'll just take out those connections as my backups and I'll get a third backup pool for just in case.

I know you weren't born yesterday, because I've seen you around.  That having been said (since I don't think I've seen you trolling):
1) LukeJr doesn't run this pool anymore, Wizkid does
2)

Weird, I would have thought LukeJR's pool would use his protocol(especially when using his miner software)...

The GBT/GW server "suggests" your miner to switch to Stratum server, and your miner accepts it.
You can use --fix-protocol for cgminer or --no-stratum for bfgminer to turn off this feature.

What am I doing wrong? the 3 are connecting by stratum when one should be GBT, one stratum and the other GW.

It is my understanding that leaking of shares might happen when something is going on with the pool which causes your miners to switch over to your backup pool. This flip flop between pools causes a loss in shares as during the switch over a percentage are lost. In this case, via IRC, I read that Eligius was under a DDoS and WizKid057 was doing a fine job combating it. Needless to say though, you would be mining fine for a while, it would hiccup, your miner would switch to a backup, then after a minute or so switch back. My various rigs have stopped doing that since late last night, so I am assuming things were handled, it's been smooth sailing for many hours. (And we've had a nice little slice of luck!)