Topic: Vladimir's essential self-defence guide for Bitcoin Miners - page 3. (Read 13274 times)

fair enough. 

Yes, what I said was entirely accurate, and I'll re-word it in case I wasn't clear enough. You cannot eliminate invalids. You can, at best, reduce them.

Yes.  Although it could take some research, i should think that it should be able to be verified by collating data from other pools and blockexplorer if a pool has falsely marked a block invalid/orphaned. 

Not entirely accurate.  WIthout going into details,  there are things which can be done to minimize and practically eliminate a block your pool has found from being orphaned (strictly speaking about BTC here and not other shitcoin chains).   In very simple terms its just making sure that you arrange very good connectivity between your pool and as many other nodes on the net as possible (especially the other largest nodes and especially geographically well spaced out).  

To see some effects of doing this, have a look at this post:
https://bitcoin.org.uk/forums/topic/117-mmc-one-of-the-fastest-bitcoin-mining-pool-in-the-world-with-a-proof/

You can see that in many cases Mainframe is the first to announce a new block to the network even when we didnt even find it.   Of course this can depend on the location from which these measurements are taken but i think it illustrates my point that measures can be taken and gives evidence that those measures can be effective.

Is there any non-falsifiable proof that a given block *was* actually invalid?

No pool can eliminate invalid blocks. Invalids are expected and are a direct consequence of the 10-minute target and network propogation effects because solved blocks do not propogate instantly to every node on the network. Another blockchain with a lower target speed, say 3-minutes, would expect to have more invalid blocks.

Your spreadsheet is slightly inaccurate because you are including invalids in the 'actual' column, but not in the 'expected' column. Until you can compensate correctly for the 'expected' number of invalids, it's best to leave them out of the 'actual' column calculations.

So, if I DDoS the smaller pools that can't afford to buy as much bandwidth in perpetuity as I can afford to rent from botnets for a week and they experience degraded performance as a result. This is either bad luck or the pool operators fault according to you. I don't agree with that assessment. Does it degrade performance and harm miners? Absolutely. Is it predictable according to expected performance by a bitcoin pool? Not in my opinion.


You don't want me to answer the first question because it would be in the form of your quotes:
As an example, of a simple check any miner could to do in order to be confident in his pool
I do not see anything wrong with my suggestion to avoid pools which had extremely bad yield in the past.
And of course, this all started with your essay on "Is your pool cheating you."

You don't want me to answer the second question because it would also be in the form of your quotes:
I have specifically chosen the most "unlucky" pool to illustrate my point.
I have calculated odds for 3 month performance of some pools on that chart..

"Chosen the most unlucky pool" from what dataset? The dataset (http://www.l0ss.net/index30.php) I posted shows a strong negative bias over the last 20 days against the theoretical 0 luck line. Does the set of data that you examined (and then presented us only the small cherry flavored segment) correspond exactly to the theoretical 0 luck?

And what do you have against BTCguild anyway? You seem inordinately hostile and intransigent on this subject, declaring repeatedly that "excuses" do not change the yield". Well, nobody here is offering excuses. I am offering explanations. Bitcoin probability does not account for faulty software patch probabilities. You presenting the aftereffects of one as "some mysterious force" or "excuses" or "incompetence" makes it seem like you have a vendetta against this pool you have chosen.

You can't even calculate the odds of finding an "unlucky pool" when you go looking for one. Why should any of us believe you can find one that is cheating you?

I think what he meant is:

You do calculate the probability of finding a valid hash but comparing it with finding a valid block (which is lower by design because of the probability of chainsplits/orphans being > 0).

No. But I don't think I can ever convince you of the difference between finding history vs the odds of it repeating itself.


If every block was being marked as invalid, would that be a sign of "bad luck" or of a software problem. If, after the pool operator applied a patch and blocks were now being found successfully, would you expect the future runs to revert to every block as invalid? Or would you treat that as a discontinuous function instead of a continuous one?


They don't change the yield, but understanding the events that were causal is much more logical than just assuming you understand the distribution perfectly. Congratulations, you found a pool that were you to possess a time machine you would be able to warn people from the past not to use it.


Well, I suppose I should just let you cherry pick your results and pass them off as statistics.


You don't understand how a DDoS from a botnet could affect performance? Or a patch that prevented submitted shares from being aggregated correctly might skew results? I think I see now why your view on this subject is so narrow.

The odds of finding outcome in a dataset is different than that outcome occurring in a single monte carlo run. You need to calculate what the odds were of finding a pool with luck that bad, not what the odds were that a specific pool would experience that luck in the future. One is more likely than the other, and misrepresenting the data in this fashion doesn't do anyone any favors.


If one were to go back to June 1st and then decide a pool to mine in, you would be absolutely correct. Going forward, what sort of prediction would you make about the performance of BTCguild? I went and asked the pool operator about events that had an outsized effect on performance that were unlikely to recur. There were 3 and they were very visible in the data set. (DDos July 4th-7th, DDoS Aug 12th, a bad pushpoold patch from June 29th - July 2nd). Also, you didn't treat the invalid blocks correctly in your data set, but that is another conversation.


The historical performance is definitely subpar. However, after dropping those 3 periods the picture changes quite a bit. I do mine with them currently, although I didn't point any significant hashing power at them until August, after the dust had settled from the DDoS and patch issue.


You are going to throw out 20 days worth of data that demonstrates a clear trend in 4 pools (including the one your are discussing now)? When I come back in 10 days with 30 days worth of data, will you dismiss that as well? You were not even curious as to why 4 pools were clearly below trend and the other 2 were flat?


First point:: I believe you calculated the odds incorrectly regarding finding a set of bad luck in the search space of pools

Second point: I believe you treated the invalid blocks found incorrectly in your data set

Third point: I believe you represented past data that contained extreme and discontinuous events as historical norms. If you going looking for a problem in the past and show how the performance deviated from expected as a result of those problems, that is fine. You shouldn't represent that search as the result of a true Monte Carlo run to be expected in future performance.

Fourth point: I believe you are dismissing data that disagrees with you because it is not "as significant". Your response should have been: "Oh, that is interesting, I wonder why that is." I wonder why those pools are under-performing and I believe that someone who claims to be providing a defense course for pool-shopping should wonder as well.

You are very right about one thing, with hindsight we should have absolutely avoided BTCGuild on 7 out of the last 60 days. I am trying to keep this discussion collegial and courteous, I am not trying to pick gnat scat out of pepper to just to troll you 

Simulations of pool block finding for different lengths of time gives a good idea of how 'lucky' pools might be in the real world. In the histograms below, the 'luck' index = log((number of shares to find x blocks by pool)/(x * difficulty)).

 

Slower pools will have much larger variability than faster ones, but even after 1000 blocks found some pools will look 'unluckier' than others.

lol - it gave me a laugh earlier today.  I don't think you are a moron, but the expression just jumped off the page at me.

Yep, you are right. one roll of 6 dice is will show a 1 66.51% of the time. /sadface
My point was the probabilities were different. I assure you, my point was not to belabor the fact that I am a moron. 

"quote" we saw 6 dice land and the odds of any one of those dice showing a 1 is 1 in 1. "quote"


WHAT?  After the event, looking at the dice, that's allowable, but rolling six dice and getting at least one "1" is [1 - (1 - 1/6)^6] ~= 0.66

What if they flipped a coin made by a "friend" who claims that he made the coin right. It is the first time he has ever made a coin, but he did his best to make sure it was flat and looked good. We should first test the performance of this coin against the universe of coins made by friends, not against the theoretical 50/50. If we need to choose a coin to flip, we should compare then to each other and then choose one.

Also, the point is not that all pools are having bad luck. The point is that the universe of pools was examined and a single unlucky one was chosen. A better analog than coins might be dice. Imagine throwing six dice. One die comes up with 1 and we calculate the odds of that die landing on 1 correctly as 1 in 6. But that is not what we saw, we saw 6 dice land and the odds of any one of those dice showing a 1 is 1 in 1. Now let us suppose that these dice were not bought from the store, but were each made by a different individual. Now we are no longer sure what the correct distribution of die faces is for any of the dice.

Finally, the pools use different infrastructure. There is work to be done by the server when a share is submitted. What if pool A waits until all that work is complete before acknowledging to the client that it has received a submitted share. If clients cannot begin work on a second share until the first is acknowledged, this serializes the share submission with server share collation at the cost of latency. If pool B allows clients to submit their share and then immediately gives them a new one to work on before the collation work is done on the server, the share submission and processing are overlapped. Pool A's method has the effect of a natural governor on the speed that clients can use the server. Pool B's method minimizes latency but relies on the server to never get bogged down. Under maximum sustained load, these two systems will measure the same. But under failure conditions, they will measure differently.


BTCguild did experience a DoS situation in June, I do not have the exact dates. But the scuttlebutt was that it was from a botnet, that it occurred over the course of a couple of days and peace finally "broke out". I am not sure just what the effect this might have on the "efficiency" of a pool, but I suppose it is possible it could skew the data.


Actually, they do. http://www.l0ss.net/index30.php
As you can see, there is very little if any "good" luck to be found in any of these 6 pools over the last 30 days.
Note btccoins.lc, btcmine, and mtred are all well below the expected luck line. What if there is friction that derives from a common component in the make up of these 3 pools? The luck for these 3 pools is demonstrably worse than btcguild, perhaps you should calculate what the "odds" of their bad luck occurring as well. Then we can calculate the conditional probabilities of those 3 pools + btcguild all having this sort of bad luck all at the same time. 

Also, one side question. How did you account for the invalid blocks?

Vladimir: After looking at your spreadsheet, I found a flaw in your analysis. You need to calculate the Poisson for all blocks found by all pools (and expected), not just the ones found by btcguild. You then need to determine what is the likelyhood that out of all the difficulty periods in all the pools, one will exhibit a run of "bad luck" such as the ones you found in BTCguild. You are failing to account for the selection bias effect.

For instance:

I separate people into many groups and have them all flip coins. I then tally each groups expected number of heads and tails. I see that one group has more heads than tails, and I calculate the Poisson distribution of that group as if those were the only coin flips thrown. Of course, this will show that groups efforts as unlikely to have occurred because I have masked out the efforts of the other groups.

Edit:

Also, you are comparing a real world empirical measurement to a perfect world theoretical average. Since there is no magic fairy dust that would result in a computer solving blocks at an average rate consistently faster than this theoretical average, we can ignore positive biases. If there was an operational friction that was not accounted for, it would exert a negative bias to all results. The way to account is to examine the poisson of all blocks found vs expected, btcguild blocks found vs expected, and all blocks without btcguild vs expected. That may produce a baseline would could illuminate any unseen frictions. It might be interesting to do that exercise for each pool, if deepbit had less friction its size may mask higher rates of friction in other pools.

Spending some of my weekend looking at this and went across to https://en.bitcoin.it/wiki/Comparison_of_mining_pools.  The range of different payment systems introduces the advantages and that was probably the argument that should have been put forward earlier.

If the payment methods were logical and consistent, then the hop vs non-hop largely disappears.  However, having systems where, for example,  "Each submitted share is worth more in the function of time t since start of current round." provides the opportunities to maximise returns.  That's more an issue with pool rules and the motivations of operators to attract people and earn fees.

Over time, miners would probably drift to an equilibrium state where consistent payout pools and hop-friendly pools separate, and then converge as the hopped pools then confer reduced advantage because they rely on non-hoppers to extract their advantage.  Simple - if you want to give away work, belong to a pool with exploits.  Again, that was the point Vladimir was trying to make.

I'll thank you to keep your size-ist remarks to yourself, Sukrim. Vlad looks quite a healthy weight in his photo.

Well, he could run a withholding attack (in any pool/payout scheme), for whatever reason - so it's not too desirable to have HUGE miners in your pool as you have to trust them too.

Why would I care if Vlad supplies say, 100ghash to a pool I mine in?

If it's not PPS based, then all it means is every member gets paid faster and more frequently.
It also lowers the huge variance that occurs at 1.8 million difficulty.

Really.. This, and pool hopping etc.. Are not advanced math. You can figure these out with common sense.

Some people here seem to oppose everything, for the sake of being against something.