Topic: WTF is this? Someone found a trick for fast mining? - page 5. (Read 15832 times)

I withdrew any claim. So, stop repeating the same thing over and over. Already Mr gmaxwell rated me as a scammer (at the same level as other people having stolen bitcoins to others) and has tried to bullshit my expertise, which I think at this point says more about him than about me.

But I do believe and I will repeat that some unusual patterns do deserve attention, in particular when the numbers show that these events are extremely rare.

Without being paranoic it is conceivable that some people found a boost on the mining performance (it wouldn't be the first time this happens), and they try to hide it for their own interest.

I fully agree that this possibility has to be treated with caution, but it cannot be ignored and we should be on the look up. If this appears to be the case at the end, some people in this forum will have collaborated concealing this fact. They will bear that responsibility.

On my side I would restrict my comments to statistical facts.

They aren't. They're only ferociously attacked when they continue to say so after it is repeatedly explained to them why such patterns are, in fact, not unusual in the slightest.

Only valiron does. He claims there's something wrong with the mining process but won't explain what. Nobody else is trying to hide it, whatever "it" is.

He hasn't and it doesn't.

Why are people so ferociously attacked every time they say there are some unusual patterns in mining? Do we try to hide something disturbing? In my view valiron has a valid point. This deserves getting a closer look.

Your guy in the blog is enumerating all possibilities of coin flips. In order to have the actual probability for the problem it is much simpler: You start flipping your coin and you stop when you end up with a streak of n=100 or reach the maximum number of throws N=10^8. The starting point of your n=100 streak can be any point between 1 and N-100=10^8-100. The probability streak of n consecutive heads is 1/2^n and the probability to reach the k-throw is about 1 (it is 1-(prob of finding the streak before we reach k)) and this explains my formula, which is accurate except for the slight approximation. You only need the recursive analysis if n is much smaller (or N much larger).  

It's slightly more complicated as it involves Bayesian probability although you're close. A run of 20 heads is a roughly 1 in a million occurrence and the required rolls goes up exponentially the higher you get. For more information, see:

http://www.drdobbs.com/architecture-and-design/20-heads-in-a-row-what-are-the-odds/229300217

What is the probability, under uniform assumptions, of a single nonce being the specific value _2167965896_?    It is one in four billion.   And yet, there it is in block 354640--- a nonce with that specific value.   Does this mean that I now have evidence for some theory in the forum of a one in four billion event?   No.   Because criteria of that specific value was selected after the fact based on the data, and so the probability of observing it is 1 and the information content of the observation is absolutely zero.

Your suggested study is making the same form of reasoning, but this fact is somewhat hidden by the additional complexity-- yet it clearly was based on observing the data (your threshold are the exact differences in the data), rather than being based on some principle which set in advance and only after was it tested against the data.

Putting that aside for a moment, even if I take your back-computed from the data 0.00015552% probability number (without order-- do you really demand ordering?-- if so, you're owed a stern lecture on fallacious arguments); with 354k blocks we would expect to see a 0.00015552%/block event about 55 times; and in the 25 days between your post immediately prior to creating this thread there would be a 42.87% chance of observing it at least once in that window; all with uniform nonce assumptions (obviously the non-uniform nonce reality make it more likely).

I provided the program to cut through some noise; rather than arm-waving we've had in this thread the programs operation is clear, and can easily be tinkered with (e.g. as soon as you assume some non-uniform distribution, you must then integrate; much easier and safer to just twiddle the numeric code and get an approximate answer; especially once you start adding any non-linear hypothesis).

"using the malleability of timestamps" There isn't any reason to assume from inconsistency of timestamps that miners are intentionally using their generally free control of the timestamps for much of anything. Large latencies in miner hardware/software/pooling (including avoiding bandwidth usage sending new midstates) contribute a lot of inaccuracy, but there is no such thing as a singular definition of time in a decentralized system; miners have their own clocks; they often only vaguely agree, the numbers are all over the place, they've always been more or less all over the place, and its not surprising. Every once in a while someone sees a block with a timestamp an hour in the future and they show up freaking out... its ordinary and not unexpected.

As far as my timestamps; I'm reasonably well connected due to connecting to the block relay network.  Stats, in seconds for the last 1295 blocks (which I'm reasonably confident is a generally uninterrupted observation), negative times are blocks 'from the future' according to my local timebase:


     Min.   1st Qu.    Median      Mean   3rd Qu.      Max.
-1539.000    10.000    26.000     1.989    44.000  1098.000

Absolute differences:

   Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
   0.00   19.00   34.00   63.73   57.50 1539.00

Stem-and-leaf plot shows that the extrema are outliers:

  The decimal point is 2 digit(s) to the right of the |

  -15 | 4
  -14 |
  -13 |
  -12 |
  -11 |
  -10 |
   -9 |
   -8 | 3
   -7 | 2110
   -6 | 76430
   -5 | 866631
   -4 | 976633332211100
   -3 | 99986433321000
   -2 | 99999888666655444443333221000
   -1 | 9999998888777765554443333222111110000000
   -0 | 99999988888777777777666666555555555554444444444444433333333333333333+42
    0 | 00000000000000000001111111111111111111111111111111111111111111111111+938
    1 | 00011112222233334444555666677788899
    2 | 02
    3 |
    4 | 5
    5 |
    6 |
    7 |
    8 |
    9 |
   10 | 1
   11 | 0

A density chart of the (-120, 120):

      +-------+-------------+-------------+-------------+------------+---------+
      |                                       ***                              |
      |                                      ** ***                            |
      |                                      *    **                           |
0.015 +                                      *     **                          +
      |                                      *      **                         |
      |                                     **       **                        |
      |                                     *         *                        |
 0.01 +                                     *         **                       +
      |                                     *          **                      |
      |                                    **           **                     |
      |                                    *             **                    |
      |                                    *              **                   |
0.005 +                                   **               **                  +
      |                                   *                 ***                |
      |                               *****                   *****            |
      |                      ********** *                         ***          |
    0 +  *********************                                      ********   +
      +-------+-------------+-------------+-------------+------------+---------+
            -100           -50            0            50           100        

This is from you


Now you are asked to do this:

|nounce(200001)-nounce(200000)| = |2,860,276,919 - 4,158,183,488| = 1,297,906,569, probability =  1297906569/(2^32-1)*2 = 60.4%

Repeat until block 200049 and multiply all the probabilities as you did

Sorry, I don't understand your homework. Can you be more precise?

On the other hand, can we agree that the probability of having independent 144 random variables taking values all below their mean value to be 1/2^144 ? Pretty small uuh? Did I do a good job on that?

You need some serious probability classes around here. No offense please...it is a joke...

If you flip a quarter 100 million times and the probability of getting sometime 100 times heads in a row is 10^8 / 2^100 x(10^8-100) =7.8 10^{-15} pretty small

In order to get heads 100 times in a row you need to flip your quarter about 10^22 times...my guess is that your quarter will disintegrate in the process...

I have no obligation to sit here and teach you basic statistics. This is the job of your stat teacher.

Anyway, do the following homework:


I'm not going to response before you finish the homework.

Interesting. Do you have statistics of time lags between your receiving time and block timestamps? How do they compare to the same statistics on other nodes? I guess by comparing timestamps on different nodes one can tell which miners and how much are using the malleability of timestamps.

Right, it's a normal and expected property of the exponential distribution (that distribution of interblock gaps) that there are a lot of small values; more than you'd probably expect after being told there was a 10 minute average. E.g. about 10% of blocks are 1minute or less apart (assuming lambda=1/600; in reality because of hashrate increases the expected time is often more like 8 minutes). Other constraints like the minimum value for the time coded in blocks also contribute to making the timestamps have less diversity than you might expect from a first guess.

In the case here, they're not even unusually closely timed the times on the four blocks in question are 12:24 13:02 13:12 13:11 (the prior one was 12:03).  Note that the times aren't monotonic, which highlights a previously mentioned limitation in trying to reason about time; they're not all drawing from the same clock as there is no singular now in a decentralized system.

I personally observed these blocks at the following times:

2015-05-02 12:25:21 UpdateTip: new best=00000000000000000f8d7a12d307ddc717cab90d2ced5c7320624a13714b0aa3  height=354640  log2_work=82.71737  tx=67422028  date=2015-05-02 12:24:26 progress=0.999999  cache=14812
2015-05-02 13:01:37 UpdateTip: new best=00000000000000000a1ebf23947c2dc38f980c66c1fd1303235326e36ea5afae  height=354641  log2_work=82.717407  tx=67424094  date=2015-05-02 13:02:15 progress=1.000001  cache=9138
2015-05-02 13:05:29 UpdateTip: new best=00000000000000000f181b8cfb70624cd74bcac01c930657bd1bde85ff59e7fd  height=354642  log2_work=82.717444  tx=67424677  date=2015-05-02 13:12:17 progress=1.000007  cache=10305
2015-05-02 13:08:46 UpdateTip: new best=000000000000000015c33a22604bd9c01806c3add1b33d6b8dd1e663da95cbd1  height=354643  log2_work=82.717481  tx=67425354  date=2015-05-02 13:11:28 progress=1.000003  cache=11807


So gaps of 36:16, 3:52, 3:17.   Given a ~10 minute expected time about a third of blocks are 3:17 apart or less.

Why is it always a conspiracy?  Flip a quarter 100 million times, you'll eventually get heads 100 times in a row. Now flaw in the quarter, just as it's not a flaw that a few blocks get solved in a short period.

Is this kind of comment acceptable in the circles you normally travel in?  If one of the people I worked with presented an argument of this form my response would be "Shame on you".  Perhaps jl2012 has other things to do than click reload constantly and wade through the page and pages of untrimmed quotations in your message?

Doubly so when it has been very clearly explained that your statistical argument is outright incorrect already by others.  Miners do not select nonces uniformly for boring engineering reasons, this is a fact, it's the behavior of hardware sitting right next to me, it's easily observable on the blockchain.   Your statistical argument is that IF nonces were uniform then it would be unlikely to see a run of similar ones.  You do not correct for multiple comparisons (we've had some 354k possible runs of 4 for this to be true in), but most importantly you seem to strangely continue to ignore the fact that we know that various hardware does not uniformly select nonces; and instead you suggest this is evidence of your secret hypothesis. Why are you surprised that we reject your reasoning and instead question your motivations?

The nonces here aren't even that close-- 2167965896, 2148513297, 2143118375, 2456983311  spans 7% of the nonce range...  Since people seem to get caught up on the the analysis, perhaps a numerical example in python might simplify things for people:


import random

trys = 1000000
threshold = 2456983311 - 2143118375
small_rng = 0
for i in xrange(trys):
  n = [random.randrange(0,2**32) for ii in xrange(4)]
  small_rng += (max(n)-min(n))<=threshold
  
print("Out of %d tries, %d groups of 4 nonces were spanned a range equal or smaller than %d."%(trys,small_rng,threshold))
print("Since there are 144 blocks (thus 144 overlapping groups of 4)")
print("We'd expect to see this every %f days with _uniform_ nonces."%(1./(float(small_rng)/trys*144.)))


Which yields:

Out of 1000000 tries, 1524 groups of 4 nonces were spanned a range equal or smaller than 313864936.
Since there are 144 blocks (thus 144 overlapping groups of 4)
We'd expect to see this every 4.556722 days with _uniform_ nonces.


Since we also know existing hardware produces a subset of nonces we should probably expect these runs to be even more common than the above reflects.
To get a feel for how non-uniformity changes this, switching to an RNG with a linear sloping probability, abs(random.randrange(0,2**32)-random.randrange(0,2**32)), increases the rate of these 'small' spans by about 2.8x.

Without a reason to believe the exact criteria was established in advance instead of based on the data we should also probably be correcting for the fact that there are many other possible "patterns" people might find interesting and might use to claim support for some secret theory; e.g. ending with certain digits in some base, or being close mod 2^32, having digits that are cyclic shifts of each other in some base, being close by a larger but also 'small' threshold (E.g. the post originally claimed 6 blocks with a nonce span of 1856814243 which we'd expect to see a bit over 8 times per day.), or having some other simple arithmetic relation, etc.

It has nothing to do with fear of a vulnerability as many of us have heard and researched these discussions many times before. It is merely us being intolerant towards nonsense presented in an arrogant manner or someone setting up a scam by making insinuations and teasing developers that you have a solution but won't disclose it. It is simply bad manners to do with open source projects where we try to share ideas openly.

Go ahead and either prove yourself by writing that research paper or mining faster than anyone else.
I look forward to seeing your whitepaper or  being proven wrong and having to apologize for judging you too quickly because in all honestly it would be really neat if you did find something new we haven't discovered or discussed over the years.


You were so close now want to begin discussing matters more without doing your homework. Please just finish that whitepaper that supports your secret claims and than some of us will happily read it and have something to discuss.

We never heard back of jl2012.

I guess he agrees that my argument wasn't fallacious. I withdraw most claims, but I stand by the claim that to see 144 times heads in 144 coin throws is very unlikely.

I will run some statistics on nounces. I am now very curious about their distribution. Anyone did study that before? 

 

I think you made my point very clear. AS you explained, I don't see any direct threat to bitcoin security.

As I said I withdraw any claims and if I have something relevant to say on this mater it will be in a public research paper. I am not trying to scam anyone, to scare anyone, nor to sell anything. On the other hand I don't either play the game of "having to defend my reputation" because Mr gmaxwell decides so.