You will agree that this facts are independent by themselves. You will agree that they happen rarely with a small probability. The probability of seeing them simultaneously is small.
You add: "This is your test for a specific non-disclosed mining optimization." No, it is not. This is my test to detect that something out of normal is happening.
Don't insist on this line. I have withdraw any claim about mining optimization. Why are you insisting into something that I won't discuss?
You accuse me of divagating but here you chase the optimization tangent which to say you've dropped at the expense of completely ignoring my argument that your own overfit criteria actually precludes your hypothesis that something "out of normal going on"; since exclusion of your training sample from the test observations results in no hits against your proposed criteria.
I did ask a very simple question and you are unable to answer it in simple form: "how would you go about proving that this is out of normal?". Let me rephrase: What is sufficient evidence to infer that something out of your knownledge is going on?
This is a question that ask working scientists ask themselves all the time.
And I thought I answered it adequately, I would only find behavior rare if it was surprising in light of a model that wasn't already fit to the self-same data.
From what you tell you seem to refuse that there is any anomaly if you don't have a working hypothesis
When the behavior can also be explained by "yes, these numbers could have arisen by chance", then I do expect to have a model before I conclude that they're actually surprising.
There is no "ideal way to reason about thing".
I didn't say it was the only way to do so, it was a suggested example idealized approach; offered with the hope of being constructive and not merely repeating that your reasoning was defective without offering an alternative approach.
Yea...so astronomers noticing the advance of the Mercury perihelion
Mercury's precession isn't noticeable or interesting at all without some kind of model. If your theory of the solar system is that the planets just wander around, any sequence of moves doesn't seem especially surprising.
Obviously we can make such assumptions with more knowledge. Let me try something that can pass your censorship: I guess we agree that the main variable for mining a block is the header of the (double) hash of the previous block. If you knew the hash in advance you would be able to premine the block. By "premine" I mean doing most of the computation work well in advance (nothing spurious suggested by "premine", just "pre"-doing the work). The Merkel root tree hash can be kept static except maybe for impact of the extranounce (for example when you mine 1 transaction blocks).
Awesome! So we can finally talk about something specific and technical!
You're talking about using the midstate, as it's normally called; hashing a block header involves running the SHA256 compression function 3 times. One to ingest the first part, leaving the 64 byte midstate, one to digest the end, and then a final one on the output. Because the nonce is at the end of the header one can perform 4 billion runs of the latter two compression runs without repeating the first. The midstate optimization was mentioned in the second page of this thread. It's an optimization exists in every miner, it was used in the original Bitcoin software while CPU mining; it's
enshrined into the getwork protocol (the original remote mining protocol); it's established in the publicly documented interfaces of mining asics-- they are usually setup to receive midstates, not headers from their control systems. It is effectively impossible to write miner software without being comfortable working with a midstate plus a tail. P2Pool even uses this characteristic of the merkle damgard design to reduce its communications costs (it needs to prove a piece of data with a particular hash ends with a particular suffix, so it communicates just the midstate and the final piece).
As an aside, the merkel root cannot be kept static across multiple blocks, it must be recomputed every block, even if you are mining a one transaction block, because the block height is required in the coinbase transaction; and of course changing that or the extranonce or anything else in the block changes the merkel root completely. If you were able to keep it static, it would mean creating a duplicate coinbase transaction; which is prohibited by the network; but if it weren't prohibited it would destroy the prior one and deprive you of your newly mined Bitcoins from it.
You will agree that someone with a performing mining algorithm will have a lot of advantage by witholding mined blocks in order to speed up the mining of the next ones and then releasing them in a short time.
I will not. This is now unrelated to the point you made above: The computation to generate the midstate is generally insignificant. My laptop can generate something on the order of 40 million midstates per second, with the factor of ~4 billion increase from nonce rolling, this one laptop could support midstate generation for nearly the entire network; if anyone worried too much about midstate generation speed, they'd move it onto an ASIC (or just a FPGA); and a single part could easily have 10 or 100 times what my laptop can generate;-- since midstate generation work is 1:2^33 that of the mining in total (assuming the whole nonce range is used, but you can see that taking a factor of 2 or 4 here or there doesn't change things much-- and thus my point before about miners not using the whole nonce range; there is no need to because midstate generation is still cheap).
Generally by withholding blocks you disadvantage yourself because you will lose a race (the network prefers the first in the face of ties, specifically to avoid incentives to delay blocks); without an information advantage (e.g. MITM other participants) you must have a very significant total fraction of the hashrate before any delay of your announcements is not a spectacular loss. This has been studied to some extent in the literature; the most optimistic simulations which assume no latency show in the no information advantage has no amount of withholding avoids being a loss until over the 1/3rd of total hashrate case.
leading to sequences of nearby mined blocks. If he wants to conceal this fact he will mine the blocks anonymously (or with another miner that he controls). This means that we have to pay particular attentiion to close by mined blocks of anonymous origin. And we have to pay special attention to similarities that can reveal that the same miner is behind these blocks
It would be great to prove that this miners are distinct and independent, but they may be collaborating. Event if these miners are independent they may be using the same algorithm that produces similar output.
It seems nothing can disprove your belief. You say it would be the same miner producing the blocks but when presented with evidence that it is different miners, you argue that he would conceal it. Why did we even look to apparently non-anonymous miners in the first place? While doing so, you ignore strong evidence against your theory: That at least two people independently observed those blocks arriving on the network minutes apart; not in a bunch as your block withholding model would predict (and not as you initially believed them to be).
If you were willing to argue that the blocks arriving at the same time, when you were mislead by the data on BC.i, was strong evidence; you ought to be willing to admit that when you find they arrived without bunching that this undermines your theory.
Could you cut this short, don't divagate, and just let us know how you will proceed in order to prove that a sequence of nearby consecutive nounces is out of normal?
How many nounces and with what proximity your criteria will show that they are our of normal?
I have provided my answer on this. We are still waiting yours. Your answer must contain a number and a % proximity. For me n=3 events and a geometric mean of prob of the order of 1% starts to trigger my interest, and light a red flag if
this occurs combined with other unusual coincidences.
Well... I know from past experience that encountering events with probability under 1:2^90, without my expectation being fixed by the data in advance, when I expected a random outcome triggered in me fairly strong confidence that something structured was happening and spent time searching for an answer; but I still did not exclude the possibility that it could be a coincidence.
This is all irrelevant because you're ignoring the model complexity and that is an absolutely critical term which cannot be discounted: I would not be surprised by a complex model I just fit against the data that I'm testing it on. It will always match. If I were to be surprised by this I would be making an error in judgement. I have made similar errors in judgement in the past but I would be unlikely to do so now unless I was tired or ill.
Your comment here of "1% starts to trigger my interest, and light a red flag" makes me laugh out loud. There are 144 blocks in a day (and thus 144 overlapping three block windows). If you really have your attention caught by 1% events then you would be constantly exhausted by them occurring almost every day, often multiple times. I am imagining a movie conspiracy theorist with a bunch of sticky notes on a wall and string run between them. "ITS ALL CONNECTED", ... sorry.
Please don't waste the forums time with observations at the once a day level; none of us have the patience for it. If you really want to work yourself up over some criteria that will happen constantly by chance, in private-- thats your own decision, but please don't inflict it on the rest of us.
You are wrong. I have retracted and erased all comments (if I have overlooked any, please let me know), in particular those that you quote (where did you get the quote?). Obviously I cannot erase people quoting me. You should ask them.
I think you must have overlooked them, I quoted-- they should be easy to find. One is in the very first message. Please note the forum logs edits.
Edit: Thanks they're gone now.
extreme aggressivity at me for mentioning the possibility that algorthms boosting the performance of mining may exist
I've participated and contributed to many minining optimizations discussions-- at least back when they were frequently interesting; I have no objection to that.
I have made crystal clear what my complaint was at every instant and moment: You pointed to some random, uninteresting blocks, alleged on a flimsy basis that they were evidence for a secret mining optimization, known to you but which you wouldn't discuss. I'm glad you're retracting your statements now. But this smelled like a scam, it's nothing personal.
You have been manipulative from the beginning accusing me of threatening bitcoin security.
Secret substantial mining optimizations would be a threat to Bitcoin security (if they were real), the level depending on how significant they were; because if significant enough they could undermine the decentralization of mining which is critical to the system's security assumptions. I have always thought it was unlikely that there was an actual threat there which you were aware of-- initially on the basis that not so long ago you were asking very uninformed questions about mining, but its not impossible that someone new to the subject might discover something that has evaded the analysis of so many others over the past 6 years. My comment there was if I were to accept your belief then it was an additional reason why you should disclose.
Well, my friend, some people reading us are not that naive as you might think.
I'm really not sure how I should understand that.
What is worrisome, really worrisome, is your attitude. People may think that you know about these boosting algorithms and you are just trying to conceal them
from public knowledge
Oh how perfectly diabolical of me to use reverse psychology! You've uncovered my true plan: to conceal the secrets of mining by insisting you reveal them to the public if you're going sulk about claiming to know things! How could you have figured me out! oh no!
using your power position as moderator. In particular, you admitted that you are actively mining, therefore you are incurring in an obvious conflict of interest: I can state, as disclaimer, that I don't mine..
LOL. It almost makes up for all the irritation, so funny; Actually my comment was "it's the behavior of hardware sitting right next to me"; I was referring to a Bitmain S1 which my foot was propped up on at the time and which hasn't been turned on in months.
But again I should be awed by your incredible sleuthing-- I
am actually mining, with downclocked SP20 at 1.27TH/s. With my $0.35/kwh marginal power this costs me somewhat under $200/mo to operate. The P2Pool stats page says that the income is ~0.0107btc/day. So you've totally caught me in my evil plot to prevent people from competing with me for my $130/month _loss_; where by I must spin an elaborate information concealing campaign on the forum in order to lose less money, because simply _pressing an off switch_ is far too difficult for my feeble brain to handle. I truly have been educated stupid, and my participating in mining could have nothing to do with my support for the Bitcoin system and my @#$@# development of the @$#@$ Bitcoin software. Not a chance. It's all a conspiracy. You caught me.
(I'm thankful for your confession of non-participation; always good to know when the writer doesn't have direct experience in the subject!
)
But I am not. I trust your good faith. Even though...I don't like to go into maters discussed in private messages, but since you go into it, I could also go into it and remind you that you wrote something that sounds very disturbing...(and to what I don't give a fuck as I explained to you by choosing the precise wording).
Feel free to quote my whole messages if you like; though I expect they'll reflect poorly on you-- not me.
You attribute yourself the privilege to talk in the name of the whole community. On which grounds?
I don't-- I speak to my personal responsibility and the what I feel is my share of the collective responsibility. But-- if I were to speak for others...
Your position as moderator and developer gives a valuable opinion (more than mine in the eyes of the community, no question about that),
... these might be some pretty good reasons to do so. But, as I said-- I'm not. I even specifically called out in my prior message that I was just speaking as an ordinary community member. Anyone is free to disagree as you have so vigorously done.
we have discovered is that you don't have any serious background on statistics or probabilities.
Among other professional pursuits which required a strong command of probabilities, I spent over a decade working on the design of compression formats; so this is a bit amusing too. While none of my formally peer reviewed publications have been squarely statistically directed, many things I've published professionally or casually have also required fairly significant statistical work-- such as fault tolerance design for large scale communications networks and predictive failure monitoring-- this may be part of why I quickly observed the flawed reasoning like failing to compensate for drawing your parameters from your data, or the impact of multiple comparisons-- none of it is a mystery but its surprising to laymen without a working experience with probability.
There is much that I do not know, thats for sure-- and I am constantly learning new things... and while I'm also sure that there are many things I could learn from you, on the particular subject of working with this kind of data, it seems that most of that would come in the form of observing your errors. I hope you could share this learning with me, but I fear your credentials have blinded you to recognizing the possibility that you're actually far off the mark this time-- you're resorting to increasingly tortured adhomenem, and at this point I can only laugh at it.
Ah, I missed this earlier (it was burred under an untrimmed quote of my graphs):
You cannot deny that there is an advantage in using the timestamp field partly as nounce for mining.
I most certainly can! I did above, in my comments related to mid-state, which apply just as well to recomputing a new merkle root. Some amount of mining gear is able to do ntime rolling, I don't believe many (or any) use it anymore; and I think its actually incompatible with most stratum pools, but I'd have to check to be sure. It isn't a useful advantage because the savings it provides is so enormously removed from the scale of mining that it doesn't matter. Any improvement prior to the nonce increment is effectively reduced several billion fold-- it's like saving 90% of a drop of water relative to the ocean, 90% is a lot of the drop, but nothing of the ocean. Ntime rolling was historically interesting because prior to getblocktemplate and stratum, a remote miner required network traffic every time the nonce space was exhausted and this could produce high load on pools; the ability for remote miners to increment an extranonce themselves in both of the newer remote miner protocols largely made ntime rolling obsolete.
I would like to discuss only technical facts and statistics. After all this is a subforum for "Technical discussions".
Thats what the subforum is for; I'm glad you've now joined its purpose. The complaint all along was that your posts were not conducive to that but instead to something else. Now that thats cleared up, I'm not sure what else is there to say-- in the post I'm responding to here you seem to be saying that 1% events with unspecified other coincidences are deserving your attention. With moderator hat on-- I'm going to ask you to not create threads in this subforum for 1%/block level events without a good, clearly stated, technical analysis that gives a rational basis to believe they're not a once a day pure chance false alarm. ... Your first abrasive message to the thread was the second post with "Last, don't insult our intelligence."-- many people have read and responded patiently. My passive patience ran out at about your dozenth post in the thread. It happens some times, I'm sure all of us are warn out from the argument.
