https://www.blockchain.com/btc/block/644927
https://www.blockchain.com/btc/block/644928
Observing two adjacent blocks 644927 and 644928, we can see them mined in the same minute. Block 644927 has 612 transactions and the next block 644928 has only 1 transaction.
Since the mining speed is too fast, transactions cannot be sent in this block, so only 1 transaction can be made.
Topic: Bitcoin Empty Blocks (Read 531 times)
But wouldn't this create many empty blocks eventually after the decades of transactions?
no, because we are talking about finding the hash within a ~second and the chances of that happening is very small that it has happened very rarely in the past 646k blocks.All it would take to stop empty blocks from being mined is simply not allowing miners to solve a block's hash without downloading the entire content for that block. I think a simple line of code could even solve this. No?
it is not that simple and there is already a much better solution: incentive. when the price is high and block reward is low and the total fee is a meaningful amount of money, it means if the miner finds an empty block they lose all that extra income.I personally think the block reward should be stripped from empty or stuffed blocks. It's just causing inflation for now real benefit.
the bitcoin design is that you get paid for the "work" you have done and you do pretty much the same amount of work for an empty block or for a full block so you get paid for it all the same, meaning the reward can not change. It's perfectly normal and is usually a result of SPV mining.
When a block is discovered, time is taken for the block to propagate and for the miners to verify the entire block. This can take a few seconds and those time would be wasted if the ASICs and the miners were just to idle and wait for the pool to validate the block. Enter SPV mining, SPV mining takes the block header of the previous block and create a new block header without including any transactions so as to prevent any conflicting transactions.
After the block is fully validated, the pool goes ahead to do everything normally and compile the unconfirmed transactions from the mempool. These kinds of blocks are usually observed when the blocks are mined in quick succession.
When a block is discovered, time is taken for the block to propagate and for the miners to verify the entire block. This can take a few seconds and those time would be wasted if the ASICs and the miners were just to idle and wait for the pool to validate the block. Enter SPV mining, SPV mining takes the block header of the previous block and create a new block header without including any transactions so as to prevent any conflicting transactions.
After the block is fully validated, the pool goes ahead to do everything normally and compile the unconfirmed transactions from the mempool. These kinds of blocks are usually observed when the blocks are mined in quick succession.
But wouldn't this create many empty blocks eventually after the decades of transactions? I know right now the size limited by that single reward it gives out to the miners. But can't a sidechain be implemented for the rewards itself? I do count myself among those few members that think that empty blocks are indeed a waste of resources. All it would take to stop empty blocks from being mined is simply not allowing miners to solve a block's hash without downloading the entire content for that block. I think a simple line of code could even solve this. No?
I personally think the block reward should be stripped from empty or stuffed blocks. It's just causing inflation for now real benefit.
Since block 600,000 (Oct 19, 2019) or in the last 45,343 blocks:
The average block mining time is 9.892152041 min
Very nice.
The average block mining time is 9.892152041 min
Very nice.
Since block 600,000 (Oct 19, 2019) or in the last 45,343 blocks:
Blocks Mined Percent Empty Blocks
1 Unknown 8,448 18.63% 11
2 F2Pool 7,797 17.20% 56
3 BTC.com 5,765 12.71% 33
4 AntPool 5,078 11.20% 23
5 Poolin 4,475 9.87% 9
6 ViaBTC 2,818 6.21% 6
7 SlushPool 2,004 4.42% 10
8 Huobi 1,842 4.06% 14
Hmmm... this was formatted properly before I posted it.
Blocks Mined Percent Empty Blocks
1 Unknown 8,448 18.63% 11
2 F2Pool 7,797 17.20% 56
3 BTC.com 5,765 12.71% 33
4 AntPool 5,078 11.20% 23
5 Poolin 4,475 9.87% 9
6 ViaBTC 2,818 6.21% 6
7 SlushPool 2,004 4.42% 10
8 Huobi 1,842 4.06% 14
Hmmm... this was formatted properly before I posted it.
Quote
I just remembered, LoyceV has everything: https://bitcointalksearch.org/topic/bitcoin-block-data-available-in-csv-format-5246271
Thank you ranochigo. This is exactly what I was looking for.
Since block 600,000 (Oct 19, 2019) or in the last 45,343 blocks there were:
- 184 empty blocks which is ~ 0.4% of 45,343 blocks.
- Average block size of 1,116,072 bytes
- Average block fees USD $3,145. Average Block Coinbase reward USD $90,806. Average profit from fees ~3.5% per block.
- Maximum USD $ / KB is $23 on 2020-02-01 and 2020-07-28. (Note: Fees were over $100 / KB for many blocks in Dec 2017)
Quote
Indeed. I remember Kano being an opponent of SPV mining and some drama between him and F2Pool.
As in f2pool and Antpool mining 4 empty blocks in a row in 2015(?) that were off-chain thanks to their jumping the gun to be 1st out the gate to start a new (un-validated) block by using results from their previous also empty and un-validated block? 
Turns out that someone else had won the Orphan race for that 1st empty & un-validated block that Antpool & f2 had based their chain on.... Oops. Thanks to the back-end monitoring that Kano has he was very quickly alerted to the issue and contacted Antpool ops about it who then quickly relayed the info to f2pool who promptly switched tracks back to the valid current chain and of course invalidating the off-chain blocks they found.
Kano.is uses a modified version of ckpool as its front-end in conjunction with KDB which also directly monitors the network via bitcoind. Most of the code is out there in gits. Mind you his changes to ckpool are not there despite him being one of the major developers of it (the 'k' stands for Kano) thanks to a certain someone kicking him off the ckpool github due to irreconcilable differences in their views on proper coding/testing & pool financial safety. Anywho, the core code is there for anyone to peruse & use but Kano's mods to ckpool and updates to his KDB remain private. Since he does not distribute his changes that does not violate the GPL .
One has to wonder about the date of that video.
Per-Kano it takes around 120-150ms to fully validate blocks. Being the operator of a pool that only uses fully validated blocks and has NEVER mined an empty one, he knows what it takes.
Indeed. I remember Kano being an opponent of SPV mining and some drama between him and F2Pool. BIP152 would've improved the propagation timing further and the effects of SPV mining would've diminished from the past. Per-Kano it takes around 120-150ms to fully validate blocks. Being the operator of a pool that only uses fully validated blocks and has NEVER mined an empty one, he knows what it takes.
I can't comment too much on how Kano and Antpool handles their mining code nor the efficacy of their code because I can't have access to either. He does have sound arguments against SPV mining and miners are mining less empty blocks in the years after that incident.
I just remembered, LoyceV has everything: https://bitcointalksearch.org/topic/bitcoin-block-data-available-in-csv-format-5246271
One has to wonder about the date of that video.
Per-Kano it takes around 120-150ms to fully validate blocks. Being the operator of a pool that only uses fully validated blocks and has NEVER mined an empty one, he knows what it takes.
Per-Kano it takes around 120-150ms to fully validate blocks. Being the operator of a pool that only uses fully validated blocks and has NEVER mined an empty one, he knows what it takes.
Andreas Antonopoulos discusses the reason for the Empty Blocks in the below video from 0 - 3:15 min. According to him "it takes about 15 - 20 seconds to validate" a block. He says this at the start of the video.
https://www.youtube.com/watch?v=dizF2S63RXY
https://www.youtube.com/watch?v=dizF2S63RXY
There are 11 empty blocks in 966 blocks or 1.13% of the blocks are empty from Block 645,198 to 644,232 = 966.
645,198
645,098
645,090
645,027
644,937
644,928
644,644
644,543
644,498
644,394
644,232
Is it possible to download Block Header Data for analysis in a tool like Excel or MS Access? I would like to analyze the average Block Size for the past 50,000 blocks. Average Block Fee in satoshis for the past 100,000 blocks etc... Also I am very curious how many Empty Blocks there are for the past 50,000 blocks.
Thank you
645,198
645,098
645,090
645,027
644,937
644,928
644,644
644,543
644,498
644,394
644,232
Is it possible to download Block Header Data for analysis in a tool like Excel or MS Access? I would like to analyze the average Block Size for the past 50,000 blocks. Average Block Fee in satoshis for the past 100,000 blocks etc... Also I am very curious how many Empty Blocks there are for the past 50,000 blocks.
Thank you
When it comes to validating a block, depending on the composition of transactions, it can take upwards of a few seconds (even more in some pathological/adversarial scenarios) to complete.
perhaps if you use a commodore64..
doublesha of the raw transaction data per transaction in block,
store in list, then hash the list,
then compare this hash with the one provided in the blockheader,
then hash the blockheader and ensure its below target,
we're talking milliseconds here..
heres an example from dash (x11 is a bit more harsh than sha256d, only enforcing the point)..
Code:
2020-08-24 23:37:04 received: block (452 bytes) peer=0
2020-08-24 23:37:04 received block 4829e011bc551411c903f35b1f436f92c2a021d89e3e9b2a12be1ffd632cb736 peer=0
2020-08-24 23:37:04 - Load block from disk: 0.00ms [0.01s]
2020-08-24 23:37:04 - Sanity checks: 0.02ms [0.01s]
2020-08-24 23:37:04 - Fork checks: 0.03ms [0.01s]
2020-08-24 23:37:04 - Connect 2 transactions: 0.01ms (0.007ms/tx, 0.015ms/txin) [0.01s]
2020-08-24 23:37:04 - Verify 1 txins: 0.03ms (0.026ms/txin) [0.02s]
2020-08-24 23:37:04 - IS filter: 0.01ms [0.01s]
2020-08-24 23:37:04 - GetBlockSubsidy: 0.01ms [0.00s]
2020-08-24 23:37:04 - IsBlockValueValid: 0.02ms [0.01s]
2020-08-24 23:37:04 - IsBlockPayeeValid: 1.63ms [0.62s]
2020-08-24 23:37:04 - Loop: 0.01ms [0.07s]
2020-08-24 23:37:04 - quorumBlockProcessor: 0.04ms [0.41s]
2020-08-24 23:37:04 - deterministicMNManager: 3.06ms [1.14s]
2020-08-24 23:37:04 - GetTxPayload: 0.00ms [0.00s]
2020-08-24 23:37:04 - BuildNewListFromBlock: 1.40ms [0.56s]
2020-08-24 23:37:04 - CSimplifiedMNList: 3.39ms [1.19s]
2020-08-24 23:37:04 - CalcMerkleRoot: 3.05ms [1.08s]
2020-08-24 23:37:04 - CalcCbTxMerkleRootMNList: 8.81ms [3.20s]
2020-08-24 23:37:04 - GetMinedAndActiveCommitmentsUntilBlock: 0.19ms [0.07s]
2020-08-24 23:37:04 - GetMinedCommitment: 0.01ms [0.09s]
2020-08-24 23:37:04 - Loop: 0.01ms [0.01s]
2020-08-24 23:37:04 - ComputeMerkleRoot: 0.01ms [0.00s]
2020-08-24 23:37:04 - CalcCbTxMerkleRootQuorums: 0.24ms [0.18s]
2020-08-24 23:37:04 - CheckCbTxMerkleRoots: 9.09ms [3.39s]
2020-08-24 23:37:04 - ProcessSpecialTxsInBlock: 12.22ms [5.01s]
2020-08-24 23:37:04 - Index writing: 0.07ms [0.03s]
2020-08-24 23:37:04 - Callbacks: 0.01ms [0.00s]
2020-08-24 23:37:04 - Connect total: 14.15ms [5.76s]
2020-08-24 23:37:04 - Flush: 0.09ms [0.04s]
2020-08-24 23:37:04 - Writing chainstate: 0.01ms [0.00s]
In perfect circumstances block validation is quite fast, that's correct!
There are adversarial cases where block validation can take quite a bit longer then the block you have presented here. In fact, one of the most significant improvements that segwit introduced was designed to partially address this exact issue, as the signature hashing algorithm was O(n^2). Now it is O(n): https://github.com/bitcoin/bips/blob/master/bip-0143.mediawiki#motivation
While segwit addresses one of these complexities, there are still others (albeit much less vicious): https://bitslog.com/2017/04/17/new-quadratic-delays-in-bitcoin-scripts/
Even if you don't assume the block is intentionally adversarial, there are lots of other factors which can effect validation time, including how much of the UTXO set the validator has cached, and how much of the block's content is in that cache. The libraries and hardware they use is also an important factor. Sipa has a great stackexchange post which details some of this stuff here, he suggests blocks can take multiple seconds to validate: https://bitcoin.stackexchange.com/questions/50349/how-long-does-block-validation-take/50407#50407
Quote from: DougM link=topic=5270775.msg55060150#msg55060150
Interestingly the timestamps for follow on blocks can be earlier in time...that was a bit of surprise to me at first, but I learned I can't assume sequential blocks are in temporal order. For example:
https://www.blockchain.com/btc/block/32649 --> 2010-01-02 04:49
https://www.blockchain.com/btc/block/32650 --> 2010-01-02 02:50
https://www.blockchain.com/btc/block/32649 --> 2010-01-02 04:49
https://www.blockchain.com/btc/block/32650 --> 2010-01-02 02:50
Still reigns true now (https://github.com/bitcoin/bitcoin/blob/0.20/src/chain.h#L18-L30), unsure of the exact rational behind it..
When it comes to validating a block, depending on the composition of transactions, it can take upwards of a few seconds (even more in some pathological/adversarial scenarios) to complete.
perhaps if you use a commodore64..
doublesha of the raw transaction data per transaction in block,
store in list, then hash the list,
then compare this hash with the one provided in the blockheader,
then hash the blockheader and ensure its below target,
we're talking milliseconds here..
heres an example from dash (x11 is a bit more harsh than sha256d, only enforcing the point)..
Code:
2020-08-24 23:37:04 received: block (452 bytes) peer=0
2020-08-24 23:37:04 received block 4829e011bc551411c903f35b1f436f92c2a021d89e3e9b2a12be1ffd632cb736 peer=0
2020-08-24 23:37:04 - Load block from disk: 0.00ms [0.01s]
2020-08-24 23:37:04 - Sanity checks: 0.02ms [0.01s]
2020-08-24 23:37:04 - Fork checks: 0.03ms [0.01s]
2020-08-24 23:37:04 - Connect 2 transactions: 0.01ms (0.007ms/tx, 0.015ms/txin) [0.01s]
2020-08-24 23:37:04 - Verify 1 txins: 0.03ms (0.026ms/txin) [0.02s]
2020-08-24 23:37:04 - IS filter: 0.01ms [0.01s]
2020-08-24 23:37:04 - GetBlockSubsidy: 0.01ms [0.00s]
2020-08-24 23:37:04 - IsBlockValueValid: 0.02ms [0.01s]
2020-08-24 23:37:04 - IsBlockPayeeValid: 1.63ms [0.62s]
2020-08-24 23:37:04 - Loop: 0.01ms [0.07s]
2020-08-24 23:37:04 - quorumBlockProcessor: 0.04ms [0.41s]
2020-08-24 23:37:04 - deterministicMNManager: 3.06ms [1.14s]
2020-08-24 23:37:04 - GetTxPayload: 0.00ms [0.00s]
2020-08-24 23:37:04 - BuildNewListFromBlock: 1.40ms [0.56s]
2020-08-24 23:37:04 - CSimplifiedMNList: 3.39ms [1.19s]
2020-08-24 23:37:04 - CalcMerkleRoot: 3.05ms [1.08s]
2020-08-24 23:37:04 - CalcCbTxMerkleRootMNList: 8.81ms [3.20s]
2020-08-24 23:37:04 - GetMinedAndActiveCommitmentsUntilBlock: 0.19ms [0.07s]
2020-08-24 23:37:04 - GetMinedCommitment: 0.01ms [0.09s]
2020-08-24 23:37:04 - Loop: 0.01ms [0.01s]
2020-08-24 23:37:04 - ComputeMerkleRoot: 0.01ms [0.00s]
2020-08-24 23:37:04 - CalcCbTxMerkleRootQuorums: 0.24ms [0.18s]
2020-08-24 23:37:04 - CheckCbTxMerkleRoots: 9.09ms [3.39s]
2020-08-24 23:37:04 - ProcessSpecialTxsInBlock: 12.22ms [5.01s]
2020-08-24 23:37:04 - Index writing: 0.07ms [0.03s]
2020-08-24 23:37:04 - Callbacks: 0.01ms [0.00s]
2020-08-24 23:37:04 - Connect total: 14.15ms [5.76s]
2020-08-24 23:37:04 - Flush: 0.09ms [0.04s]
2020-08-24 23:37:04 - Writing chainstate: 0.01ms [0.00s]
That's the other puzzling thing, how is it possible that a block could be successfully mined within seconds when it should take on average 10 minutes?
This is an average time range, it could be faster or slower depending on some factors such as the one given by @ranochigo; luck, cause finding a hash is based on randomness. The difficulty level is adjusted approximately every 2 weeks or 2016 blocks to maintain the average block time of 10 minutes.Maybe so...I checked the metrics from blocks 1 to 38,091 (Feb 4, 2010) around the time of the first difficulty change from 1 to 2: Min -65 mins, Max 7,719 min (FYI delta 0 to 1 was an anomaly next largest was 1509 min) and the average was close to 15 minutes versus target 10 minutes. This means the average after that must have been less than 10 minutes for a while to compensate.
Interestingly the timestamps for follow on blocks can be earlier in time...that was a bit of surprise to me at first, but I learned I can't assume sequential blocks are in temporal order. For example:
https://www.blockchain.com/btc/block/32649 --> 2010-01-02 04:49
https://www.blockchain.com/btc/block/32650 --> 2010-01-02 02:50
Compact Blocks (BIP 152), a standardized form of Greg's efficient block relay, a modified version of which is used in the FIBRE network, is mainly designed to reduce the amount of bandwidth required to propagate blocks. It does have the side effect of lowering latency during propagation when High Bandwidth mode is enabled, but ultimately validation of the block is still required:
true but what i meant was that technically the mining node receiving a new block still doesn't have to verify the whole block ie. verification of each transaction inside that block because it has already verified them before when it received those transactions in its mempool. and that is the same principle as BIP-152.
considering the fact that verification takes more time than downloading (specially for legacy transactions with multiple inputs since they have multiple ECDSA and lots of hashing) things could be significantly optimized that way.
You might be interested in the concept of Weak Blocks which is a "pre-consensus" process by which miners propagate blocks with PoW below the target difficulty with the general assumption the composition of that block will remain the same in the event that miner satisfies the PoW. It increases bandwidth but would allow for validation to be front-loaded. It would be trust-full and not consensus enforced, presumably. There is a lot of skepticism of this idea, for various reasons related to incentives etc.
Quote
Once again 2 empty blocks within 1-2 hours of each other (8 blocks apart)
Appears both of those blocks were found immediately (within a second) after the block that preceded them.
Once again 2 empty blocks within 1-2 hours of each other (8 blocks apart)
645,098
645,090
645,098
645,090
Compact Blocks (BIP 152), a standardized form of Greg's efficient block relay, a modified version of which is used in the FIBRE network, is mainly designed to reduce the amount of bandwidth required to propagate blocks. It does have the side effect of lowering latency during propagation when High Bandwidth mode is enabled, but ultimately validation of the block is still required:
true but what i meant was that technically the mining node receiving a new block still doesn't have to verify the whole block ie. verification of each transaction inside that block because it has already verified them before when it received those transactions in its mempool. and that is the same principle as BIP-152.
considering the fact that verification takes more time than downloading (specially for legacy transactions with multiple inputs since they have multiple ECDSA and lots of hashing) things could be significantly optimized that way.
When it comes to validating a block, depending on the composition of transactions, it can take upwards of a few seconds (even more in some pathological/adversarial scenarios) to complete.
that is the main reason for empty blocks but keep in mind that miners (or any node for that matter) don't have to download or even verify the whole block 99% of the time because when their node has been online and receiving transactions that means it already has all the transactions in that block too (unless the other miner included transactions that weren't broadcast to the network which would be a small portion of all transactions) and has already verified them when they received the transaction in its mempool.
which is why the number of empty blocks like this has decreased drastically compared to 3-4+ years ago (eg BIP-152 was introduced in mid 2016).
Compact Blocks (BIP 152), a standardized form of Greg's efficient block relay, a modified version of which is used in the FIBRE network, is mainly designed to reduce the amount of bandwidth required to propagate blocks. It does have the side effect of lowering latency during propagation when High Bandwidth mode is enabled, but ultimately validation of the block is still required: https://github.com/bitcoin/bips/blob/master/bip-0152.mediawiki#pre-validation-relay-and-consistency-considerations
FIBRE of course improves propagation speeds even above HB mode of Compact Blocks. Extremely careful selection of relay points, no validation and forward error correction.
But yes stale rates are very significantly down. This has less to do with validation speeds and more to do with improvements in propagation techniques. Once upon a time everyone pooled up in btcguild, GHASH etc. to deal with this exact problem.
In terms of actually building a block template as quickly as possible, one thing that has improved the speed by which miners can do this is a tighter relationship between CreateNewBlock and the current state of the mempool. This doesn't help you validate the inbound block any faster, but does help you build the next block with greater efficiency: https://github.com/bitcoin/bitcoin/pull/6898
When it comes to validating a block, depending on the composition of transactions, it can take upwards of a few seconds (even more in some pathological/adversarial scenarios) to complete.
that is the main reason for empty blocks but keep in mind that miners (or any node for that matter) don't have to download or even verify the whole block 99% of the time because when their node has been online and receiving transactions that means it already has all the transactions in that block too (unless the other miner included transactions that weren't broadcast to the network which would be a small portion of all transactions) and has already verified them when they received the transaction in its mempool.
which is why the number of empty blocks like this has decreased drastically compared to 3-4+ years ago (eg BIP-152 was introduced in mid 2016).
Jump to: