Topic: Why not to mine on pools mining empty blocks, and why do pools mine empty blocks (Read 624 times)

Great topic Kano and very informative this is the kind of information that should be pushed out to the masses you have a great understanding of all this and I have gained a lot of knowledge from your post's over the years.

Keep up the good work!   

This is irrelevant, I just showed you how a mining pool like F2pool can make 1/2 a million dollars a year by mining the way they are now, not sure why you bring in orphan race here, the previous block (which has transactions in it) has been verified by the pool that found it, it's almost impossible for an invalid transition to make it through, all that you need to do is verify the hash of the last block, this doesn't result in any forks as you claim, we have an average of 2 empty blocks a day, if what you are saying is correct then we would have 2 forks on daily bases.


Not sure how does it fail badly, at least 99% of the hahsrate is at the hands of mining pools that mine empty blocks, those pools that don't are too tiny to be considered, your pool loses 0.0167% of it's hashrate by refusing to mine empty blocks, it's good for bitcoin as a whole but not for your clients, it doesn't matter how you look at it.


If that happened, then it only did because the pool's hashrate increased or its luck did, the fact that other pools were mining on a different blockchain doesn't increase other pools' ability to find blocks.

Unfortunately there are no obvious, usable solutions to this.

As my 1 in the last 2 years points out, indeed empty blocks 'may' occur, though very rarely.

The issue is that if empty blocks where not allowed, any time the mempool gets empty, all the pools mining on the only non-scam coin, BTC, would have to stop mining BTC until there were transactions available.
If BTC transactions counts were very low, then this could lead to a down hill spiral that could possibly kill bitcoin.

That's not really a design option for Bitcoin.

Then of course there's the VERY simple work around, that I also joked about to gmaxwell in the Taproot thread
Any pool could make transactions that shift BTC around in the pool wallet, but not put any of the transactions out on the network.
Thus until they confirm those transaction, they can keep putting one or more of them in their work, to make their transaction count 1 or more instead of zero.
Then when they confirm any, just switch to new transactions that again move some BTC around in the pool wallet.
Of course you can simply increase the number of these transactions to be whatever is required by the network.

FYI: when a block is sent out to the network, the transactions used in it, or course must be available.

Also, the mempool can be empty unexpectedly, e.g. by using a bitcoind that's out of sync with the network transactions, or has been stopped for an hour or so and restarted.

The bottom line is that any block makes the BTC network more secure by adding another link on the top of the blockchain.
But more specifically, in the circumstance where it may be an unavoidable empty block.

Adding a rule to bitcoind to decide to reject blocks on very shaky grounds, like 'what transactions are available', since that will vary from one pool to the next, is quite liable to cause extended network forks/splits.

I remember that incorrect fork.  I was mining on bitminter we were on the correct fork.  We did very well because quite a few were on the correct fork. I think we grabbed 2 or 3 very fast blocks due to that happening.

Well this issue could have Been put to bed years ago.   As developers could have forced it onto the network.

They choose not to do that.

I wonder if we will ever have a new false fork issue.

I was reading bitmain is doing a partnership with viabtc so it looks like hash of pools is linking up.

I wonder if we get more empties due to this.

Firstly, it's mining that could cause an orphan race, that you would be the late one to the party.

Secondly, although I've already done the numbers before, correctly, but using your 150ms is:
150/600000 of an average time block = 0.00025
So one in 4000 network blocks, if every pool was ignoring confirming blocks properly.

So a pool that is 10% of the network, could have one less orphan race, where they are the late one to the race, once every 40,000 network blocks.
i.e. 40,000 network blocks gives 1.3149 times a year.

So, no.

Assuming that: since they have lots of money means they will do the intelligent thing (as is often your argument on the forum) fails badly.

For the sake of simplicity, we will assume that you are not cherry-picking blocks that suit your theory, and let's just assume by mining empty blocks you save 150ms (75% of the whole process you mentioned).

Now since this happens almost every 10 mins, then that is A LOT of money to be made by mining empty blocks, and here is how.

There is 86400000ms in a day, and 144 blocks to be mined, so by utilizing 100ms every 10 mins you get a total of 14400ms which is 0.0167% of the total 24 hours, in other words, Kano pool wastes 0.0167% of it's hashrate by not mining empty blocks.

When a pool like F2pool saves 150ms for every potential block, that's 552510 terahash worth of mining, if that's hard to understand think of it this way.

When F2pool knows about a new block coming, and while going through that 150ms process they switch their hashrate to another PPS pool and say that is Viabtc, by the end of the day they would get paid $1500 from mining for 150ms every 10 mins or a total of 21.1 seconds a day with 22,100,420.00th worth of hash power, but instead of doing this, they would just mine to their own pool and still by the end of the day/week/month/year, they will theoretically make that same amount of money.

And it gets even better when you don't have to download the block in the first place when your pool is synced with the other pools via any other protocol and all you need is their hash (not even the block header) then that makes it well above 200ms, but even with these numbers F2pool can potentially make over half a million dollar in profit yearly, or otherwise, lose it if they use your code and not mine empty blocks.

Edit:

Not saying it's a good thing to do so, nor do I encourage it, just trying to show you that no pool mines empty blocks for the sake of it, doing so means leaving money on the table (transaction fees), but fully making use of every second makes them a good amount of profit, and since all of these pools are motivated by profit only, then it makes sense why would they do so.

Zero EMPTY blocks.
Only once over that 2 year period, did my pool generate a single work item that was sent to the miners that was empty.
(of the approximately 2.2 million work items generated, and the approximately 105000 network block changes)

That one work item falls under the reason why it is 'possible' for there to be empty blocks.
As already explained, when the mempool is empty and no new network transactions appear in the 0-30 second gap.

Alas that is indeed the only time in the past 2 years when any pool could claim that they had to generate an empty block.
2019-02-17 00:14:17.702612 UTC
If they found it within 30 seconds of the block that emptied the mempool.

However, the network block was found about 4 minutes after that, so had 1054 to 1385 transactions available.

So kano.is was 1 block  in  last 2 years correct.

off the top of my head is that about 1 in 300 was empty?

checked the kano.is stats 100 weeks is about 2 years and you made about 100 blocks.

so your % of not empty is 99% correct?

so we just need to see % of not empty for big pools.

If 98% you are better

If 99% you are tied

who feels like checking

Antpool
f2pool
viabtc

for last 2 years.

this is linear.

so 10x or 100x  the hash should not change % of full blocks.

Try reading - I wrote that in the second post ...
Though with the new master server and various upgrades it's even faster

P.S. being ignorant of the relevance of the information, and providing information about your grandmother, makes your post sound ridiculous
The txn sizes I've provided give quite clear insight into how often the mempool gets close to being empty.
(Edit: for the approx 105000 network block changes over that period)

This is irrelevant for the topic that we are discussing here, my grandmother knows that the chances of the mempool being empty at any given time are slim to nothing, what we need to know is how long does it take you to download a full block, verify all of its transactions, remove the current transactions from your mempool, and create a new blocktemplate.

As I've pointed out It is rare for the mempool to be empty.
They are not mining tiny blocks, they are mining empty blocks - no transactions (other than of course the coinbase transaction).
Also, when a pool does empty it, there will be new transactions added between the time the work was sent to the miner and the time the miner found the block.

What this means is: Pool A finds a block, Pool B removes all the txns that Pool A used.
But the transactions that Pool B has available will be at least any sent out by the network after Pool A sent the work to the miner that found the block some 0-30 seconds after the work was sent to it.
Of course if Pool A didn't empty the mempool, there will be txns left over from before that.
Of course this 'can' be no new transactions, but as I already stated:


The catch in it all is that they are actually ignoring the block's transactions and not verifying them before they send new work to their miners.
This has happened before where 2 of the largest pools at the time (antpool/poolin and f2pool) went off on an invalid fork for 6 blocks.

The Taproot change makes this possible again if it is activated.

----

Edit: I've gone through the logs for the last 2 years - since the beginning of 2019.
There has been only one case of empty work generated by my pool in all that time.
The bitcoind debug.log output for that shows:

For that entire 2 year period (2019-2020) low work generation sizes:

Once you get above 4 txns, they all have a block weight of more than 4k.

Currently fees are again around 1BTC (for the last 24 hours or more)
Thus an empty block throws away that extra 1BTC - 16% in fees ...
The mempool has been around more than 10 blocks of transactions for the last 24 hours (and longer)

I don't think it's a matter of a lot or a few, it's either 0 or x, where the value of x is irrelevant in this subject, what happens is that when your pool solves a block and propagate it to me, I can do one of two things.

1- rush to start mining the next block:

all I need is to alter in my block template is the hash of your block which is in the header.

2- Download the whole block, see which transactions did you put in your block, verify them, and then delete them from my mempool so that if I solve the block I won't include something you have already included.

I think some pools are also directly connected and share their hashed blocks outside of the node concept, maybe using stratum API of some kind, where I don't even need to check the block header, I just need to see that you solved block x which has a hash of y and without looking at anything else I start mining the next block right away.

I don't know how long does it take to download, verify and deal with the mempool, but I know it's >0 in time, Kano would know better since he owns a mining pool, or perhaps someone else who owns a mining pool will know, it could be 10 ms, it could be 10 seconds, but no matter how short, that time you spend is time that you could potentially solve a block in, so your loss will be the number of hashes you could have tried while going through that process.

This may cause a hard fork because if your block 10 has something wrong in it and I build block 11 on top of it and send it to another miner who builds block 12 on top of my block, the three of us could be mining on a different blockchain if the others rejected your block number 10 because it has an invalid transaction or a timestamp that is way back in the past or an invalid coinbase address or anything that renders a block invalid which I failed to verify.

But, most mining pools seem to trust each other's work, and so they do mine blocks without verifying them knowing very well that they could lose a huge amount of money if one of the blocks they build on is invalid (especially if they pay PPS).

Some people believe that some mining pools are refusing to add transactions just for the sake of it, i.e they could have included the transactions but they chose not to, it's like saying they left money on the table which is an irrational argument, with that being said, I do believe that antpool has probably done so in the past (the ratio of empty blocks was way too high around the blocksize-war, which could indicate that bitmain was purposely trying to prove that a 1MB block is too small and transactions were taking forever). however, with things the way they are (and have been) for a few years, I don't think anybody does stuff like that, and empty blocks are just the results of mining pools trying to fully utilize every hash they have.

I am interested in what is done to not mine an empty block.

We know there is a time factor involved here.

Ie a block solved 1 second after the last block won't have a lot of tx in it.

So if a big pool solves quickly due o to lots of hash they would be more likely to have tiny blocks

This couldn't be further from the truth, this would have been true if (only if) empty blocks were mined for the sole purpose of not confirming transactions, but as long as it's due to any other reason then your whole argument is wrong.

Pool A solving block 100 before being able to settle block 99 transactions has 0 effects on pool B's probability to find block 101 and include pending transaction, in fact, that empty block has no effect AT ALL on A's chances of solving block 101, that hash only corresponded to that set of incidents and it was not affected by anything that happened prior to it nor it has any effect on the things that will happen after it's occurrence.


Again there is something fundamentally wrong with this logic, while the number of blocks mined by the whole network isn't affected by empty blocks, the number of blocks solved by each pool is.

in other words, if you managed to solve a block that only has the Coinbase transaction in its template and chose not to propagate it because it's empty then that is a block LOST, in the same time if you code your miner NOT to start solving an empty block in the first place then that time x you wait every time you sense a new block is your loss, it could be a millisecond or a fraction of that, it doesn't matter, a wasted hash is a wasted hash.

This whole argument of yours would have been valid if ALL pools would agree on not to mine empty blocks either by not propagating them or not attempting to solve them in the first place, but as long as there is a single miner that does that, then your loss is propositional to his gains with both pool size's taken into consideration, it's not like he is talking "your blocks" but with your hashrate being equal to his hashrate, his overall profitability compared to yours will be higher.

If this is too hard to grasp, think about it not mining empty blocks as being equal to not mining blocks that have less than 2btc in fees or while the temperature in Japan is below 20 or any other condition that you see fit, any rule you add to your code will reduce your total potential profit, there is really no difference between not mining empty blocks and not mining blocks that have less than 2btc in fees as far as one's profitability is concerned.

Now is mining empty blocks bad/good/ethical/scammy is a whole different story which we have already milked quite a lot.

More information about empty blocks for those misunderstanding their effect.

Empty blocks do not mean that more blocks are found each day.

Empty blocks have ZERO effect on luck finding blocks.
You will find, on average, over a large number of blocks, 1 block per Network Difficulty of hashes you do.
Doesn't matter if they are empty, or full, you will not increase your luck by mining empty blocks.

Reducing the time you mine stale work ONLY effects the chance of your pool being involved in orphan races.
i.e. reduces the number of orphans (by a small amount) that you expect to get.

Even if SPV(Spy) mining meant that those pools were mining an extra 1 second per block change of non-stale work, that would mean a total of about 1 extra block of non-stale mining, for each 600 blocks a pool finds using SPV(Spy) mining.
However, in reality it is more like an extra 100ms, so more like one extra block not involved in an orphan race, for every 6000 blocks mined.

No matter how fast the network is, or what short cuts miners might use, the network difficulty adjusts each 2016 blocks, so that the expected average time per block is 10 minutes, based on the average time of the last 2015 blocks.
Some may be quick, some may be slow, but all will be expected as per a poisson distribution - being empty is not a factor in that.

THUS since the TOTAL number of blocks mined per month is roughly the same and a % of those blocks, which are empty, are NOT confirming any transactions, empty blocks increase the average time to confirm transactions.
While this may be obvious to most, alas it is not obvious to all.

Good links regarding SPV mining (empty blocks) and issues it can cause:
What is SPV mining?
Info about it from Core developer Gregory Maxwell that he posted on Reddit
Really have to love this one, again with input from Greg
Pretty safe bet that there is a ton more very old pro's/con's discussions about it here in the Forum to be found by using SPV as a search term.

All in all, maybe back in 2014 there could have been an arguable reason to work only on block headers due to the BTC network being much smaller and other technological parts being slower/not as well established but for a long time now it is pointless to mine empty blocks. The possible reasons to do it simply do not exist anymore.

As Kano has said, what matters most is getting the found block information out the the rest of the network as fast as possible so your found block is the one built upon for new work. Jumping the gun to be first one starting work on a (empty) block is pointless since it's only going to be like a maximum 100ms gain or 1 in 6000 blocks.

Removed you from my ignore list to see what you had to say about this subject, very informative to be honest, and surprisingly you managed to write all of this without insulting anyone like you do 99% of the time.

You are a knowledge guy i give it to you, probably one in a few thousands, if not for your attitude your pool would have been a lot bigger given your experience and lead time advantages, but attitude is not something you can improve via code, so you only keep losing more people, indirectly sending "us" away to the other pools, and quite honestly i doubt you will be able to fix this.

Edit: I've removed some of the irrelevant banter.

My KDB pool code reports ANY work generated that is ever empty, of the work generated once every 30 seconds over the years.
Over the life of my pool, that would amount to less than you can count on one hand.
This, of course, is the same work that is available to EVERY pool at the same time, not just my pool.

Generating work:

I make my bitcoin debug.log report the transaction information for each work it generates, and I can also see in the bitcoin debug.log what is available in the first work after any block change (VERY relevant example further below)

----

"Mandatory" empty blocks have to do with the time between the last work generation sent to the miner that found the block, before finding a block, and the next work generation once your pool found a block.
The transactions that appear on the network during that time (or unused by the block, still in the mempool) decide it ...

It doesn't matter how big or how small my pool is, my pool is part of the bitcoin network and sees the same transactions as every other well connected pool on the network - though mine is most likely the most well connected of all pools

When a block change occurs, my pool will ALWAYS see every transaction that was in the block, that's how bitcoin works.
It will also see what transactions were left over and available to use mining for the next block, in the new work it will send out to the pool's miners.
This is the time gap/number of available transactions that decides if pools MUST generate an empty block with the new work they send to their miners.
This number is the "maybe" less than half a dozen in the past many years.

What the other pools do, to purposefully produce empty blocks, is to not use a template with transactions generated by bitcoind, but simply generate work based on the hash of the block just found on the network, and nothing more - they cannot include transactions in the work when they do this, since they don't already know which network transactions have already been included in the block they don't want to get the transaction details of.
i.e. they aren't verifying the full block before producing a new block work template for their miners.

This was the cause of the failed/incorrect fork by Bitmain and F2Pool in 2015 - they lost all their 5 or 6 blocks on that bad fork since they built new blocks, on a fork of the network based on an invalid block, produced by another pool that ignored the network rule changes.

You could probably force most of the current large pools onto an invalid fork by mining a block with an invalid transaction in it

---

It takes less than 200ms from receiving a block in bitcoind, until it can produce a full new block template with transactions.
I have a message in debug.log when the block arrives with a timestamp to microseconds, another when it has been processed, and another when the pool has been sent the new work.

The largest block in the last 8 hours was 631950 it was 1783877 bytes.
bitcoind debug.log shows (as I mentioned)

Also of interest in this case, it had to extend some disk space for the block.

Total time from when it arrived until the pool got new work 19:07:54.174358 - 19:07:53.985131 = 189227 microseconds 189 milliseconds

As for empty blocks, the one I see in the last 8 hours is 631926 by ViaBTC
I'll include the block before as well for more information:
In this case firstly, you can see the block before, it was 20 seconds before it.
Secondly, in this case the work kano.is generated included 1933 transactions and fees: 0.18324281 BTC
However as can be seen from the next block on the block chain, it was only 315 bytes  ... instead of our "block weight: 3964991"
(You'll have to check the 315 bytes number on any block explorer to see it's an 'empty' block)
i.e. our work that miners were working on was a full block, but ViaBTC miners were working on an empty block when they found it.

It took 196639 microseconds to process i.e. 197 milliseconds

... and of course it didn't take 197 milliseconds longer than ViaBTC - they didn't process it in 0ms

You can run a bitcoind yourself and you can see this information - though you'll have to patch it and recompile it to display the extra information

I've been around in bitcoin since 2011 working on code related to it.
Working on mining code, pool code, and even simple patches to bitcoin itself.

---

Now as to why not to mine on these "empty block" pools?

Well as is clear in all discussions about empty blocks, those pools don't fully process the block before sending out new work.
Thus they can fork bitcoin and cause all sorts of problems with exchanges and transactions, if the empty block they generate is based on an invalid block e.g. with an invalid transaction in it.

This occurred before in 2015 by Bitmain and F2Pool where they forked the bitcoin network and kept ahead of the correct fork for 5-6 blocks with a large % of the bitcoin network, until certain people contacted them and told them they were mining on an invalid fork (I contacted Bitmain ...)

This topic already exists in the past with varyious information,
but it would seem that at least some people have not read the historical threads since they are buried in the past,
and I'd suggest that necroing it is probably against the forum rules,
and some of the old information is out of date.

It is relevant to the pool's area in the same way that the other thread here "Why all miners need to mine on a pool that pays them the tx fees" exists here for a long time.

I'm moving my recent relevant posts here where it is more suitable.

Which pools currently do NOT mine empty blocks, i.e. which pools always fully check each bitcoin block before sending out new work?

kano.is
ckpool solo
laurentia (this pool isn't open yet)