Topic: Empty blocks - page 7. (Read 23000 times)

I didn't publish it... it's just something I wrote up for myself.  To be honest, it's not the cleanest Java code I've ever written.  For example, rather than handle Exceptions, I just throw them.  Everything is in one file - multiple defined private classes and inner classes.  Hardcoded bitcoin daemon connection parameters.  No comments.  No build system (like maven, etc).

I'll check it out.  Glad to help

Is your code for this open source or available for people to download and run themselves? If so, where can I get a copy.

Thanks so much!.  Here's some preliminary results, if you're interested: https://bitcointalksearch.org/topic/m.11809512

So it just finished... here are the numbers:

As of block 364125, there are a total of 85,422 empty blocks.

AntPool: 5.16% of their blocks are empty, latest one was today.
Discus Fish: 2.88% of their blocks are empty
KnC: 2.88% of their blocks are empty
Eligius: 2.26% of their blocks are empty

EDIT: I made the changes to the code to include the block size... re-running it now.  I'll post it and provide the link when it's done.

Format of the data is:
Block Height, NumTx, BlockTime (UTC), BlockSize (Bytes), WhoMined (if I could figure it out, BTC address of coinbase transaction if I couldn't, or unknown if no BTC address could be deciphered from coinbase transaction).

EDIT 2: Just finished running it again with the addition of block size per your request.  I've uploaded a zip file here: https://www.dropbox.com/s/2rt64pt6hyhr41b/btc_stats.zip?dl=0.  Has blocks up to 364144

Archive contains two files:

empty_blocks.csv has the raw data in the format described above
stats.csv has data on pools, how many blocks they've mined, how many were empty and percentage of empty to total.

Enjoy.

Haha no problem.  The current file format will give me plenty to get started on.  Thanks again!

I suppose it could be... but since I've already started the program execution and it takes quite a while to complete (done 326,000 blocks so far), you'd have to wait a while longer for me to make the code changes and run it again .

Thanks!

Edit: it would be helpful if there was also a "blocksize (MB)" column, if that's easy to add!

I took the file down a while ago... let me run it again and get you the most up-to-date data.  I'll post a link as soon as it's done.

Great work!

I'd love to do some statistical hypothesis testing on the data in this file, but the link seems to be broken.  Any chance you could repost the raw data?  

Today, there is a stress test on bitcoin, and lots of transactions have been made (>7k unconfirmed as of the time of writing), however there are still empty block: https://blockchain.info/block/0000000000000000088390da860e3624e9c6f3162dcce3f2f166bb371e018de7 (height=363266). So I don't think empty blocks should exist as they make people waiting for confirmations wait even longer!

This would be a very interesting study to compare block change times across several pools, the information could be very useful.  Not only might it help people decide where to send their hash (one reason anyway), but it might also motivate other devs to improve their pool code.

... actually
If I get time to, some time soon, I may start up a thread about that.
Specifically comparing pool block change times.

Even big pools like AntPool use Eloipool and also merge mine.
I'd not be surprised if they have crap average block change times also.
I doubt any pools in the past have even bothered to look into this - or if they have, may have got bad results they didn't want to report about their pools.

An expectation of >1% orphans seems quite high ... when the block average is expected to be 600s per block (or less when diff is rising)

I would not be surprised at all if my pool was one of the fastest, on average, to send out block changes.
Will be interesting to see how quickly other pools - not using eloipool - handle block changes - eloipool is slow so it may be a bit pointless comparing it running on other pools.

Although it's only been 254 blocks, the number of orphans at https://kano.is/ is only 0.8% (2) - and my ckdb reports ALL block shares submitted to the pool before it knows about a block change, no matter what bitcoin says about them - there are no 'hidden orphans'.
We've only had 2 so far, one back when the pool started, and one since then that was an example of what other pools may not show, where the block was late for the network, but not late for the pool - yes sometimes network blocks can take a few seconds to get to the pool

ckpool work handling by -ck is fast, but I also monitor the information relevant to block changes and act on it
(yeah my pool even wakes me up in one of the rare cases that causes block delays)

or at the very least has negligible impact considering other delays

For something slightly more generic on the topic: https://tradeblock.com/blog/bitcoin-network-capacity-analysis-part-6-data-propagation

Another update

Last 326 blocks:
95% of Eligius block changes were slower than https://kano.is/
( only 5 of the 326 block changes were slower on https://kano.is/ )

So, if you were mining at Eligius, almost half an hour of mining on stale work in the last 326 blocks.

Clearly their empty blocks are not relevant to the argument they hide behind.

So I've continue this Eligius vs my pool https://kano.is/ comparison
... and I'll repeat: feel free to try this yourself, point 2 cgminers: one at kano.is and the other at eligius to see the block change notifications

I am of course still getting faster block change information from kano.is than from Eligius - well OVER 90%
No wonder the eligius pool luck is so bad.
The block change information from kano.is is a block of transactions that are available after the block change.
The block change information you get from Eligius is an empty block ... but still slower.

I've got over 5.5 days non-stop of stats and I will add, that for 12 hours on my pool a few days ago the stats went crap again for my pool - regularly 10-15s slower than eligius for around 12 hours (that's why I say over 90% and not over 98% as it usually is)
When I've seen a few blocks more than a few seconds slower than Eligius I KNOW something is wrong and do something about it,
since I also know eligius sux slow at block changes and if my pool is slower that means something must be wrong.
That particular 12 hours was a side effect of me not seeing a particular problem ... that I wont miss seeing again

There was an interesting 1.5 hours today (6 hours ago), where Eligius sent no block changes ... mining was still running along fine ... on the old block for 1 hour after it changed (it was a slow network block of about 0.5 hrs - plus 1 hr of mining on it stale from eligius)

Maybe the fact that they don't have someone full time looking after the pool means things go to crap and no one cares

Now there's yet another issue about all this.
Any pool mining using the standard bitcoin uses an RPC function called getblocktemplate()
Here's the actual commit:
https://github.com/bitcoin/bitcoin/commit/3390014fd0d91b0148425e794ac01c10b646a682
Yeah not much of a code change, but of course which name is on it?
Luke-Jr
The same guy who bypasses using it on block changes coz he says it's too slow
Gotta love the ignorant lack of change control on that one
I will also add that the code could be written to provide a block of transactions faster on a block change ... but that is clearly beyond his mcdonalds burger flipping skills where he used to work.

--

Edit: sigh - and on Eligius today they say how they've finished changing stuff and no one should have noticed anything.
Of course no one noticed anything, coz they hid the fact that part of that was an hour of mining stale blocks i.e. the whole pool was mining rubbish for an hour (as stated above) that could only find orphans

Update of my stats for the last 150 blocks:
mining on Eligius has been 1,345 seconds (more than 22 minutes) of stale mining when compared to kano.is
with still >90% of eligius block changes slower than kano.is (making up that 22 minutes)

It's a debate that has supporters on both sides of the argument.  I am pretty firmly entrenched on the side that believes empty blocks are a waste if they are added simply to add a block to the chain.  Now, if there really are no transactions to be included - which was the case quite a bit at the beginning of the block chain - then, by all means if you find a valid solution add the block, because as chalkboard17 points out, even an empty block provides value in securing the blockchain.

Some pools are coded in such a way as to provide empty blocks to their miners until the pool can properly provide transactions.  Whether this is a limitation of the language in which the pool is written, the inability of the pool's coder(s) to properly include transaction, or some other factor, it's wrong.  Kano has provided very clear evidence that the claims of "we do it this way because it's faster" that the empty block guys stand on is indeed patently false.

Since there are almost always plenty of transactions to be included in a block, then I believe it is up to the pool to properly include those transactions in any work it provides to miners.  Creating a block just for the sake of creating a block because your pool can't push work out quickly enough is an excuse for poorly coded software and not a valid reason.  Why should all those transactions have to wait another ~10 minutes to be included on the blockchain?

I don't think that this discussion went in the right way. In general I believe that people don't know if empty blocks are good or bad (neither do I). Instead of discussing the total amount of empty blocks someone should try to explain this in a single post that we can relate to in the future.
I'm not really interested in the exact number, however looking at the picture we can clearly see that AntPool has the highest percentage.

Can someone confirm this or elaborate the usefulness of empty blocks?

Why the Eligius empty block excuse is a scam:
https://bitcointalksearch.org/topic/m.11638137

default string leftovers?  reasonable assumption

Well running ckpool on it's own (without ckdb) is as easy as, for almost anyone without too much expertise to do.
Of course there are other issues - but they are the same for anyone running any pool.
There are a few pools that have used it and probably then changed the line of code that says ckpool after finding 1 or more blocks.
I know of at least 4 - even BW.com has used it for quite a few blocks - but not sure if they now only use ckpool or what else they use.