Topic: High orphan rate and long confirmation time discussion (Read 9910 times)

Well - the fix for higher network TH/s related to this statement, is to move it to the miner ... rather than increase the nonce size ...

https://bitcointalksearch.org/topic/handle-much-larger-mhs-rigs-simply-increase-the-nonce-size-89278

I guess since Satoshi decided a 32bit nonce will be big enough originally, it must be big enough for eternity ...

Either solo miners or the pool.  Whichever entity is putting the block together and including the transactions.

Miner or pool? I thought the pools decided on tx's to include. I could be wrong, this really isn't an area I know well.


EDIT: I think you were referring to solo miners and pooled mining there?

No, very few of them are caused by this anymore.  The transaction linked earlier (http://blockchain.info/tx-index/3618498/4005d6bea3a93fb72f006d23e2685b85069d270cb57d15f0c057ef2d5e3f78d2) is the only one I'm aware of.  As far as I'm aware, that transaction was not created with a standard client, it was purposely crafted in a way to be valid under non-BIP16 bitcoind clients and invalid under BIP16 bitcoind clients.

You can follow on that page the number of orphans that have been caused by it (4 this month).

It is up to the miner to prevent the transaction from being included, yes.  Any bitcoind released in the last few months will automatically reject the transaction and not include it in a block.  There is nothing that can be done about people who refuse to update months after the change was made.  BIP16 did not appear overnight, there was a lot of debate and warning about its implementation.

So it's up to the pool to prevent the old transactions from being included. It is likely in your opinion that at least a good part of the orphans atm are being caused by old and non-BIP16 transactions?

Transactions like the one linked will only cause an orphan to the miner/pool that make a block containing it.  The block they make is invalid per the majority of the network's rules (BIP16), and all properly updated clients will continue building the chain off the previous block, ignoring the invalid one.

So the these non BIP16 tx's will continue to pop up and cause orphans?

If a miner tries to include the transaction above because they have not upgraded to BIP16, they will create a block that is not accepted by the network, other miners won't build on it. That transaction is still floating around on the network, the last time it was included in a block (getting the miner an orphan instead of 50 BTC) was a week ago.

There was a huge spike of orphans in April (around 10% of network blocks had orphan races, one orphan actually had several siblings), but BIP16 is not causing orphans anymore. Are you saying that left over tx from the BIP16 orphans are still yet to be completed and adding to the current average tx size increase?

BIP16 caused a huge amount of orphans. All it took was for one person to push out a transaction to split the network, and some miners have still tried to include it this month:

http://blockchain.info/tx-index/3618498/4005d6bea3a93fb72f006d23e2685b85069d270cb57d15f0c057ef2d5e3f78d2

I'm aware of that. The postulation was that it may be causing the variance in block size.

organofcorti - yes some pools have varying rules about ignoring transactions (or putting in no transactions)
Some of course are very good with putting lots of txn's in their blocks.
This is a grounded well known fact - not a postulation.

tl;dr:
1. Orphan rates for the network have been increasing since May, not February.
2. Block size may have an effect on the likelihood of a particular block being confirmed or orphaned.


A few days ago I finally got network orphan rate and size data. I got it from Blockchain.info (couldn't get bitcoin-tools to work on my mac even with the updated BerkeleyDB module - I apologise for hammering your servers there guys) - since then I've been analysing it and trying the best way to show it, and here it is.


                       


The first chart shows the cumulative sum of orphans since August last year. The big peak in April I assume is due to the BIP16 change, but since then the rate of orphaned block creation has increased somewhat, although since Block height 180000 the number of orphans has increased in an approximately linearly, slope ~ 0.02, or 2 orphans for every one hundred blocks.

The second chart shows the same data as number of orphans per difficulty period. After settling down just after April, the number of orphans per difficulty period has been increasing, as seen in the cumulative orphans chart from block height 175000. This actually tallies well with the blockchain.info chart I posted a page back which shows a spike in number of transactions since May, and may indicate a relationship after all.

The last chart shows boxplots of the orphaned block size minus the related confirmed block size in kB (1e03). For the rare cases of more than one orphaned sibling, the mean was taken. This shows a significant change in the variance of the difference since BIP16 -  lots of small blocks competing against larger ones, sometimes winning and sometimes losing, although the median of the last difficulty period shows a skew toward orphans being larger than the related confirmed blocks.

My hypothesis was that larger blocks with more transactions would propagate more slowly and be more likely to be orphaned. The boxplot for the last difficulty period does lend some credence to the idea, but we'll have to wait another difficulty period or two to be sure.

Another explanation may be that some pools are making bocks with fewer transactions, and other pools aren't. This in itself may be creating random variance and not be related to the orphan production.

Any other ideas?

Actually, I thought of what might be a reasonably simple way to coerce the network a week or so ago ...
Run lots of bitcoind's ...
If you look at blockchain.info it is connected to what? around 250 nodes - would it be that hard to run 250 optimised on their actual block databases and setup appropriately, and point them at a big pool?
However, Luke-jr ignored it when I suggested it ...

Timestamps aren't accurate either.

I've thought about it, jetmine, and I simply cannot see any way that a pool can convince a majority of nodes to accept the block it has produced over someone else's. That would require network coercion on a massive scale.

A chain of blocks from deepbit isn't surpising. Deepbit has about 28% of the network hash rate. They will on average solve 28% of the blocks. Each block solved has a 28% chance of coming from them.

This tends to be more accurate with regards to block ownership: http://blockorigin.pfoe.be/blocklist.php

If only blockchain.info reported block finders correctly. it hasn't for a long time. and is a well known issue.
lots of conspiracy theories come from it
carry on

If you find a block with a super-low hash, you can just broadcast it right away and you GET THE REWARD.  There is no incentive to hold the block back, and play chances.  The chances would be to have to compete against a later block, which could turn out to be an even lower hash.  You can calculate those chances (based on difficulty and your super-low hash), but they never turn out favorably against the guaranteed reward.

Also, how do you know that similar schemes are not already active?  I assume your pool doesnt do it, based on what you write in the post I am replying to.  But there is plenty of evidence that blocks are not just elected by arrival time (standard client).  The two blocks mentioned here in this thread can serve as example.  Blockchain.info RX timestamps were half an hour apart, yet based on same clock source.  There was no 3rd block at this height, so we can safely assume that Deepbit has either not known, or voluntarily ignored the new block for half an hour.  Do you really think that network propagation has caused this phenomena?  Or rather it suggests that Deepbit used a different election criteria than arrival time ...

Think about it.

EDIT: The block in question is 183255.  If you look at the preceding block 183254, you will notice that blockchain.info tags it as Deepbit too.  The same goes for the superceding block 183256.  We are looking at a chain of blocks from Deepbit, with a silenced attempt of interruption by someone else.  But Deepbit "pokered" during half an hour, and eventually killed the competitor.  What was the algorithm used?  It isnt the standard client, so tell me.

No, we're talking about a scenario where there is already an orphan. It cannot be avoided. Even if the whole network focusses on extending only one of those immediately, your next block still has the same risk of an orphan. So it doesn't gain anything. On the other hand, using the lower hash opens the door to new attacks: if I find a block with a super-low hash, I can just wait to broadcast it until someone else finds it with a higher hash, and instant-orphan them!

Using "the smaller hash" has a benefit:  It is unambiguous!

Orphans are the result of a network split (different portions of the network consider a different block as the best current block).  If a client knows two blocks and has to decide, using the smaller hash will always lead to the same decision.  This works towards the goal of reducing orphans (reducing the occurrance of differing decisions on distinct clients).

Your proposal of using the "most beneficious block" does not necessarily fulfill this goal.  Since transactions themselves suffer similar network split problems, two distinct clients can make differing decisions about the same set of blocks.  Thus leading to the same result: orphans.  In fact, your solution may possibly produce more orphans than before.

In the past I have thought about the propagation problem as well, but refrained from posting a formal proposal so far.  However, I came to a similar result.  The single best solution is to use the "smaller hash" (as in "more work" or better: "more luck") which is universally testable for everyone who knows the same particular set of competing blocks.

Also, in my thoughts, it makes sense to consider a maximum expected propagation time.  A block will typically take no more than N seconds to propagate to most clients.  I made calculations, extrapolated from number of hops, current network size, but dont remember the exact outcome.  Whatever the best value of N is does not matter here.  What matters is that when a client first receives a block of new height, it should open a "propagation curtesy window" of N seconds.  When a competing block is received within that window, then it can be reasonably considered potential victim of network skew and a resolution can be made by prefering the "lower hash" block.

If a competing block is received outside of the N seconds window, it can be disregarded(like now).  It can't be victim of just network skew (unless N was poorly chosen).

The advantage is obvious.  Mostly everyone will work on the same block, even in situations that currently produce an orphan.  AND, the proposal is completely voluntary and 100% compatible.  Nobody would be able to tell whether or not you have implemented it (unless he is monitoring your network traffic and sees in which order you have really received the blocks).  This is the best proof for compatibility.  There is no disadvantage to the current system.  The more miners implement it, the lower the orphan rate goes.

And last not least, this approach doesn't open a security vulnerability either.  It is more difficult to create a block with "lower hash" than to create a "just enough" block with better height.  Therefore there is no benefit of trying to create "lower hash" blocks for already existing blocks.  Since this method relies on nothing else but the block hash, is is "unhackable".  Your proposal however does not have this property.  It also depends on the transaction memory pool, which can easily be poisoned.

I have thought about it since about 3 months, and I am absolutely convinced that it is the universally best solution.

For what it's worth, mtred have moved their bitcoind over to 0.6.3rc1 after I pointed it out to them and I have noticed a significant improvement in their block change speed (with much less lag across longpolls) already, even though a lot of the network has not yet changed over to the potentially faster protocol.

https://blockchain.info/
slush just hoovered up over 900 txns

the unconfirmed number is very dynamic, sort of depends on whether pools accepting lots of txns get a run or those limiting txns get a run of blocks
looking every few days doesn't give a good representation

869 Unconfirmed Transactions now

Hmm, no orphans for over 24 hours (since 185346) according to blockchain.
Well at least some good news even if it is unrelated to any code change

... jinx ...

That low?    It was over 9,000 unconfirmed just a few days ago...

lol we did have a nice run while I slept
currently 704 Unconfirmed Transaction as I post, was ~300 when I woke up

We need Ozcoin to hit a block to clear the backlog.   

No it is target difficulty.  Generally speaking longest == more difficult.  The reason why the network checks difficulty and not raw length is because otherwise an attacker could exploit the dynamic difficulty and pull off a 51% attack with <51% of hashing power.  For most intents and purposes outside of possible attack longest == most difficult but the code only looks at the sum of the chain's block target difficulty to determine the best chain.

~1000 isn't rare at the moment.
I restarted my bitcoind 12 hours ago - but the highest I've got since then was only 1338 - so not as bad as some days.
... 1 less and it would have been 1337

1200 Unconfirmed transactions as I post this.   

I concur with TheHarbinger. I wish forum admins would install the SMF Bookmark Mod, which provides the same user experience as "subscribing" to a thread.

I had gotten the impression somewhere that "most difficult chain wins" meant the hash difficulties, not target difficulties. If that is not the case, the only time that there would be a more difficult chain would be on difficulty changes - i.e. if you had received two block 183456s (a block with a difficulty recalc), one may have a more difficult target calculated by its miner. A unique hack that might make your recalc block win an orphan race (once in a million) would be to round up the difficulty recalc within the limits of network acceptance. Outside of that, if the only policy Bitcoin miners use is "first new block received", orphan prevention would seem to be just a race to publication then, that you tell all other miners about your block before they find their own or receive a block from another pool.

longpoll is not the issue, but merely a symptom of other issues. Fix those and longpoll will still suffice... that and I'd be really pissed if I had to support something else after spending months getting longpoll working a treat in cgminer.

An effort to create a bitcoin backbone is now underway...

^ This +1. Any smart pool should be blasting its blocks out across as many nodes as it is able to as quickly as possible. One way to do this is to manually connect to several reliable super nodes.

... implement a protocol rather than use something designed for web servers ...

I agree, longpoll notification is slow. I wonder if using a lightweight message queuing system like zeromq would do a better job instead of the http based long polling.

Chain proof-of-work is calculated based on the hash target, so if you get another block at the same height there is no benefit to keeping the one with "the smaller hash".

Maybe if you receive a second block solution, keeping the block that removes the most transactions from the memory pool would be the right "good for the entire ecosystem" policy.  That way even if small blocks propagate slightly faster that might be offset if a larger, slower block was found.  (but making new-block-flooding independent of the size of the block is an even better solution, and that shouldn't be too hard to implement)

Creating a semi-trusted backbone of connections to other pools/miners so your new blocks propagate quickly is a good idea.

The first poster discovering the client-block-receive to miner-long-poll-receive delay time reveals another factor unrelated to the Bitcoin network block propagation speed - the time it takes for a pool to transmit a new block to all miners. In my past experience, depending on the number of pool miners, the number of pool servers, and the pool software and it's delays communicating with Bitcoin, going through the list of miners and sending each a LP response can take a significant amount of time, generally four, but up to ten seconds from the first to the last miner being notified. This from my own research profiling pool LP times as a predictor of who found the block.

If large block propagation was a significant factor, you would expect to see pools, more highly connected and on faster connections, sending your miner new block notification before your Bitcoin client gets it through the P2P network. That it is the opposite indicates instead the internal pool latency in pushing new work to all miners.


One could make the argument that if your pool and a 0-tx mystery miner both generated a block at the same time, the much smaller block could propagate to miners first and be built upon first with another fast block find. The chief optimization, if you are dedicated to creating larger blocks than other miners may be creating, would be to ensure all your pool servers are highly connected to other pools, and communication of new blocks to other pools' known IPs takes priority. Pools should publish and share IPs of bitcoind's with each other (best it be a closed/trusted list to prevent DDOS facilitation), creating a high-priority sub-network to lower orphans for all.

Another coding optimization would be if you receive a second block solution, to immediately discard the first and push out the newer one if it has a significantly smaller hash, of course still keeping track of both as valid work from miners.

I would caution people to only check this page when you really need to - listing the current 2342781 byte backlog of transactions makes for 7.7MB of HTML.

There are two parts to slowing down network propagation:
1) Sending crap repeatedly
2) Wasting time before sending it

It does seem rather odd that (even after I mentioned it) people are still debating what is actually the problem and looking for answers ...
Is it really that hard to put some debug in bitcoind and send a big block from one place to another and see the results at both ends?
Repeat and rinse.
Is this sort of testing beyond the level of bitcoin devs?

I think he meant that should be a fairly urgent update in order to reduce the problem for pool owners.

It's mostly the rest of the network that needs to upgrade for this to be of any benefit.

That's very interesting and if it makes a big difference should be a fairly urgent update for pool owners if the problem is as big as it appears to be.

Well, a standard ADSL here in (a big city of) Italy is something like 10-20Mbit/256-512Kbit, so a 400-500Kb block requires 8-10 seconds just to be sent to the next peer.

If you are sending it to 10 peers at the same time it takes more than one minute, so it seems that sending a smaller header would help a lot.

spiccioli

P2pool now transmits the block header, so it's possible to see how much earlier
p2pool (150+ peers) saw the block compared to my local bitcoind. This gives
a decent estimation how fast blocks propagate.
 
http://blockchain.info/block-index/239691/000000000000035b015bc1b9cd6e5cbfa4e37371ebae7f444f6774428dcb36c8
block size: 396.08 kB
My bitcoind received it 29 seconds later than p2pool.
 
http://blockchain.info/block-index/239693/00000000000006a3bb36582136c46daad07991460b800534d0195ac99bed939b
block size: 223.01 kB
My bitcoind received it 16 seconds later than p2pool.
 
From what I've seen, the time varies from 0 s to more than a minute, mostly
depending on the block size, but also bitcoind connectivity.

I'd conclude that ATM bitcoin network cannot handle 250+ kB blocks reliably.
No sane pool operator should make bigger than 100 kB blocks with low
0.0005 BTC/kB fees.

Bitcoind:    "connections" : 54, -- so it's well connected, with 10
connections it would be much worse.

My PC is good with no load, internet connection good (10-20 Mbit/s). This is
a bitcoin network thing.

The most recent bitcoin/bitcoin.git has "signature verification cache" which should speed up block validation quite a bit.

The no transactions in the recent bitparking blocks was due to a bug in my attempt to lower the cpu percentage that processing transactions was consuming. I've now fixed this and it should start showing transactions again.

The issue the bitparking pool has had is the priority ordering of transactions in CreateNewBlock in main.cpp was causing lag in the pool and high percentage cpu usage of the bitcoind process when the memory pool had a large number of transactions. I have yet to investigate why since that function is supposed to only be called every 60 seconds or so. Perhaps it takes a lock that causes timeouts for miners or something. At some point I'll investigate and see if that was the cause. Changing this function so that less transactions were processed in the queue seemed to result in a large drop in cpu % (from 80% to 5%).

I've always thought the protocol was missing a `getbody` command.  But what it really needs is four commands:

• getblockfull (got this, it's called 'getdata')
• getblockheader (got this, it's called 'headers')
• getblockheaderandtxids
• getblockbody

The `inv` broadcast after a block is found should result in many `getblockheaderandtxids` requests.  Most miners will have the vast majority of the transactions in any block already, so there is no need for the entire block to be rebroadcast just so they can get the coinbase.

The others `get` messages would be for light clients and previously-not-connected clients.

---

Edit: It also occurs to me that this isn't a huge problem (barring the waste of bandwidth); the difficulty will adjust to take account of the orphans, since they are effectively lowering the hashing power of the miners.

went and dug through the DB
We have had 12 orphans since we went DGM 30/12/2011
2 were our fault when we didnt get the BIP16 updates right
so 10 orphans for over 6.5 months
The last 2 were 15th and 8th of this month, whilst either or both of these may be attributed to current issues, we are not showing an unusually high orphan rate.

We spent quite some time over the last 24hours looking to see what we could do poolside to help the longpolls, stales and big block issues.
We came to the conclusion that apart from optimising some logging and databases some more to reduce load there isnt really much we can do, this is a network issue and the pools will need help from the devs to resolve.

Ozcoin will continue to accept all standard transactions into our blocks, we do not see any advantage to restricting the number of transactions in our blocks.

On a side note, value of txns included with blocks has gone from 0.0x to 0.x with a lot around 0.3 and we had one block with more than 1BTC in txn fees.

Graeme

... and more to that, that is unexplained, is for me block 184660 (and some before that) showed up in my miner 45 seconds after my independent blockcoind had 'ACCEPTED' it.
i.e. block 184660 was VERY slow propagating about the net.

Around the time of that block (before) it happened twice more that I noticed.
The one I saw before 184660 (NFI what block it was) was almost 60 seconds and the 1st I didn't notice the exact amount of time but that was what made me pay attention to it.

But there was clearly a propagation problem for those blocks (in case anyone was thinking otherwise)

bitcoind:
got 184660 at 06/15/12 13:15:08 UTC
and 'ACCEPTED' it at 13:15:17

miner (different computer):
LP was 2012-06-15 13:16:02 UTC

This also shows that my bitcoind was slow (9 seconds) to accept the block, however that can be ignored since my bitcoind has 2 extra processing modules that can slow that down.
(clearly unrelated to the problem since my bitcoind accepted the block before the miner knew about it, not after)

And, yes my clocks are always correct
I noticed the difference itself while watching both logs on the screen, and I've been using ntpd for ... years ...

The question then, is, does this make a difference to them? Are deepbit (and to a lesser extent bitparking) keeping a low orphan rate or is this barking up the wrong tree? I still get the feeling we're missing the underlying cause for this since it predates the abrupt rise in the number of transactions.

Just doing what we can to reduce the txn backlog
More helpers welcome

Yes, because the "notify" link sends an e-mail when there is a new post.  If you are watching 20 or so threads, that's 20 or so junk e-mails you get.  Where as if you make a post in the topic, "sub", then you can just click on the nice "Show new replies to your posts." at the top of the forums and get a list of all the threads you follow with new activity.

So, there is your reason.

What is with all this "sub" spam? Is there some reason you guys can't just click to stupid "notify" link next to "reply"?

It looks like Ozcoin is processing all transactions, Bitparking is passing along no transactions, and Deepbit has a transaction fee threshold that prunes lower fee and/or free transactions from their blocks.

There is the risk that you don't propagate your block fast enough if some other pool finds a block you are witholding.
A 50 BTC risk.

What incentives are there toward propagating blocks? Does the code look for a different node if one node appears to be delaying sending a specific block?

The reason why I ask is this, it would seem that if you are a mining pool that has a reasonable idea of the hashing power of Deepbit, you could take a block they've solved in a faster timeframe than normal and ignore sending it on immediately, instead mining on it for some time simply to get an advantage over the other mining pools that don't yet have the block. Sure, you run the risk of orphaning the block and making your work irrelevant, and it is likely that other nodes connected to Deepbit would send it on fast enough anyway, but the less clients connected to the Bitcoin network, the greater advantage withholding the block nets you.

I'm not saying such a thing is happening here, but there would seem to be some advantage to such a delay.

The size of the 0-confirm queue is shown in debug.log every time addUnchecked() is called just before a txn is stored in memory pool.
And that is called every time - so you can watch it count up with each txn or orphan and drop down with each block
e.g. just now: 06/18/12 21:13:21 addUnchecked(): size 1141
That 1141 is the number of txns in the memory pool - that is the number of 0-confirm txns

Edit: though it is only correct if your bitcoind has been running for a while since you wont know all 0-confirm txns if you just started your bitcoind

Edit2: see here's what happens in debug.log when a block is found - just now - Ozcoin (with other stuff around it removed):
06/18/12 21:24:40 addUnchecked(): size 1465
06/18/12 21:24:50 SetBestChain: new best=000000000000007b3cb3  height=185186  work=359073512741370072393
06/18/12 21:24:50 addUnchecked(): size 1018

So that block (185186) processed 448 txn's from the memory pool (+1 coinbase that wasn't in the memory pool as per expected)
244.80kB in size

Edit3: however the next block by Bitparking was (they ignored all 0-confirm txns):
06/18/12 21:30:13 addUnchecked(): size 1114
06/18/12 21:30:15 SetBestChain: new best=000000000000046b404d  height=185187  work=359080312542205322700
06/18/12 21:30:19 addUnchecked(): size 1115

That block (185187) processed 0 txn's from the memory pool
0.25kB in size

Edit4: the next block was again Ozcoin
06/18/12 21:46:40 addUnchecked(): size 1460
06/18/12 21:46:56 SetBestChain: new best=0000000000000732fdc3  height=185188  work=359087112343040573007
06/18/12 21:46:57 addUnchecked(): size 822

So that block (185188) processed 639 txn's from the memory pool (+1 coinbase ...)
243.31kB in size

Edit5: and for the last edit - the next block was Deepbit
06/18/12 21:49:04 addUnchecked(): size 862
06/18/12 21:49:11 SetBestChain: new best=000000000000085b53c3  height=185189  work=359093912143875823314
06/18/12 21:49:12 addUnchecked(): size 689

So that block (185189) processed 174 txn's from the memory pool (+1 coinbase ...)
49.80kB in size

Sub'd.

M

It seems like part of the problem with blocks not including transactions is the maximum block size of 500kB. Doesn't this pose a scalability problem?

Maybe they should fix the uPnP not working on at least Ubuntu(can't talk about other OS's as I have no experience with them).
I have uPnP enabled on the client, uPnP enabled on my router, yet, if I don't forward the port on the router I never get more than 8 connections.
Maybe an increase in better connected nodes will solve the block propagation time?
Transactions get propagated fairly quick even with 8 connections but maybe full blocks don't as they are bigger in size.

I developed a theoretical concept for a swarm client, that would likely solve this issue.

Basically blocks are split up into branches so that no clients have to propagate, store nor confirm a full block.

It's over in the development discussions.

Really that is the security risk?

Propogation should take at most a few seconds so miners would be wasting work on a bad block for a few seconds and only on the few time (<1% ?) that a bad block w/ valid header is propagated.  Current miners are wasting hashing power on every block from the time it is found until the time it is propagated to them.

Remember the header can't be faked a valid hash on a block with a random nonce as the merkle root hash requires the same amount of work as a valid hash on block header with a merkle root hash which corresponds to an actual set of valid txs.    Not much attack there (attack would by definition needs to waste 100x to 300x as much resources as he causes good miners to waste).  Maybe in very isolate incidences a miner may produce an invalid block and that error isn't detected for a few seconds but miners that do that aren't getting compensated so they have a vested interest in fixing that flaw.

Done.

Only when there's an orphan issue ...

In fact as I told Luke-jr in IRC over a day ago - most bitcoind's already have every txn except the coinbase txn in their mempool.

And in the meantime, all the miners are working on an invalid blockchain...

What security risk?  The proof of work can't be faked.  From the Pow point of view the tx in the block are immaterial.  The merkle root hash is merely a 256bit input.  Also i didn't mean download tx one at a time.  

More like:
broadcast block header -> node verifies and relays -> node verifies and relays -> node verifies and relays

Then nodes request the "block body" from peers and propagate.  If there is no valid matching merkle tree for the header the block is discarded as invalid and network revert to the prior longest chain.

The 0.6.0 RC2 client was released around that time, I think.

No; since you now don't have the block until you've requested every txn you're missing, it would probably take longer. And you can't accept the block until you have all its transactions, or you create a security risk. That being said, it might still help to cut down on bandwidth use; my suggestion in this area is a compromise: send blocks as header+txnlist initially, but keep track of which transactions you didn't have when you received it, and automatically send those to peers you relay the block to before they ask for them. This way, nodes only have to explicitly ask for transactions they didn't have, but the peer relaying the block to them did have.

To address the block propagation problem itself, however, my first recommendation is to relay the block before checking transactions (but after checking proof-of-work and other fixed-cost checks, so DoS attacks are still impractical). The blocks with more transactions still have the cost of transaction verification delay (that cannot be fixed without changes to the Bitcoin system itself), but that delay shouldn't affect orphaning of the block itself.

In all my years of network management, this kind of chart clearly has indicated that I've made a change to my system that's resulted in the unanticipated increased use of a resource.

I'm certainly not an expert on bitcoin or it's code, but this definitely suggests that a change in the code has caused increased confirmation times.

You should probably avoid the use of the word microtransaction: Commonly used meanings span at least four orders of magnitude in value.

Is bitcoin a system suitable for $1 scale value transactions.  Perhaps.  Is it a system for sub-cent transactions, very very likely not.  At the end of the day bitcoin is a worldwide broadcast medium. The broadcastness provides very high confidence that it can't be artificially inflated and that old txn can't be reversed. But the fact that no information is hidden does ultimately limit the scaling— though I don't think that its much of a problem. You don't need that kind of security for very tiny transactions. So you can use something that isn't broadcast based to trade small values and frequently settle against bitcoin. ::shrugs::

Of course, none of this has anything to do with improving the performance for what we have today, of course that has to be done.   But the solution to that is "shut up and code" and "shut up and test", not more navel gazing on the forums.

An increase in the number of transactions on the network is of course what EVERYONE wants.

What everyone may or may not want is how that happens.

But seriously, the BTC network is a tiny ant in the monetary world and certainly needs to be able to deal with something a LOT more than it can (without people complaining about it) today.

Attempting to reduce the transactions (as I've seen some suggest) is, IMO, completely against the BTC ideal.

IMO, anyone thinking that anything but a few rules should be encoded into the basic bitcoind spec and base code is missing the point.

Slowing down or ignoring txns/addresses/fee levels is up to pools and solo miners.
As long as pools aren't lying about what they do (or hiding the facts) there is no technical issue there, since people should mine at pools they are happy with the way the pool handles transactions.
The base client should not have much if any of this at all.

The problem that comes with this is, however, reality.
Some pool OPs may lie or hide what they are doing.
A lot of miners don't really care if their pool is fucking up the network.

Still, it's a BTC free market, not a BTC dev fascist state (even if some don't see it that way


Definitely the software needs improving, not more rules and restrictions added to it, to stunt the growth of BTC.


OT: Just a point about block times - yes if you look through the block-chain you will regularly find blocks that have a timestamp before the previous block.
If you look at the previous block you will often see that it is Eligius (at there, a 2000% E: is considered possible and reasonable)
That's life ...

That solution is indeed possible and already being worked on.

There are some other things being looked at as well, for improving block propagation times.

Confirmations can't take zero minutes on average, that graphic is broken. Presumably the data doesn't start until the point where it first ticks above zero.

Can you update your post to reflect this? The implication that delays started so early is causing people down thread to jump to erroneous conclusions.

Well I am less concerned about block prop times as doesn't the new (0.7) of bitcoin eliminate the double verification of block txs.  That should dramatically help with prop times.

Also correct me if i am wrong but could block prop be spit into a two step process.

1) Broadcast block header.  Nodes can verify the validity of the pow w/ just header.
2) Nodes request block txs from headers they have confirmed are valid.

This would result in full block taking no longer to verify and propogate than a 0 tx bock.  It would also provide other useful benefits.


I am more concerned with some unexplained factor causing a rise in avg confirmation time.  The again if I was running a major pool I might not feel that way.

Now it would be interesting to see if the orphaned blocks have more transactions included than the blocks in the main chain.

The avg confirmation time isn't the (immediate) problem. The problem is block propagation time. If that isn't solved, you will likely see another sharp rise in avg confirmation time soon.

Thanks for the charts it illustrates the symptoms clearly.  That sharp rise in avg confirmation time can't be explained by increased tx volume.

Full history of orphaned blocks at p2Pool, ordered by numbers of blocks solved:



Although it looks like this could just be because more blocks are being solved, there's no obvious correlation between orphans and hashrate:



I'd like to get data from a much more established pool though.

Bear in mind that block generation times can be not remotely accurate with rolltime enabled. cgminer limits rolling to about 10 minutes, so the time could be 10 minutes out from the real local time, but apparently luke-jr has removed all limits on his rolltime on eligius meaning work from there could appear to be from any time in the future.

I'm looking over some of the orphan races at:

http://blockchain.info/orphaned-blocks

Races where 2 blocks are found with timestamps a few seconds apart is to be expected. But I'm seeing quite a few races where block are minutes apart. If it's taking blocks minutes to propate across the network, we've got a problem.

Like here's a particularly nasty one.

http://blockchain.info/block-index/232898/0000000000000873fda1001c4986936dfa3697ec84ff74e48c11fe9d355be07f

vs.

http://blockchain.info/block-index/232896/000000000000069c27242a928e33e45ddd27d6fd014c77946ba4d3d2891f5398

Some poor miner finds a block, and 30 minutes later DeepBit still hasn't received that block and thus orphans it when they find a block. 30 minutes!

Smaller tx fees started with Bitcoin vers 0.3.23, which was available from June 2011.

Edit: as gmaxwell suggested I contacted Blockchain.info and indeed they only started recording Avg tx conformation time In February. That being the case, it doesn't seem that there's a sudden large increase after all.


Just to make it clear what we're talking about:





In case anyone can think of anything that occurred in conjunction with the increase in tx conf times, the start of the current tx conf times is 2012-02-26, with previous spikes on 2012-02-06 and 2012-02-15 to 2012-02-16.

After the two weeks you get the miners who hadn't switched to the BIP 16 enabled code getting orphans. Maybe there's a bunch of standalone miners running non-upgraded code that are unattended (remains of botnets, unattended servers, etc?)

I'm not implying it's actual BIP16 transactions that are responsible, but whatever code changes or protocol changes were required to support them since the correlation with rising transaction time is surprisingly strong.

Interesting, but the increased confirmation times and orphans persisted, so perhaps it's something about the protocol itself rather than it interacting with the non-BIP16 world? The timing of 2 weeks before correlates almost exactly with the graph showing increased transaction time.

IIRC a bitcoin release candidate was used by some pools that had a BIP 16 switch on time 2 weeks earlier than what the official time eventually became. Pools and individual miners using this RC would have produced a lot of orphans until they noticed. This could cause increased confirmation times for a brief period when a chunk of the network was effectively offline. I don't think this was very long, nor was it a big chunk.

I have no idea but it is the one most significant change to bitcoin that correlates very strongly with that time frame, unlike satoshidice's increase in transaction volume. Is BIP16 the same time we started accepting smaller transaction fees? They're allegedly 1/20th of what they were, but I don't know when that change hit. Maybe they really did need to be bigger?

Do you know of any way to find out when pools started to apply BIP16, without having to trawl the threads? I could look at every pool and try to correlate orphan rate with BIP16 take up.

Why would BIP16 cause increased confirmation times?

It was accepted April 1, but there were pools that had already adopted it if I recall correctly.

Don't think so - BIP16 was introduced 1st April, wasn't it? Increased confirmation times began in March. Can you think of anything else that changed then?

Interesting... is this BIP16 then?

This is a very interesting question and I'm glad you made a thread for it. From the little investigation I've done it seems that there's been an increase in orphans since about March, which coincides with a sudden increase in confirmation times. The strange thing about this is the sudden increase in confirmation times does not coincide with the increase in transactions per block which started to increase from May, two months later.

This can be seen here:

http://blockchain.info/charts/avg-confirmation-time
http://blockchain.info/charts/n-transactions-per-block

So the increase in tx per block does not seem to have caused the increase in confirmation times. Satoshidice doesn't look like the culprit to me.

It is a major issue - if you look at the sort of transaction volume most stock exchanges or major payment processors get, it seems doubtful that the network as it stands could cope.

Well said.

We are working on this issue for Ozcoin today, if it is suitable and works we will share our solution, I know others are trying different things too

So it appears pretty clear that most pools are suffering from a significant rise in orphan rates and many pool owners believe the issue appears to be related to slow block solve propagation due to the massive increase in transaction volume (mostly due to satoshidice).  Now it's clear that while only a small number of us are gamblers and care about satoshidice's success, the fact is most users of sd are obeying the transaction fees as determined by bitcoin (which are rather small now, but still present). So we're hitting some scalability issue with the network sooner than imagined. Some say that bitcoin was never meant to be a system for micro-transactions but that actually appears completely wrong to me if we're to assume widespread acceptance of btc.

I've seen massively delayed longpolls from various pools now and some just accept the shares long after the block has changed before they're sending out the longpoll, while other pools have a deathly pause, not returning a response about shares for up to 25 seconds, then sending out a longpoll and 25 seconds worth of rejected work. Some say that deepbit's transaction limits means they're avoiding this problem, but I'm not informed enough to know if this is true, but it would seem to me this must incur significant transaction delays. Some solution needs to be worked on that benefits all - pools, miners, and bitcoin at large. I don't want to see transactions delayed for up to 6 hours when even transaction fees are paid, but nor do I want my pools to become a massive orphanage.

Please discuss.