Topic: Increasing the block size is a good idea; 50%/year is probably too aggressive (Read 14297 times)

I misunderstood, I thought it was +40% and stopping at 20MB.

Under Gavin's 2nd proposal, (starting at 20MB and +40% per year) >32MB is 2 years out.

Most blocks are about 1/3 MB, so the proposal is more or less for about 100x current utilization 2 years forward.

What is the reason to stop at 20MB?  That just seems to be pushing things further into the future to make the decision.

It does mean that the 32MB message size limit wouldn't be a problem though.

I didn't see if this was addressed elsewhere already...
The projected attack would come from a mining concern that is looking to shut out smaller players and consolidate their mining regime.
The cost of the attack is the marginal cost of a winning block being orphaned.  The transaction fee is paid by and to the attacker, at no cost.

However if it is not orphaned, the reward is significant.  While the block is being downloaded and verified by lower bandwidth nodes, the "attacker" is already at work on the next block, and with some decreased competition has some advantages.  It is essentially a denial of service from larger bandwidth miner to lesser bandwidth miner.

Not sure I missed something when I read the source (or rather the comments). The way I understand it is that the TXs get sorted by priority and added. So with now over 5k TX in queue all blocks should be close to the limit, unless there is some sort of minimal priority I missed. Currenlty 328 / 1424 TX have a priority of 0.00000000

I made a 2nd database this morning to get data for TX with and without fee. The load was to much for the server to handle [1] so I had to reboot. Thus the data after 21:00 is not significant for now.

I think the message is clear anyway. The majority of TX without fee get ignored even though there is space left in the blocks. Transactions without fee queued up to over 2000 waiting 3 times in the last 24 hours. The transactions without fee might be spam though. Maybe David Rabahy can share some Excel results?



up to date pictures are on the nodes info page [2] below the connection and traffic stats. I might change the order though.

[1] the white spaces in the pictures indicate that the task to update the database could not be handled within 30 seconds and was terminated. The way rrdtool works is that it handles missing values as "unknown" which are shown as blanks in the graphics.
[2] http://213.165.91.169/

This is the sort of fundamental analysis that would also get us closer to having a client that can tell its user how much TX fee to include in order to expect the transaction to be confirmed in X minutes.
A nice feature to have.

Based on this, for example, Polmine tends strongly toward smaller blocks.  Meanwhile, DiscusFish/F2Pool does a much better job of producing bigger blocks.

A significant number of pools simply run the default bitcoind client and transaction rules. Some have their own rules claiming their rules are "anti-spam" (see the many threads about luke-jr's custom patches to the bitcoind client in the gentoo packaging, and probably any pool running his pool software does so too). Most of the alleged anti-spam selection criteria are (presumably an ideological objection) aimed at gambling sites.

Thanks, I appreciate it. Pictures are indeed on the main page [1], but I thought I just post the picture directly. Less scrolling involved.


Glad you like it. I think the long queue is mainly due to transactions without fee and miners beeing greedy. The TX in question has small (in BTC value) input/outputs, does not pay a fee and the input is not very old, which in my experience results in miners just ignoring your TX. There are plenty of TX with fees, so why should they confirm this one? I had a similar one for testing with 4 inputs and a single output of 0.0022. I didnt confirm in 8 days - by that time the inputs were 2-3 weeks old - so I had to remove it from core and "doublespend" it. I am not sure this would change with a bigger blocksize as most blocks are not full yet, even though they could be. They way the mempool currently is each block should be at the limit now, but they are not [2]. Maybe someone that operates a pool can shed some light on this.


[1] http://213.165.91.169/
[2] https://www.blocktrail.com/BTC/block-all/1 - you can sort by pool by clicking on its name, but I have not found one that has exclusivly big blocks.

Windows 8.1
Satoshi v0.9.3.0-g40d2041-beta
example output from Debug console "getrawmempool true" command;

{
"000308b9c51a0ba76d57efd8897159d95b8278e4fc0e3cb480b3d15343a1aadd" : {
"size" : 374,
"fee" : 0.00010000,
"time" : 1414369834,
"height" : 327133,
"startingpriority" : 4976624.92307692,
"currentpriority" : 5160750.84553682,
"depends" : [
"60c66a89e247760aa4cb29517ba79bbb2bbe773823996135fc7035c74f8be171"
]
},
"00349a4799b7b787e9733f38fc01a8f5dc801f7e35e3071a706831395d67086e" : {
"size" : 520,
"fee" : 0.00000001,
"time" : 1414209735,
"height" : 326867,
"startingpriority" : 40.33333333,
"currentpriority" : 10311.10448718,
"depends" : [
"75ba09c16b35b3495a7d829030dbafbed4e8e6806c8bc58207f8472e85749187"
]
},
...
}

DOS batch file to collapse output so that each transactions ends up on a single line (good for feeding into Excel);

@echo off
Setlocal EnableDelayedExpansion
SET new_line=
FOR /F "delims=" %%l IN (raw.txt) DO (
  if "%%l" == "}," (
    echo !new_line!
    SET new_line=
  ) ELSE (
    SET new_line=!new_line! %%l
  )
)

To invoke;

C:\bitcoin>collapse >btc_txn.txt

Inspirational!

Starting from your spark I found https://blockchain.info/tx/e30a4add629882d360bc87ecc529733a9824d557690d1e5769453954ea4a1056.  It appears to be the oldest transaction waiting at this moment.  It was 31:34 old at the time of block #327136.  Block #326954 was the first block that could have added it to the block chain; 182 blocks ago.  One wonders how old the oldest transaction is that includes a fee.

"The holiday season accounts for between 20 percent and 40 percent of typical retailers' total annual sales."
http://blogs.constantcontact.com/fresh-insights/holiday-shopping-stats/

Now, we have to hope there is just a doubling (the 20% case) of daily transactions during the next Holiday Season (November/December 2014).
Now, Bitcoin help limit expenditures, as it is a "deflationary currency" increasing its value over long time periods.
But, given we are about 80K transactions/day a doubling will be around 160K/day. And it will not be evenly distributed during the day. It will peak at Europe and America business hours and days.

Now, compared with the last year, we have a lot more and larger retailers online accepting BTCs and probably four/ten times brick and mortar places.

We could get long delays during this shopping season, not the next. And it would not be pretty.
The slowdown of the increase of the hash rate will not help, because part of the reason we saw smaller blocks in the past and larger now, it is just there are less blocks per day, something like 1/6 less blocks.
Hope no large miners have any problem during this season, because if the hashrate fall for some reason at critical time, the network could find less 25% less blocks for some hours increasing queue time and confirmations.
And remember good luck is blind but bad luck see you perfectly even in complete darkness. I would not like if miners, cause of bad luck, didn't found a block for an hour during peak shopping time.
And hope the "40% of retailers typical total annual sales" do not happen this year for bitcoin, because the network is in no way able to manage a similar load even without any cascading failure do add insults to injuries.

Nice work shorena!  I sent you a small tip to the address in your profile.  I checked out all the other stats on you full node, overall it is an awesome testament to Bitcoin's openness.

I dont have historical data. But I just setup a rrdtool database to track the number of transactions on my full node. The stats for the last 24 hours are shown here [1], the pic is updated every 30 minutes and I will add more for 30 and 360 days once the database has enough data. As you can see from the little data thats allready there (collecting ~1 hour now) we are allready closer to 4000 transactions waiting than to 2000.
The raw data is gathered every minute with the following command

and is not filtered in any way that is not inherent to bitcoind.


shows that right now 2792 of 3685 TX are without fee. I might make another database to improve the stats.

[1] http://213.165.91.169/pic/mempool24h.png

There have been no unaddressed flaws yet in what I have advocated, other than the fact that it does not include a concrete proposal for a specific finished algorithm.  
What I have been advocating is for a flexible solution to be designed so that we won't have to do this again.  It hasn't been accomplished yet.

If folks decide that reaching an average of 1/3 full blocks is a sufficient impetus to implement something without delay, even if that implementation may well have to be adjusted again in the future (one way or the other), and in future years when it may be much harder to put through such a change... then of course the expedient 2nd Gavin solution will be implemented.

If however before the implementation, there is a flexible proposal that doesn't also introduce unmanaged perverse incentives, I suspect folks may line up behind that.  

In the mean time, I expect to continue taking this role of the loyal opposition in order to either 1)  Find that better solution, or 2) Galvanize the consensus.  
If this discussion we are having here doesn't happen publicly, and look at every option, people may think that what is selected is not the best that we can do under the circumstances.

If after exhausting all arguments against it, the time comes to implement (probably early 2015), the discussions and debate should have concluded with every criticism being having had a chance to be heard, and the best we can do at the moment being implemented.  That will either be "the simplest thing that can possibly work" or something less simple but with better chances of working indefinitely.  Either are an improvement.

+1 Great summary.

OT, but relevant. An interesting piece by Jake Yocom-Piatt, highlighting the CPU bottleneck of signature verification, which seems to rear its head next, after the artificial constraint on block size is lifted:

https://blog.conformal.com/btcsim-simulating-the-rise-of-bitcoin/

The road ahead might be rocky, but at least it is better than facing a dead-end.

My sentiments exactly.  I have actually found this whole thread to be quite heartening.

NewLiberty - you have done a good job tirelessly advocating for a certain approach.  It almost seems as if every even-numbered post is yours, and every odd-numbered post is from a different poster who steps up to the plate to explain to you the flaws in your position.  You seem like a perfectionist, which in general isn't a bad thing at all.  But action is needed now, even if we don't have some perfect 'forever' solution. Fortunately we don't need a 100% consensus to move forward, just an undefined super-majority.  I'd guess we have it around the 20MB / 40% concept.

My personal opinion is 20MB /40% is a kick-ass combo.  Between that, the headers-first downloading, the O(1) miner backbone system, and (hopefully) sidechains, we are looking at the most important set of technical improvements since bitcoin started.

Gavin, if you're still reading in my humble opinion we need a few things to move forward.

You're still the closest thing this community has to Satoshi/leadership so I think impetus comes from you. IMO you should update your Scalability Roadmap to reflect 40% annual increases noting that seems to be able to garner consensus. Maybe a second update includes a step-by-step process to rolling out the update (fork), so people know what to expect. I think starting to talk about things in a matter-of-fact way will engender confidence and expectation of what's to come. Thanks again for your hard work.

For the rest of us I think it's helpful to be supportive, in any way we can, of pushing forward this update. Notice I've updated my signature. If we can explain to those wondering whether it's the best option that it certainly is by implementing something, while covering the most possible bases people may go along.

So if there were a flexible adjustment that kept the highest average below 70%, or even below 50% to be safer, then we would have a flexible adjustment that would be fit for purpose?   Maybe even better than a fixed increase?  Would it be even better if the target max size were 400% of the average, to bring the average to 25% of the max?

The largest block of that period  by a good margin was in the 880KB range.  https://blockchain.info/block-height/271998
The average was bit less than half that block.


What did you see and where did you see it.
200Kt/d may be several years away, yes?
https://blockchain.info/charts/n-transactions?timespan=all&showDataPoints=false&daysAverageString=1&show_header=true&scale=0&address=
(If you are fond of extrapolating this might tell you something)

Or 200kt/d may be much sooner.
So flexible limits are probably better than "whatever" right?

But clearly some blocks are already full right up to the 1MB limit.  I've been doing transactional systems for 30+ years; the serious trouble will start when the average over reasonable periods of time, e.g. an hour or so but not more than a day, begins to approach ~70%.

http://en.wikipedia.org/wiki/Little's_law

Per https://blockchain.info/charts/n-transactions?showDataPoints=true×pan=1year&show_header=true&daysAverageString=1&scale=0&format=csv&address=, Nov. 28, 2013 had the most transactions in a day, i.e. 102010.  From https://blockchain.info/block-height/271850 to https://blockchain.info/block-height/272030, i.e. 180 blocks that day, one wonders what the block size distribution looked like.  Gosh, it would be useful to have the size of the pool of waiting transactions at that time.

Per https://blockchain.info/charts/n-transactions-per-block?showDataPoints=false×pan=1year&show_header=true&daysAverageString=1&scale=0&format=csv&address=, we had an average of 560 transactions per block (only the 8th highest day so far).  Feb. 27, 2014 had the highest average transactions per block of 618 so far.

April 3, 2014 had the highest average block size at 0.365623MB.  Arg, a day is too long.  I just bet the hourly average peaks around 70% of 1MB.

Does *anyone* have a record of the pool or waiting transactions?  That's our key.  When there are ~2,000 transactions in the queue waiting then we would expect a full 1MB block to be coming out next.  When there are ~4,000 transactions in the queue waiting then we would expect two full 1MB blocks to be coming out next.  In this state, transactions can expect to take ~20 minutes to confirm.  ~6,000 waiting -> 30 minute confirmation times.  And so on.

7t/s * 60s/m = 420t/m, 420t/m * 10m/block = 4200t/block.  That does not match observations:  Observations reveal only about 2000t/block.  2000t/block * 1block/10m = 200t/m, 200t/m * 1m/60s ~= 3.3t/s.  Who thinks we can squeeze 4200t/block?  3.3t/s * 86400s/d = 285,120t/d.  Trouble is closer than we thought.  70% * 285.120t/d = 199,584t/d.

Gentlemen, I've seen this too many times before; when the workload grows to somewhere north of 200,000t/d we *will* begin to see the pool of waiting transactions grow to tens of thousands and confirmation times will be well over an hour.

Increase the MAX_BLOCKSIZE as soon as is reasonable.  20MB, 32MB, whatever.  Then enhance the code to segment blocks to exceed the API limit after that.