Topic: Idea on the "Blocks are [not] full problem" (Read 2097 times)

What do they charge for outgoing bandwidth?

What exactly would be the purpose of running a node that simply validates blocks and doesn't relay them to anyone?  Oh, wait, I guess they could relay it to one node, one of the 'powerful entities' that runs a pool.

1Gbps dedicated ports w/ unlimited traffic still cost quite a bit.  They probably won't in a few years.

I think people should just pay their 10c and stfu.

Hi! My first post here.

I don't consider this padding of blocks a good idea.

It can lead to a "compact blocks miners conspiracy ". Some miners agree to pad their blocks with predictable data, like, let's say, only zeros or only ones. Then they relay a compacted version of the blocks between themselves, like [header] + [valid data] + [number of zeros]. Or something.

Members of the conspiracy would suffer with much less orphaned blocks than the other miners. As the conspiracy grows, [header] + [valid data] + [number of zeros] becomes the new bitcoin standard.

Marvelous! This is so far the best post I have read on this topic!

The core value of bitcoin is to get rid of the risk of a centralized organization disturbing the money supply and transactions, that value is highly desired when people store and transfer significant amount of capitals

But for daily grocery spending, people don't complain if their credit card company has made the price of a bottle of milk 3 cents higher, and they don't care that bitcoin can make them pay 3 cents less either, because the risk involved here are so small that it does not worth the effort

Maybe for purchasing of cars and houses people could use bitcoin, but for each person that is typically 1 transaction per 5 years

Today, lots of transactions are coming from mining pool payouts, that frequency could be reduced if a consensus is reached

A possible future: Just like multimedia filled today's high speed network with lots of digital trash, the increase of bitcoin block size might push another round of network and storage upgrading world wide, it will create lots of jobs  

No, Bitcoin is a consensus system. If A, B, C, D, E, F, G are telling you that the chain is invalid because miners have started inflating it— but no one else is taking action because no one else is running a validating node, no one else is bothering to consult with A, B, C, D, E, F, G or believes them— well then, tough luck for you. You can stay on your true Bitcoin all alone with A..G, while almost everyone else ventures off on inflatacoin. There is currently no alarm bell in the system, and currently no way to give an alarm bell compact proof so that it could be acted on automatically. (Though it's possible to change things so that they can be done, and I think doing so should be a necessary precondition for anything with substantial risk of hurting decentralization)

Besides, why should you believe A..G for the security of _all_ your coins? You missed the point I was making that one shot securing single transactions isn't the same as securing everything. Maybe on one transaction you're secured by A..G on another you're secured by E..H, and yet another avoids the cost of arbitrators by being performed in a trust-less way, and meanwhile you have a stash of coins sitting safely without risk of losing their value. The diversity spreads your risks and minimizes chances of total loss.

They don't have to trust me, they can check for themselves or pay multiple other people they marginally trust to check for them. Hm? Selling data and exchanging digital assets aren't uncommon cases. Doing them in more secure ways certainly is, but here you were telling me it was impossible to be trustfree. It will take time for people to adapt to what Bitcoin makes possible (and from the constant stream of theft and betrayal in Bitcoin space making them realize they need to look into alternatives).

Replacing the blockchain has an entirely different risk/reward payoff than inflating the currency. You generally don't see central banks breaking into people's vaults and taking money these days…  And as I mentioned, extensive explicit collusion isn't required. Remove the maximum subsidy check ... advertise that you've done so in the coinbase. Produce bigger blocks when a supermajority indicates they have. SPV clients will happily follow along.

Classical currencies are based on trust, and their trust argument is _far_ stronger than what the bitcoin community can offer: regulated instutions, the enforcement of laws backed up by firepower, millions of people who can absorb and correct problems, experts with mile long lists of credentials. Etc. And yet many hold the view that the trusted guardians of the popular currencies are violating this trust and will continue to do so. When you look at the Bitcoin ecosystem— it's hard to find arguments for well placed trust even a fraction as convincing as the trust thats failing in the wider world.

Complete trustlessness probably can't be achieved— with a wide enough definition, e.g. I haven't personally proven all of mathematics to myself or inspected my cpu with a home built electron microscope, but it can be approximated apparently arbitrarily close.

But first, there is a big distinction between trusting in the a world wide currency and trusting in a single trade of some small value.  However trust dependent Bitcoin itself is no set of transactions denominated in BTC can be less trust dependent than the underlying Bitcoin.  If you make ten small trades what is the probability that _all_ ten screw you?  It is something like the probability of an individual failure raised to the tenth power (e.g. probably negligible) plus the probability that Bitcoin screws you— The currency is a single point of failure, encompassing all the money in its economy, so it must be strong if anything is to be strong.

Secondly, this is in fact not the case: In situations where people are buying and selling digital data or services which occur within the context of machine verifiable processes it's sometimes possible to trade with absolutely no risk at all. E.g. CoinSwap can let you trade Bitcoins and foocoins in a cheat proof way (so long as Bitcoin and Foocoin are secure), or e.g. this protocol can allow selling knoweldge in a bidirectionally cheat proof way, in both cases without any third parties at all.

Even when a cheat-proof protocol isn't possible, as is the case for physical goods, you're not stuck with "a" trusted third party. In Bitcoin you can construct blockchain escrows with release rules like "(A and B) or ((A or B) and any-3-of(C, D, E, F, G))".   How likely is a single arbitrator to screw you?  Fine, add more until the potential for dishonesty is negligible, if what you're doing justifies it.  But this only works if the system itself isn't imposing its own trusted parties on you.

These are all why Bitcoin has script to begin with— simply making transactions isn't enough to achieve trustlessness. Powerful transactions are required to achieve Bitcoin's goals.

Then there is the issue of betrayal by 1000 cuts, by expedience, and with all honest intentions. In classically human mediated systems there is a constant pressure to bend the rules a little bit: "Read this guys email, he's no good", "block this transaction, just this one, the funds are stolen. I swear!", "just print a bit more money, we need to fund a war, it's important.". We find it hard to say no to small compromises, even when the logical outcome, shown by history to be inevitable, is no mystery to us.  Improved trustlessness is improved robustness to small betrayals adding up to big ones.

WRT SPV, it's only secure so long as there is an {adequate} supply of fully verifying nodes to keep miners honest. If there isn't then miners as a group (consider: even without an explicit conspiracy, they have some highly correlated interests…) can freely deceive SPV nodes. If there is, then for many use-cases SPV security is quite close to a full node (e.g. accepting confirmed payments with values small relative to the generated coins confirming them).  SPV is what makes compact mobile devices viable, and it's also necessary for efficiently binding Bitcoin into other system. But its security absolutely depends on validation being very well distributed.

True but SPV don't require a trusted third party.  SPV perform the same validation of txs but rather than performing them on the entire blockchain they just perform them on the subset of the blockchain that the node is interested in.

Full node = I will keep a copy of entire blockchain and validate all txs even those which I have no personal connect to.
SPV node = I will keep a copy of block headers and request in realtime subsets of blockchain I need to verify my own txs.

Both perform the same sets of validations the difference is the scope.  Also obviously the SPV node concept only works if the are some full nodes.  Someone (preferably a lot of independent someones) needs to maintain a full copy of the blockchain that can be provided in realtime to SPVs as needed.

No there is no merit.  It is completely unworkable.   Nodes share txs, and the txs known between any two arbitrary nodes is generally similar with maybe a few unique txs.  However a node has no idea what particularly subset of the memory pool an arbitrary miner will include in the next block.  A node doesn't even know what txs are in the memory pool of an arbitrary node, nor can they know for certain that they are aware of all txs in known to all nodes.  Also mining nodes don't announce themselves as mining nodes, and shouldn't for a variety of security reasons.  For the purpose of message relaying (blocks, txs, etc are all simply messages) they are no different than any other peer on the network.  So every tx set of every active miner would need to be relayed to every peer on the network to ensure they reach every miner.

Even if such a scheme was workable given that using tx hashes as a proxy for full txs already reduces the size of a block message by 90%+ that is an incredibly increase in internode bandwidth to reduce the already reduced block message by a small amount.


The ordering of tx can easily (with a hard fork) be made deterministic simply sort tx by tx id hash.  Tada.  That is a non-issue however no node can know what tx are known by any other node.   So while the order can be deterministic the entirity of the memory pool is not.

Example node 1 knows of txs
B, C, D, E and we have put them in deterministic order.

However node 2 knows of txs
A, B, C, F and they also are in deterministic order.

However it is clear the two nodes don't share the exact same set.

Still even for two nodes which share the same set a miner will chose some unknown (or only known to the miner) subset of the memory pool for inclusion in the next block.   So given a set of n txs sequenced by tx id how would a miner identity the tx set for a block without using tx ids?

Nodes should already have the txs which is why I indicated the block message can be reduced to a list of hashes for the txs included in the block and that reduces the block message by 90%+.   Even if you assumed that no miner would need to change their tx set per block do you really see a system where each miner pre-broadcasts their exact tx set in advance is going to scale?  That would mean every node receives one tx set from every active miner before every block.   We should be advocating as many independent miners as possible.  So if there are 100 miners then each node on the network needs to receives, store, and relay 100 tx sets for every block?  If there are someday 10,000 miners it is 10,000 tx sets for every block? 

There is a difference between counterparty trust and a trusted third party.  You may have to trust your counterparty but Bitcoin wasn't desolved to remove that trust element, Bitcoin was designed to remove the need for a trusted third party.


Alice <-----> Bob
Alice and/or Bob need to trust each other but they don't need to trust anyone else.

Alice <------ Bank ------> Bob
Both Alice and Bob can't simply trust each other they now must trust the "trusted" third party. 


Satoshi talks about removing the trusted third party in the introduction of the whitepaper. 
http://bitcoin.org/bitcoin.pdf

If there are changes in the pipeline which require a hard fork anyway, might as well fix that problem too by increasing the size of the nonce field.

The vast majority (as in ~100%) of blocks are never solved.  A particular blockheader only has ~4 billion possible nonces which even a low end rig runs through in a fraction of a second.  Difficulty is currently 700 million which means for every solved block roughly 700 million block headers are constructed and no solution is found.  

Visa is an erroneous comparison. Bitcoin is a currency with a payment network. Visa is a payment network. I expect that someday people will use Visa to pay people with Bitcoins (though thats not a way of using Bitcoins that I'd prefer).

I don't appreciate the partisan "where you stand" language, which appears to be asking me to draw a bright line.  This is a complicated matter engineering and it involves nuanced trade-offs and understanding. Not bright line politics.

I support the vision of Bitcoin as I originally learned about it, which is a trust-less decentralized system intended to make it easy for people to transact without interference or mediation by trusted parties. Bitcoin is a system where we minimize the interaction of failure prone men in making decisions— with the best of intentions— which are pushed on others without their consent— it's a system made somewhat immune to even majority whim by the widely distributed hard enforcement of system invariants which are purposefully difficult to change, especially if the changes are controversial.

To the extent that trustlessness and scaling aren't in conflict, I say— and I think everyone with honest intentions would agree: great! lets have all the things!  I believe that as technology improves the level of conflict free scaling improvements will increase.

I am, however, also concerned that even at today's scale we have operating cost and education problems which are endangering the system's security assumptions. I believe these are perfectly solvable problems.

Generally, to the extent that trustlessness and scaling come into direct conflict I believe that Bitcoin should tend to prefer trustlessness. There are a couple of reasons I believe this:

Scaling with trustlessness should improve in time just from the march of technological progress. Trustlessness seems harder to regain (e.g. p2pool didn't replace all the centralized pools, though I believe if it had come first we'd probably have no centeralized pools today or at least they'd be substantially different) than scaling is to add.

Scaling can always be achieved through alternative payment mechanisms, but I know of no similar way to improve trustlessness through overlay systems. Alternative payment networks for Bitcoin must exist in order to support things like instantaneous (semi-)irreversibility, improved privacy, offline payments, etc. Bitcoin will never be as scalable as a less decenteralized payment system. E.g. If we had to choose between Bitcoin having a dozen miner supernodes that do all the validation which we effectively must trust, or having to run low value bitcoin transactions through distributed 5-party chaum banks, which exist by the hundreds around the world... I'd prefer the latter. This is doubly true because if Bitcoin is too bloated it will harm diversity in these alternative payment systems.

Finally, trustlessness should be preferred because it is the unique value that Bitcoin (and its clones) provide. Visa does the scaling game better than we do, for fundamental reasons. Many prior ecash systems with ample funding and brilliant technology failed, largely because their trusted center provided a chokepoint.

But all thats assuming that there is a conflict and I think our job in Bitcoin is to advance the technology and minimize the conflict, foremost.  And then we don't have to answer the hard questions like who picks who wins and who loses, it's better to have everyone win. Past that, it's a balancing act— one where either extreme diminishes Bitcoin's value to the world.

I wasn't trying to be extreme but I did mean what I said.  The current transaction rate appears to be below 1 TPS, so obviously that needs to go much higher.  I'm not sure what figure people quote for Visa but it's in the thousands.  My fear is that if bitcoin gets stuck at 1 TPS, then there isn't a future for it.

If the problem isn't related to orphan costs, and it simply due to miners running defaults, then that's great and I look forward to seeing block sizes go up as the new releases come out.

It's not exactly clear to me where you stand, gmaxwell - is there some approximate TPS figure that you imagine might be possible in the future?  Is there a TPS limit that you would deem to be unacceptable for bitcoin success?  How long do you see 1 TPS being adequate before transactions are unaffordable, given the growth in transactions and bitcoin price?

I know you are an influential developer so it would be nice to know where you stand on this.

Thanks

Time Warner Cable has been pretty good about that so far. OTOH I've been paying for the highest-tier package and not obviously using it for Bittorrent.

Several other countries have it even better.

That is an incorrect assumption.  The point of replacing full tx with tx hashes in the block message is that you will already have received those txs.  So while it reduces the bandwidth requirements for timely relaying of the block message it doesn't reduce the overall bandwidth requirements.  Also remember Bitcoin is a p2p network so for the network to be robust each peer needs to have multiple connections to other peers.  While a node can get by with only eight connections to reduce the risk of an isolation attack a healthy connection would be far more.

So if we consider that there are 144 GB of blocks and you are connected to 16 peers.  Either they will relay to you or you will end up relaying to them.  That means 144 GB daily of synchronous bandwidth for tx messages plus the reduced overhead of block message is still say another 10% so maybe 158 GB daily or 4,700 GB monthly.   While in theory a 15 Mbps connection could handle that, try to find an ISP which won't cap, throttle, or simply bad you for using 5 TB per month.

Of course even this example excludes the additional load on the average node in order to accommodate bootstrapping new nodes, or nodes which are out of sync.  The network would be very very unhealthy IMHO at even 10% of that on residential connections.

A transaction rate which produces 144 1 GB blocks per day is achievable on a home internet connection which I can get for less than $100/month, assuming transaction messages are suitably optimized as mentioned by D&T.

By the time the network is processing that many transactions, the cost of such a connection will be even lower.

The answer is to optimize block propagation into something that resembles efficient use of bandwidth so that we don't have to fear success.

If Bitcoin blocks become gigabytes in size such that only powerful entities bother running validation nodes, then I think Bitcoin will collapse since it loses its main feature that is appealing (truselessness).

There is careful balancing which is required and hysteria ("biggest threat", "MUCH higher") is not helpful.

At either extreme— where it's too costly to transact, or where it's too costly to validate Bitcoin loses value.  If Bitcoin becomes defacto centralized due to being too costly to validate then it will be _impossible_ to build anything more decentralized on top of Bitcoin, and this is arguably a greater risk both because it can gradually end up in that state and because the risk of it being too costly to transact can at least be answered by alternative payment networks (which are mandatory in any case to achieve things like strong privacy, instant payments, efficient very low value transactions, and offline transaction).

It is an interesting idea.  It essentially eliminates any "orphan cost" due to bandwidth related propogation delays.  As mentioned some complex tx can be computationally intensive but bandwidth & latency are likely are more critical resources.  Something would probably need to be done to handle significantly older blocks in a more efficient manner.  For example Bitcoin blockchain is ~270,000 blocks, if the padding was enforced for all block messages to bootstrap a new node would require ~270 GB, compared to ~12 GB actual so older blocks need to be handled without padding.

Still it is an interesting concept and if not used in Bitcoin it could be refined and used in altcoins.  This could be combined with another improvement, replacing tx with tx hashes in block message, to generate better results at reduced cost.

Block message format
magic # (4 bytes)
block size (4 bytes)
blockheader (80 bytes)
tx count (1 to 9 bytes)
transaction list (variable - average 600 bytes ea)

Proposed tx less block message format
magic # (4 bytes)
block size (4 bytes)
blockheader (80 bytes)
tx count (1 to 9 bytes)
transaction hash list (variable - 32 bytes ea)

The average tx size appears to be ~600 bytes, tx hash is 32 bytes so this reduces the new block message by >90%.  Now for bootstrapping nodes the first format is more useful but for up to date nodes they should have block txs in their memory pool.  It is also in a miners best interest to ensure their peers have all those tx to reduce the propogation time.  If a node is missing some tx they can simply request those tx by tx id (hash) from peers.  This method can be used by itself to reduce the orphan cost by itself or it could be combined with the concept in the OP to both reduce the bandwidth requirement and eliminate the additional tx orphan cost. 

One needs to consider how much padding would be required as there is no fixed relationship between size of all txs in a block and the size of all tx hashes in a block.  Since the goal is just uniformity some lower bound could be considered.   An average tx size of <300 bytes is dubious (would require a block of all single input, single output txs) so in additional to max tx size a max tx count limit could be enforced.  If the limit was 1MB and 3,000 tx this would make the max block message size 94 KB (4+4+80+9+3000*32).  Block message could be padding enforced to 100 KB (for current block size limit).

Block-txless message format
magic # (4 bytes)
block size (4 bytes)
blockheader (80 bytes)
tx count (1 to 9 bytes)
transaction hash list (variable - 32 bytes ea)
Randomized padding (100KB - rest of message size)

How would this work in practice.  Well lets look at a recent large block:
https://blockchain.info/block-index/444118/0000000000000002166ff15ec0ce6427f21c2f7ce55676d280b0677fe04c1f2a
Tx Count: 1368
Size with current block message: 882 KB
Size if padded to 1MB: 1024 KB (16% overhead compared to current)
Size using txless block message: 43KB
Size using padded txless block message: 100KB (89% reduction compared to current)


Pros:
For blocks over 100KB the new format even with padding is still smaller than the "full block" format without padding so there is no bandwidth costs to miners decrease.
Simplifies tx inclusion economics.
Can be implemented as a non forking change by creating a second new block message type and legacy nodes can still request block in older "full" format.
Doesn't increase on disk storage requirements.

Cons:
More complex broad propagation logic, older blocks need to be handled in legacy manner to avoid massively increasing bootstrap requirements.  
Hard enforcement at protocol level requires a hard fork (which may not be possible to obtain the consensus needed).
With a non-forking expansion, enforcement is only at the client level.  Miners could locate other miners and communicate directly in more efficient block message (i.e. block hash only with no padding) to gain advantage on other miners.

Indeed. I already suggested this a few days ago (see pull reqs). However 0.8.6 is going to go out with a relatively small bump of 50 or 100kb in the default size, to see how many miners follow.

Some are, some aren't.