Topic: Once again, what about the scalability issue? - page 6. (Read 11259 times)

I never said the market wouldn't take care of it, just that you might not like the outcome.

One outcome would be block sizes becomes so large that pools >51% of hashing power run private links to each other in the same datacenter in order to keep orphans out of line.  That drives other pools out of business, the major 3-4 or pools/solo-corps grow even larger and decide to optimization of the network to simply exclude the blocks of any pool/solo not in their organization.  That is one way of the market "taking care" of the problem that demand for higher tx volume exceeds the capabilities of public peer to peer links.  The market doesn't necessarily care about the advantages of decentralization, transparency, fair play, and the "solution" arrived may not be one that most bitcoiners find desirable.  It isn't a prediction, it is a risk that is all I am saying.  As long as avg block size is low relative to bandwidth capacity between miners on open public peer to peer links there is no catalyst for such a move, there is no problem for the market to "solve".

Considering I am part of this market and am building these kinds of solutions, your claim is 100% right on about the market sorting it out.

What makes you think the market won't take care of the average miner, such as by limiting blocksize normatively? Any claim that the market won't take care of something should be justified by specifying how you think the market is broken for that function - because the norm is for markets to work, even if we can't immediately see how.

The network effect has to be balanced with the centralization effect.  What you want is a single currency (for pricing purposes) that scales across many banks (chains) for decentralization purposes.   

You would end up with a situation where transacting within a single chain is almost free (like transacting with a single bank) but transacting between different chains is more expensive (like a wire transfer).    If you want to send someone money you have to know both their bank and account number.   

Of course, because private keys are good on all chains you can send them money on your chain and leave it up to them to move the funds to their normal chain.   Large centralized wallets would be able to integrate all of the chains into one 'account' to give the appearance of a single large chain while still allowing individual users to with 1 Mbit internet connections to participate.

Assuming 10,000 trx / sec at 1024 bytes / trx  it would require an 80 Mbit connection to handle all of the transaction traffic which means that 256 chains could probably handle the entire transaction volume of VISA / Master Card / PayPal and all wire transfers combined, and yet you would only have one chain per state on average.    Of course, a lot of this transaction volume will probably still flow through trusted 3rd parties that enable 'instant' transfers rather than 3+ confirmation transfers.

Assume there exists a demand for cryptocurrency-demoninated transactions. This demand will require a certain amount of bandwidth to satisfy.

Suppose the demand is high enough that the entire cryptocurrency ecosystem requires 10 Gbit average bandwidth.

How much does it matter if this 10 Gbit/sec global transaction demand is satisfied by 100 cryptocurrencies or 1 cryptocurrency?

Other factors to consider:

Would the average person prefer to manage a balance of 100 different cryptocurrencies, or would they prefer to hold their savings in a single currency that works everywhere? If you're having trouble figuring this one out, consider whether the average Internet user prefers to have a single global networking standard that makes all resources accessible from any ISP, or if they would prefer to go back to the 1990s walled garden days of AOL, Genie, Compuserve, and other non-interoperable services.

What does the n² scaling property of the network effect imply for the value of a single network that can handle all 10 Gbit/sec of transactions itself vs 100 networks that can handle 100 mbit/sec each?

this is the same solution i came up with when i first started thinking about this issue.

Bandwidth is more critical than disk space for decentralization. 

Parallel chains of with a fixed limit on bandwidth per chain would be nice.

The ability to move value between chains would also be nice.

To move value between chains means a 3rd chain is required that is merge-mined with the other 2 chains.  This 3rd chain is responsible for confirming movements from one chain to the other and vice versa.   The cross-chain would have to be very low bandwidth, say 1% of bandwidth allocated for the main chains. 

With this approach you could have decentralization while still having only one crypto-currency.

good information thanks.

do you think gaven has the leverage/influance/power to remove the block size limit. i dont think he does.

Yes however the risk is centralization.  It isn't that the network "couldn't" handle unlimited blocks it is that we might not like the consequence of unlimited blocks. As the block sizes gets larger and larger it becomes more difficult to keep orphans to a manageable level.  Orphans directly affect profitability and for pools it is a double hit.  High orphans mean less gross revenue per unit of hashing power but it also means the pool is less competitive so miners move to another pool.  The pools revenue per unit of hashing power is reduced and the overall hashing power is reduced too.  So pools have a very large incentive to manage orphans.    Now it is important to remember it doesn't matter how long it takes for ALL nodes to get your block just how long it takes for a majority of miners to get your block.  The average connect between major miners is what matters.  

If the average connection can handle the average block then it is a non-issue.  However imagine if it can't, and orphan rates go up across the board.  Pools are incentive to reduce orphans so imagine if x pools/major solo miners (enough to make up say 60% of total hashing power) moved all their servers to the same datacenter (or for redundancy the same mirrored sets of datacenters around the world).  Those pools would have essentially unlimited free bandwidth at line speed (i.e. 1Gbps for cheap and 10Gbps for reaosnable cost).  The communication between pools wouldn't be over the open (slow) internet but near instaneous on a private network with boatloads of excessive bandwidth.  This is very simple to accomplish if the miners are in the same datacenter.  The miners just share one LAN connection on a private switch to communicate directly.   Now for these pools a 40MB, 400MB, or even 4000MB block is a non issue.  They can relay it to each other, verify, and start on the next block in a fraction of a second.  Near 0% orphan rates and reduced bandwidth costs.  For other miners however the burden of these large blocks means very high oprhan rates.  How long do you think it will take before CPPs abandon their pool with 5% orphan rates for ones with near zero.  That isn't good for decentralization of the network.

I don't want to sound doomsday but this why BANDWIDTH (not stupid disk space) is the critical resource and one which requires some careful thought when raising the block limit. It is important that average block size not exceed what a miner on an average connection can broadcast to peers in a reasonable amount of time (3 seconds = ~0.5% orphan rate) on an average public internet connection.  Granted those are pretty vague terms.  Obviously 1MB block is below that critical level and 1GB block is obviously above it.  Is 100MB fine today? How about in 10 years? Is 5MB fine today? How about 2MB and doubling every two years? I am not saying I have the answers but that is the kind of thing we (community at large not just Gavin et all) need to think about critically before saying "yeah lets go to unlimited blocks and let the market figure it out".  I have no doubt the market will figure it out however one might not like what end state it reaches.

The good news is bandwidth is still increasing rapidly and the cost per unit of data is falling just as rapidly.  This is true both at the last mile (residential connection) and in datacenters.  So it is an problem which is manageable as long as average block size doesn't eclipse that growth.


Don't take any of this as "set in stone" it is more like when they ask you "how many windows are there in New York City?" in an interview.  Nobody cares what the exact number, what the interviewer is looking for is what logic will you use to come up with an answer.  If someone thinks my logic is flawed (and it certainly might be) well that is fine and I would love to hear it.  If someone can convince me otherwise that is even better.  However show me some contrary logic. If the counterargument is merely "unlimited or it is censorship" well that doesn't really get us anywhere.

There are four different bandwidth bottlenecks.

Miners
Miners are somewhat unique in that they have to broadcast a block very quickly (say 3 second or less target) to avoid excessive orphans and the loss of revenue that comes with it.  This means their bandwidth requirements are "peaky".  That can be smoothed out somewhat with protocol optimizations however I would expect to see miners run into a problem first.  The good news is it is not that difficult for pools or even solo-miners to setup their bitcoind node in a datacenter where bandwidth is more available and at lower cost.

Non mining full nodes
Full nodes don't need to receive blocks within seconds.  The positive is that as long as they receive them in a reasonable amount of time they can function.  The negative is that unless we start moving to split wallets these nodes are likely on residential connections which have limited upstream bandwidth.  A split wallet is where you have a private bitcoind running in a datacenter and your local wallet has no knowledge of the bitcoind network, it just communicates securely with your private bitcoind.  An example of this today would be electrum client connecting to your own private secure electrum server.

Bootstrapping nodes
Another issue to consider is that if full nodes are close to peak utilization you can't bootstrap new nodes.  Imagine a user has a 10 Mbps connection but the transaction volume is 9 Mbps.  The blockchain is growing at 9Mbps per second so the user is only "gaining" on the end of chain at 1 Mbps.  If the blockchain is say 30 GB it will take not ~7 hours (10Mbps) but 70 hours to catch up.
The good news here is there is some slack because most residential connections have more downstream bandwidth then upstream bandwidth and for synced nodes the upstream bandwidth is the critical resource.  

SPV nodes
The bandwidth requirements for SPV nodes are negligible and unlikely to be an issue which is a good thing however SPV don't provide for network security.  While SPV are important so casual users are not hit with the rising costs of running a full node at the same time we want to ensure that the ability to run a full node for enthusiasts remains an realistic option.  Maybe not everyone can run a full node but it shouldn't be out the reach of a the majority of potential users (i.e. requires 200Mbps symetric low latency connectivity, 20TB of storage, enterprise grade RAID controller, 64GB or RAM, and quad xeon processors).  How many full nodes are needed.  Well more is always better, there is no scenario where more hurts us in any way so it is more likely how few can we "get away with" and stay above that number.  If 100 enough?  Probably not.  1,000? 10,000? 100,000?  10% of users, 1% of users?  I don't have the answer it is just something to think about.  Higher requirements for full nodes means less full nodes but more on chain trust-less transaction volume.  It is a tradeoff, a compromise and there is no perfect answer.  1GB blocks are non-optimal in that they favor volume over decentralization too much.  Staying at 1MB blocks forever is non-optimal in that it favors decentralization over volume too much.  The optimal point is probably somewhere in the middle and the middle will move with technology.  We don't need to get the exact optimal point, there likely is a large range which "works" the goal should be to keep it down the middle of the lane.

so even with an unlimited block size there would still be a market for transaction inclusion in blocks since miners who attempted to relay a block that was too large would find it orphaned. that's important for network security.

also correct me if im wrong but individual miners wouldnt even need a 100mb connection would they? just the pools.

100 tps is way plenty, even if we assume the load of all credit card companies combined 100 tps would be enough to allow anyone who wanted to do an on-chain transaction to be able to afford it (excluding micro transactions but who cares about that). which is all that matters, we dont need a system where every transaction avoids all counter party risk, what we need is a system where avoiding counter party risk is affordable. 100tps would provide that.

this post put my mind at ease. i mean im already pretty significantly invested in bitcoin because even if there was no solution the the scalability problem bitcoin would still have great utility, its nice to know however that there are solutions.

Thx, I'll learn about it to make sure I'm not supposed to trust it to be able to use it.

Multibit.  It's being promoted as the default now by bitcoin.org.  Or so it seems to me via placement on the web page (and statements on the 'sticky' thread which my very much on-topic post was deleted from since it was not good marketing material apparently.)

  http://bitcoin.org/en/choose-your-wallet

To Multibit's credit, the strings 'peer' or 'p2p' to not appear obviously anywhere on their site though they still feature front-and-center on bitcoin.org.  Again, it seems pretty scammy to me.

https://multibit.org/

It is linked to and recommended from bitcoin.org

Can you show me where Satoshi said this?

I thought there wasn't any. Which one would u recommend?

I was trying to be a bit facetious in use terms like 'unequal peers' and '[server|client]-variety-peers'.  Certain of the more technical here might appreciate it, and I suspect that you are among them.

Anyway, I'm glad I err'd on the side of brevity (this time) and allowed you to make the point that the the solution we are migrating towards looks an awful lot like what we see in ACH.  How long it takes to get there (if ever) will I suspect be dictated mainly by the transaction per unit time growth.

I also suspect that you may not be thrilled about this evolution, but very well may see it as a necessary evil.  If so, I respectfully dis-agree.  In my mind it makes the solution not much good for much of anything, and that is particularly the case in light of the Snowden revelations (or 'confirmations' to some of us.)

---

Again though, I find it to be scammy and offensive to prominently label the system 'peer-2-peer' as long as there is a liklihood that it's going SPV, and changing the default recommendation to Multibit is ample evidence that that is exactly the path chosen by those calling the shots.  The main things Bitcoin has going for it are that it is 'first' and it is 'open-source'.  It is honest and appropriate to dwell on those things because they happen to be true.

I think it is more simple than that.  If you aren't a full node you aren't a peer. Period.  All peers are equal but not all users are peers.  One way to look at it is inter bank networks are a form of peer to peer networking (where access to the network is limited to a selected few trusted peers).  If you send an ACH or Bank Wire you are using a peer to peer network but YOU aren't one of the peers.  The sending and receiving banks (and any interim banks) are the peers. 

I think a similar thing will happen with Bitcoin with one exception.   It doesn't matter what computing power you have available or are willing to acquire.  The banking p2p network is a good ole boys club, peons not invited.  With Bitcoin you at least have the CHOICE of being a peer.  In the long run (and this would apply to other crypto-currencies as well) a large number, possibly a super majority of users will not be peers.  They are willing to accept the tradeoff of reduced security for convince act become a user not a peer of the network.

TL/DR:
No such thing as less than equal peers, you are either a peer or your aren't.  In Bitcoin v0.1 100% of nodes were peers today some large x% are in time that x% will shrink.  Peers are still peer but not everyone will want or need to be a peer.  There is a real cost of being a peer and that cost (regardless of scalability improvements) is likely to rise over time.


I don't see any support for that claim.  On the contrary ...
https://bitcointalksearch.org/topic/block-chain-sizestorage-and-slow-downloads-for-new-users-252937

It is a non-trivial issue.  For complete security we want a large number of independent nodes maintaining a full historical copy of the blockchain.  It doesn't need to be every node but enough that there retains a decentralized hard to corrupt consensus of the canonical history of transactions.  There is a real risk in a jump to a pruned db model that information is lost or overly centralized.  It doesn't mean the problem is unsolvable however it is better to err on the side of caution. 

Heh.  I'm still waiting for the bitcoin project to get honest and state that not all 'peers' are 'equal peers' in the 'p2p' network.  Somehow it seems not to be a priority.  Funny that.

It also would not hurt (from a perspective of truth in advertising) to stipulate that Bitcoin is 'eventually-deflationary', non-scalable, far from anonymous, and the fluff about blockchain pruning was either marketing BS or is de-prioritized (one suspects in order to assist in the formation of 'server-variety-peers' and shifting of non-commercial entities into the 'client-variety-peer' category.

My guess is a lot depends on how much Bitcoin grows and how quickly.  Also bandwidth is less of an issue unless the developers decide to go to an unlimited block size in the near future.  Even a 5MB block cap would be fairly manageable. 

With the protocol now lets assume a well connected miner needs to transfer a block to peers in 3 seconds to remain competitive.  Say the average miner (with node on a hosted server) has 100 Mbps upload bandwidth and needs to send the block to 20 peers. (100 * 3) / (8 * 20) = 1.875 MB so we probably are fine "as is" up to a 2MB.  With avg tx being 250 bytes that carries us through to 10 to 15 tps (2*1024^2 / 250)

PayPal is roughly 100 tps and using bandwidth in the current inefficient manner would require an excessive amount of bandwidth.  Currently miners broadcast the transactions as part of the block but it isn't necessary, as it is likely peers already have the transaction.  Miners can increase the hit rate by broadcasting tx in the block to peers while the tx is being worked on).  If a peer already knows of the tx then for a block they just need the header (trivial bandwidth) and the list of transaction hashes.  A soft fork to the protocol could be made which allows the broadcasting of just header and tx hash list. If we assume the average tx is 250 bytes and the hash is 32 bytes this means a >80% reduction in bandwidth required during the block transmission window (assumed 3 seconds to remain competitive without excessive orphans).  

Note this doesn't eliminate the bandwidth necessary to relay tx but it makes more efficient use of bandwidth.  Rather than a giant spike in required bandwidth for 3-5 seconds every 600 sec and underutilized bandwidth the other 595 seconds it would even out the spikes getting more accomplished without higher latency.  At 100 tps a block would on average have 60,000 tx.  At 32 bytes each broadcast over 3 seconds to 20 peers would require ~100Mbps.  An almost 8x improvement in miner throughput without increasing latency or peak bandwidth.

For existing non-mining nodes it would be trivial to keep up.  Lets assume the average node relays a tx to 4 of their 8 peers. Nodes could use improved relay logic to check if a peer needs a block before relaying.   To keep up a node just needs to handle the tps plus the overhead of blocks without falling behind (i.e. one 60,000 block in 600 seconds).  Even with only 1Mbps upload it should be possible to keep up [ (100)*(250+32)*(*(4) / 1024^2 < 1.0 ].

Now bootstrapping new nodes is a greater challenge.  The block headers are trivial (~4 MB per year) but it all depends on how big blocks are and how far back non-archive nodes will wan't/need to go.  The higher the tps relative to average node's upload bandwidth the longer it will take to boot strap a node to a given depth.

ya i already knew all that . i was just wondering how you thought the bandwidth bottleneck problem would be dealt with.