Pages:
Author

Topic: A rolling root to solve Bitcoin's scalability problem? - page 3. (Read 7368 times)

legendary
Activity: 3472
Merit: 4801
One of the more talented trolls I've seen in a while.  I actually thought he was trying to learn and understand at first.  It took almost 24 hours before I realized that it was a ruse to bait knowledgeable people into boosting his ego.  Looks like my kill file is getting another entry.
staff
Activity: 4284
Merit: 8808
Except your own Wiki says the opposite:
The wiki is a public document that anyone can edit. The scalability page is almost entirely the work of a single person who has removed views conflicting with his own onto other pages (including my own, for that matter).  You should be a sophisticated consumer of wiki content and not believe everything you read.
Quote
Quote
I think you should stop using the word trustless while you have multiple bitcoin developers telling you that you do not understand it.
The way my mind works is that I do not accept BS from anyone. It doesn't matter how large an angry mob gets if what it is saying is complete nonsense.
I've addressed every objection to this proposal, and have received confirmation on it now from two people who seem to know something about Bitcoin: Realpra, and a friend of mine whom I know is intelligent (works on SocialVPN).
You haven't really addressed _any_ of the objections here, because apparently you do not understand them. I spent quite a bit of time on IRC trying to help you understand but I have failed you. I'm sorry that I'm unable to help further.
newbie
Activity: 58
Merit: 0
Quote
The fact is that ancient blocks, at any given point in time, are already highly trusted by the network. If they weren't, Bitcoin would simply not work.
There is already "trust" in the Bitcoin network, for these ancient block
Every node verifies the history, this is how you can be sure that claims like "satoshi premined a billion bitcoins and is secretly waiting to spend them" cannot be true. We do not trust, and cannot trust— because it would be trust founded on nothing.

Except your own Wiki says the opposite:

Quote
Only a small number of archival nodes need to store the full chain going back to the genesis block. These nodes can be used to bootstrap new fully validating nodes from scratch but are otherwise unnecessary.

From: https://en.bitcoin.it/wiki/Scalability#Storage

Notice how nowhere in that section does it say anything close to "Every node verifies the history.", and instead refers to "archival nodes". It also says "Only a small number" of them are needed.


Quote
I think you should stop using the word trustless while you have multiple bitcoin developers telling you that you do not understand it.

The way my mind works is that I do not accept BS from anyone. It doesn't matter how large an angry mob gets if what it is saying is complete nonsense.

I've addressed every objection to this proposal, and have received confirmation on it now from two people who seem to know something about Bitcoin: Realpra, and a friend of mine whom I know is intelligent (works on SocialVPN). And while not a vote of recognition, one of the Bitcoin is Vulnerable authors didn't say that the idea wouldn't work when I shared it with him, but said "i'd feel nostalgic if i had to part with the genesis block." Tongue

Edit: I called it 99.9% trustless, not 100%.
staff
Activity: 4284
Merit: 8808
Quote
The fact is that ancient blocks, at any given point in time, are already highly trusted by the network. If they weren't, Bitcoin would simply not work.
There is already "trust" in the Bitcoin network, for these ancient block
Every node verifies the history, this is how you can be sure that claims like "satoshi premined a billion bitcoins and is secretly waiting to spend them" cannot be true. We do not trust, and cannot trust— because it would be trust founded on nothing. Without verification there is no reason for unidentifiable, anonymous parties not to lie in favor of their own common interests. What actually makes the Bitcoin state trust worthy is because people verify it instead of trusting it.

I think you should stop using the word trustless while you have multiple bitcoin developers telling you that you do not understand it.
newbie
Activity: 58
Merit: 0
Quote
1:31:31 PM sipa: sugarpuff: now, about dealing with ways to improve performance, IF you are giving up zero trust

I just want to point this out, to make sure that no one overlooked it.

Lots of problems get really, really easy once you give up security.

That is nonsense. Just a false characterization of what is being proposed.

Security is not being given up.

Quote
Trusted systems have wildly different properties than trustless systems.  If you don't mind moving from a trustless to a trusted system, then what you are proposing can work*.

And people on the internet often make "wildly" thoughtless comments, like this one here.

This is not a black and white proposal like you are trying to make it out to be.

It is not a suggestion in the slightest to "give up security."

The fact is that ancient blocks, at any given point in time, are already highly trusted by the network. If they weren't, Bitcoin would simply not work.

There is already "trust" in the Bitcoin network, for these ancient blocks.

For established nodes, there would be zero difference.

And for brand new nodes:

1. They would actually exist.

If the "doomsday scenario" pans out, and 14TB is needed every 3 months, there would be virtually no new nodes. Your trustless system would be mostly useless to most people, including, probably, you (unless you have a datacenter in your back yard).

2. It would be not entirely trustless, but "approximately trustless" (like 99.9% trustless, once the entire window has been consumed). Read the rest of that IRC chat. New full-nodes would have security greater than the already very-secure light clients that we have today like Electrum.

3. There would be nothing stopping those nodes, if they really want, from downloading an archive of the no-longer-useful data from the discarded blockchain tail.

Quote
Even the posts telling you it won't work are recycled.

So far not a single post has shown that this "won't work". Every objection was answered.
kjj
legendary
Activity: 1302
Merit: 1026
Quote
1:31:31 PM sipa: sugarpuff: now, about dealing with ways to improve performance, IF you are giving up zero trust

I just want to point this out, to make sure that no one overlooked it.

Lots of problems get really, really easy once you give up security.  Trusted systems have wildly different properties than trustless systems.  If you don't mind moving from a trustless to a trusted system, then what you are proposing can work*.

P.S.  Nothing in this thread is new.  Even the posts telling you it won't work are recycled.

* For some values of work.  If bitcoin had been designed as a trusted system (even a distributed-trust system like you are proposing), I'm not sure that they would have worked in the sense of being worth trading 6 cents for, much less $600+.
newbie
Activity: 58
Merit: 0
Unless I'm misunderstanding you, I think I addressed that in an earlier reply:

I thought this was obvious, but perhaps not:

New nodes could ask others for the current root then. If nodes lie about it, there are one of two possible scenarios:

- They give a root that is too old. This can be mitigated by asking multiple nodes and picking the majority answer, combined with simply downloading the rest of the blockchain. If it got unlucky and the majority lied, it will eventually catch up and discard the outdated root on its own anyway because of the rolling window that's hard coded.

Downloading the rest of the block chain requires someone storing the entire block chain, no? You can't download something which everyone has deleted.

I do not believe that is correct.

The relevant bits of "the entire block chain" are always carried forward by the network and stored.

This proposal is addressing the irrelevant bits, which can include some or all of the following:

- Lost coins (a coin can be declared "lost" by some metric, like coin age > T, where T = 2 years.)
- The histories of coins that were spent and are now living farther down in the blockchain (within the rolling window).
- Any other outdated information.

There's no reason to store the history of a "coin" (in quotes because we're really talking about balances) all the way back to its creation. It can simply have a new, verified "origination point" in the blockchain.

- They give a root that's too young. This can be mitigated, again, by asking multiple nodes. If it also gets unlucky with its chosen majority, it should be able to figure this out because it's assumed that not all nodes are evil, and it will eventually stumble across a trustworthy one, which would make it have a longer blockchain than the ones that the lying nodes gave it, and it would automatically adopt that one.

This is an invalid assumption. Your ISP could simply block packets from honest nodes, for example.

Yes, and that's a problem that Bitcoin also faces. There's no difference that I can see here.
legendary
Activity: 905
Merit: 1012
Unless I'm misunderstanding you, I think I addressed that in an earlier reply:

I thought this was obvious, but perhaps not:

New nodes could ask others for the current root then. If nodes lie about it, there are one of two possible scenarios:

- They give a root that is too old. This can be mitigated by asking multiple nodes and picking the majority answer, combined with simply downloading the rest of the blockchain. If it got unlucky and the majority lied, it will eventually catch up and discard the outdated root on its own anyway because of the rolling window that's hard coded.

Downloading the rest of the block chain requires someone storing the entire block chain, no? You can't download something which everyone has deleted.

But besides that you have an even bigger problem: it costs me, or any network attacker approximately nothing to make sure that every single node you connect to is a compromised node I control. This is the problem which bitcoin seeks to solve! And the answer which Satoshi came up with, and the only answer which we have at this time is to validate the entire chain from start to finish and trust the longest valid fork.

- They give a root that's too young. This can be mitigated, again, by asking multiple nodes. If it also gets unlucky with its chosen majority, it should be able to figure this out because it's assumed that not all nodes are evil, and it will eventually stumble across a trustworthy one, which would make it have a longer blockchain than the ones that the lying nodes gave it, and it would automatically adopt that one.

This is an invalid assumption. Your ISP could simply block packets from honest nodes, for example.
newbie
Activity: 58
Merit: 0
True VISA scale (i.e 5,000 tps on blockchain)?

Figure 5,000 transactions per second.  Average tx is 400 bytes that is maybe 2MB/s (16 Mbps) sustained to write to the disk.  Average block would be 600 * 5,000 * 400 / 1024^3 = 1.2 GB.  So pretty much nothing.  Now to validate txs requires reading the outputs from the UXTO so maybe double that in terms of random reads.  Still we are talking a tiny fraction of the IO that even high end disks (to say nothing of SANs) can handle today.

Everyone thinks of disk first because it is the most obvious but in my analysis of the resource requirements I see it as the least critical bottleneck for full nodes.  If I had to rank them it would be node bandwidth (especially for residential connections), then memory, then computing power, and way back in last place would be disk capacity and speed.

Thanks for that analysis DeathAndTaxes! Very informative. Smiley

Also, just an FYI, I edited the OP to add:

Quote
Edit2 March 5, 2014: Something not mentioned here is that bitcoind can (and might already in some ways do) prune its locally stored blockchain, but only after it downloaded the entire thing once. Thus, this issue is more of a problem (I think) for new full-nodes, which currently must download the entire blockchain before they can prune the entire thing.

Based on a convo I just had about this topic over on #bitcoin-dev, I'm realizing that I have some knowledge gaps that I need to fill to better understand some of the alternative proposals, but I think others here might find this interesting so I'm sharing it here (with sipa's permission):

Quote
1:31:31 PM sipa: sugarpuff: now, about dealing with ways to improve performance, IF you are giving up zero trust
1:31:35 PM sipa: sugarpuff: see it this ways
1:31:47 PM sipa: sugarpuff: internally, every full node maintains a database of all unspent transaction outputs
1:32:13 PM sipa: sugarpuff: this is a high-efficiency, compact database with just outputs (not signatures for example) and not those that have already been spent
1:32:31 PM sipa: sugarpuff: every block you can see as sort of an authenticated patch against that database
1:32:45 PM sipa: (add outputs of trenasactions in the block, remove the outputs spent by transactions in it)
1:33:08 PM sipa: if we can somehow agree on a snapshot of that database at some point in time, and you trust it, we can just copy the database
1:33:13 PM sipa: instead of copying all blocks before it
1:34:01 PM sugarpuff: sipa: right, and that sounds like going into checkpoint territory. the rolling-root proposal is similar but not entirely the same
1:34:27 PM sugarpuff: but old blocks should have almost unanimous agreement in the network
1:34:36 PM sipa: sugarpuff: not done yet Smiley
1:35:07 PM sipa: sugarpuff: there are proposals to make such a snapshot of that database essentially for every block, and put a hash of that snapshot in the block itself
1:35:14 PM sipa: sugarpuff: and have this validated by miners too
1:35:43 PM sugarpuff: sipa: ah, yeah, that's a good idea. is that what UXTO is about?
1:35:56 PM sipa: the UTXO set is just what we call that database
1:36:10 PM sugarpuff: ah, lol, i just got the acronym
1:36:11 PM sipa: (Unspent TransaXtion Outputs)
1:36:14 PM sugarpuff: Smiley
1:36:29 PM sipa: now, this has a very significant impact on the network
1:36:41 PM sipa: as validating and maintaining these database snapshots is not cheap
1:37:16 PM sipa: (there are ways to make it pretty efficient, by using a merkle tree for that entire database, but the impact is not neglectable)
1:37:48 PM sipa: now, you can have a node that just asks a peer, "give me your current UTXO set", and it can validate that at least the majority of hashing power agrees with it
1:38:02 PM sipa: without needing to downloading any old blocks
1:38:36 PM sugarpuff: sipa: just by asking enough nodes for it and assuming the majority is right?
1:38:56 PM sipa: sugarpuff: no no, not the majority of your peers
1:39:06 PM sipa: sugarpuff: you can actually just validate the entire old blockchain, but only the headers
1:39:14 PM sipa: sugarpuff: which is very cheap (80 bytes per block)
1:39:18 PM sipa: sugarpuff: like SPV nodes do now
1:39:31 PM sipa: sugarpuff: but the headers would be committing to the hash of these UTXO set snapshots
1:39:42 PM sipa: so you can verify that the chain agrees with the snapshot
1:39:50 PM sipa: which implies the majority of miners do
1:40:31 PM sugarpuff: sipa: i'd like to do more reading on this and not use up more of your time, do you have some recommended links about this topic? Think SPV + UTXO thread is a good start?
1:40:54 PM sipa: several mailing list posts, forum posts, ...
1:41:00 PM sugarpuff: too broad...
1:41:05 PM sugarpuff: i could spend a year doing that
1:41:18 PM sugarpuff: i love the bitcoin paper, i've read it like 3.5 times now
1:41:18 PM sipa: there's no nice centralized knowledge repository of all these experimental ideas Smiley
1:41:28 PM sugarpuff: ah that's unfortunate
1:42:15 PM sugarpuff: sipa: ok, well thanks, i think what you said here is enough to get me started. i'll go do some more research on it. right now there are too many gaps in my knowledge to give you useful replies on this
1:42:23 PM sipa: Smiley
1:42:23 PM sipa: yw
donator
Activity: 1218
Merit: 1079
Gerald Davis
If you are a major exchange and are making $1B a month in profit (and you would if Bitcoin is so huge it has higher tx volume them VISA) do you think it might be worth it to pay $1,000 a year in additional storage cost to add some drives to the SAN?  I think it might be.

With that much data, what do you think would be a reasonable estimate of the disk I/O?

True VISA scale (i.e 5,000 tps on blockchain)?

Figure 5,000 transactions per second.  Average tx is 400 bytes that is maybe 2MB/s (16 Mbps) sustained to write to the disk.  Average block would be 600 * 5,000 * 400 / 1024^3 = 1.2 GB.  So pretty much nothing.  Now to validate txs requires reading the outputs from the UXTO so maybe double that in terms of random reads.  Still we are talking a tiny fraction of the IO that even high end disks (to say nothing of SANs) can handle today.

Everyone thinks of disk first because it is the most obvious but in my analysis of the resource requirements I see it as the least critical bottleneck for full nodes.  If I had to rank them it would be node bandwidth (especially for residential connections), then memory, then computing power, and way back in last place would be disk capacity and speed.

To give you an example of why memory would be more of an issue, is that latency for UXTO lookups will start to become excessive.  It may be less of an issue for non-miners (who can take 5 to 10 seconds to validate a block) but miners will want to validate potential blocks very rapidly.  To avoid the slow expensive pulls from disk, if possible one would want the entire UXTO and memory pool (list of unconfirmed transactions) in memory.  The only thing being written to disk would be the blocks.  The memory pool shouldn't be a problem even if it grew to 30x the size of the average block we are talking <64GB (which sounds like a lot but consider the doubling effect of Moores law for a decade or two).  The UXTO would be larger however the good news is the UXTO grows more linearly than the blockchain or tx volume.  

However eventually it may not be possible to have the entire UXTO in memory.  What I would consider an interesting project would be to design a probabilistic model which determines how likely an output would be involved in a transaction in the near future.   For example spam outputs which are uneconomical to spend would be less likely to be included and could be dropped from the UXTO in-memory cache (if they are used it would just need pulled from disk).  Other examples of low probability outputs would be "coins" with very old age, outputs to addresses which have a large number of other older outputs but few spends (probably paper or storage wallet).  Some true statistical modeling of the blockchain could provide better insight those were just some examples.  The client could score outputs on the likelihood it would be spent soon and the highest tx kept in the in-memory cache.

I am not convinced Bitcoin (at least on blockchain) will ever become that large but if it does clients will adapt to meet the needs of full nodes.  The protocol currently is rather inefficient in terms of resources (other than disk).  Tx are double verified, block messages include the full transaction (which most nodes already have) there will be a lot of room for optimization as transaction volume grows.


donator
Activity: 1419
Merit: 1015
If you are a major exchange and are making $1B a month in profit (and you would if Bitcoin is so huge it has higher tx volume them VISA) do you think it might be worth it to pay $1,000 a year in additional storage cost to add some drives to the SAN?  I think it might be.

With that much data, what do you think would be a reasonable estimate of the disk I/O?
newbie
Activity: 58
Merit: 0
Your suggestion won't work because there's no way to resolve conflict over which purported branch is valid except by reference to the historical block chain data.

Unless I'm misunderstanding you, I think I addressed that in an earlier reply:

New nodes could ask others for the current root then. If nodes lie about it, there are one of two possible scenarios:

- They give a root that is too old. This can be mitigated by asking multiple nodes and picking the majority answer, combined with simply downloading the rest of the blockchain. If it got unlucky and the majority lied, it will eventually catch up and discard the outdated root on its own anyway because of the rolling window that's hard coded.

- They give a root that's too young. This can be mitigated, again, by asking multiple nodes. If it also gets unlucky with its chosen majority, it should be able to figure this out because it's assumed that not all nodes are evil, and it will eventually stumble across a trustworthy one, which would make it have a longer blockchain than the ones that the lying nodes gave it, and it would automatically adopt that one.
newbie
Activity: 58
Merit: 0
Realpra, sorry, I tried, but my google-fu is failing me. The closest thing I was able to find is this post of yours mentioning it. If you could provide some history/links it'd sure be appreciated!
newbie
Activity: 58
Merit: 0
This was debated before, what you call  "rolling root" we called "ledger-solution"/"ledger-block".

Same idea; every say 1 year you would take all the 20 years or older transactions and move their unspent outputs to the latest "ledger block". Amounts in the ledger block would be much like coinbase transactions/miners fees.
Barring minor minor data growth from address fragmentation and perhaps block headers, this puts a quite final limit to the blockchain size.

Another idea (of my own) was "swarm clients" that individually would validate only parts of the blockchain, but as a whole would validate the whole thing - all zero trust etc..
This would allow for almost the smallest devices to participate in block validation forever.

You can google these terms, and I do mean google, the forum search is shite.

The main reason nothing has happened yet is laziness and people too busy getting rich off of Bitcoin - and no you can't lump me in with that group I have been hard at work helping Bitcoin for a while now.

Thank you sir! *That* is what I was looking for. I will do as you suggested and search for those terms.

If you're feeling especially kind, could you summarize the current status of the "ledger-solution"/"rolling-root proposal"? Did you already summarize it by saying, "The main reason nothing has happened yet is laziness and people too busy getting rich off of Bitcoin"?

And does that mean that it actually is a potential solution (as vetted by people with brains)?
hero member
Activity: 815
Merit: 1000
This was debated before, what you call  "rolling root" we called "ledger-solution"/"ledger-block".

Same idea; every say 1 year you would take all the 20 years or older transactions and move their unspent outputs to the latest "ledger block". Amounts in the ledger block would be much like coinbase transactions/miners fees.
Barring minor minor data growth from address fragmentation and perhaps block headers, this puts a quite final limit to the blockchain size.

Another idea (of my own) was "swarm clients" that individually would validate only parts of the blockchain, but as a whole would validate the whole thing - all zero trust etc..
This would allow for almost the smallest devices to participate in block validation forever.

You can google these terms, and I do mean google, the forum search is shite.

The main reason nothing has happened yet is laziness and people too busy getting rich off of Bitcoin - and no you can't lump me in with that group I have been hard at work helping Bitcoin for a while now.
legendary
Activity: 3472
Merit: 4801
Still looking for confirmation on this or a "you're suggestion won't work for reason(s) XYZ."

Explaining why your solution won't work in a way that you can understand will first require you to have a much better understanding of the problem that bitcoin solves, the way that is solves it, and the technical details of exactly how a transaction works.

Of course, once you have all that knowledge, the "reason(s) XYZ" will become obvious to you, and you'll be here with the rest of us trying to explain to the newbie what you've already tried to explain to dozens of other newbies that thought they were the first to come up with this solution.

I don't have time to take you through all the education you need at the moment, I'm getting sleepy.  If you don't have a satisfying grasp of the problem with your solution in the next day or so, I'll try to return and walk you through the steps necessary for you to understand the technical details of bitcoin.

Note:  At this point you have 5 "Hero Members" all trying to tell you that the idea has been suggested before, the idea has been analyzed, and that the idea won't work. While the moniker "Hero Member" is mostly meaningless (It just means that the member has been here a long time and is verbose in their posting frequency), it seems statistically likely that at least one person that has been here that long and has participated that frequently would have gained some useful knowledge by now.  The fact that there are 5 that are all saying the same thing, should cause you to start thinking along the lines of "there appears to be something here that I need to understand better" rather than "there appears to be something here that they all need to understand better".  It's not impossible for 5 experienced people to all fall into the same faulty line of thought and assumption, but the odds are that you should give some consideration to the possibility that there's something significant that you're misunderstanding.
legendary
Activity: 905
Merit: 1012
Your suggestion won't work because there's no way to resolve conflict over which purported branch is valid except by reference to the historical block chain data.
newbie
Activity: 58
Merit: 0
The restrictions are there for a good reason, to encourage learning before teaching.

The problem you are about to tackle is hard. I recommend this thread to get an insight of its complexity.

https://bitcointalksearch.org/topic/ultimate-blockchain-compression-w-trust-free-lite-nodes-88208

Thanks, I'd had a look at that thread already, but a second, closer read-through is probably warranted (especially since etotheipi has revised his idea a few times).

Still looking for confirmation on this or a "you're suggestion won't work for reason(s) XYZ."
hero member
Activity: 836
Merit: 1030
bits of proof
The restrictions are there for a good reason, to encourage learning before teaching.

The problem you are about to tackle is hard. I recommend this thread to get an insight of its complexity.

https://bitcointalksearch.org/topic/ultimate-blockchain-compression-w-trust-free-lite-nodes-88208
newbie
Activity: 58
Merit: 0
Only the private key holder of the tx output can create a new tx.  There is no mechanism to "copy the outputs that were coped as new transactions to the head".

The new protocol dictates that your node create this new transaction for you. Edit: I suspect that might not even be necessary, since the balance wouldn't change. It would just be an alias for the old txn.. not entirely sure ATM.

If this method works, and disk space becomes a real problem, it might just have to work this way.

(BTW, I'd love to give ya'll faster replies, but this forum has imposed a bunch of restrictions on this account because it hasn't posted much. I can currently post a maximum of 1 reply per 6 minutes, and I don't know if that timer resets each time I try.. Annoying.)
Pages:
Jump to: