Topic: Blockchain is 360GB and growing, can it be consolidated? - page 3. (Read 872 times)

more that prune.. means more that DONT HAVE THE FULL BLOCKCHAIN.. why.. because they pruned
yep pruned nodes dont have the full blockchain. you can tell because they only have 7gb of space used not 360gb.
more people with only 7gb which is mostly only the last fortnight means more people that dont have all the data since genesis.
this then means more leachers are all then relying on less full nodes that do have 360gb full blockchain. and those less full nodes with full blockchain have to broadcast more data to those leachers meaning more centralised.


its an old phrase 'supernodes', but its a thing. the big main fullnodes of pools and exchanges select who they want to connect with for anti-DDos security and also to ensure they are linked to other important nodes for transaction relay efficiency.  

imagine pools/exchanges at the centre of a network and they transmit data out. like an exploding star. where each peer(planets) expands the universe of nodes outwards.
the lite/pruned wallets/nodes wont be seen as a priority nodes which the planets near the explosion want to transmit to. instead lite/pruned nodes are far out at the edge of the universe.
[not to be confused with an alt-nets bad naming buzzword for a channel]


a pruned node keeps X amount of most recent block
new users wanting to IBD(initial block download) start from block 0 not from block 560,000
so a pruned node only keeping block 560k to 712k wont have blocks 0-559k. and thus the new peer will disconnect and try finding another peer. meaning the pruned nodes are not getting any sustained downstream connections and there for they become the outer edge of the peer universe.

its the same for old version nodes that dont store all the witness data. new nodes will reject the data because its not full witness and disconnect from the node leaving the old node at the edge of the universe

I may be wrong but the P2P protocol needs to change for this to work. For now the pruned node can only announce that they are pruned and not able to tell the other node how many blocks they have stored so any node that connects to a pruned node assumes they have the minimum number of blocks. This means it may not matter if you store 100 GB or 500 MB they won't ask for old blocks from your node.

You're bringing up an interesting point here. I was always mostly discarding the idea of pruned nodes as helpful for the network, but one might actually prune for example to 150GB or 100GB; thus still serving new nodes almost half of the blockchain, which isn't unsubstantial and still relatively easy on hard disk space. Many games these days take 100GB, so I feel allocating 100 gigs to helping the Bitcoin network like '30%' utility, is better than nothing, if you don't have the space for a full blockchain copy.

Probably my pessmistic view of pruned nodes came from the idea of pruning down to like 5GB, but of course the limit can be set to anything you want.
Added benefit of e.g. keeping the last 100GB: You will probably have a copy of each block mined since you entered and owned BTC if you're still new to Bitcoin.

The more people that prune means more nodes, which definitely not means less decentralization or less peers available to get the full chain from.

What do you mean the edge? That you'll need to redownload the block chain if you ever wanted to add another wallet? That you do not share any block and hence, not help for the bandwidth of the network? If it's the former, that has nothing to do with the network and if it's the latter, you do share those few blocks you have.

Or even worse:  They can link your addresses. What you're describing as a problem can be tackled by connecting through Tor.

I agree with franky1 here. A pruned node is downloading the entire blockchain and verifying each transaction. There is no trust involved. Pruned nodes just doesn't achieve the entire blockchain, and save HD space. which is exactly what the OP is talking about.

Personally, i don't know what is their limitation, but you can even use Bitcoin Core software to run a pruned node.


Imo, a pruned node is a full node.

Let's look what bitcoin wiki says:

Imo, a pruned node fits that description.

Anyway, the network needs full nodes to verify all transactions and all blockdata without any trust.

For example, let's image you are a big exchange (like Binance or Coinbase), or that you are blockexplorer. you need to run a full node to verify each single UTXO, not just the ones that you are interested with.

If someone sends you a coin which was received in 2013, you need to be able to fully verify it was not spent without trusting anyone.

Full nodes are also important for privacy purposes. When you use an spv wallet, this wallet asks for the balance of your addresses to other full nodes. Those nodes may be tracking your requests, and they might link your IP with your addresses.

To add to what franky1 said, you can also look at people who don't leave their nodes on full time the same way.
You open your client, sync the blockchain, do your transaction(s) and shut it down.
Yes, you can send out old data while your node is syncing, but you are still, more then likely taking in way more data then you are giving out.
Outside of keeping your TX on your own node for privacy, there is really very little reason to run your own node that way.

-Dave

pruned nodes are not a trust thing.
all nodes(pruned or full) initially see a complete copy of all blocks and all transactions within.
just a pruned node does not archive them all.

this means a pruned node is not a (torrent buzzword) 'seeder' peer to then give that full information to others. thus peers disconnect from them and search for proper full nodes to get the data

if everyone pruned then no new users can then see the full blockchain to then do a full verification of all transactions.
.

this means the more people that prune. the less archives are decentralised. meaning less peers available to get the full blockchain from.
.
if your a random user that doesnt do much transactions or spend much time with your node running, whereby being a seeder or wanting to be part of the network is not important to you then fine prune. but if you care about decentralisation and want to be a full node then dont prune.

but just be aware that being a pruned node does not make you part of the full decentralised network, you are more of the edge of the network, deemed a (torrent buzzword)'leacher' peer, not a seeder peer. by which you are shrinking/centralising the network by reducing the amount of full node(seeder) peers available, plus also putting those remaining seeder peers under more strain to transmit their full blockchains downstream to random peers by being relied on more as blockchain sources

this can lead to a paradigm where full node archive nodes (seeders) need high bandwidth(unlimited data) internet plans because they are getting more demand, just to service the leachers. leading to another drop off of full nodes due to expense of servicing leachers. or leachers/new peers having a delay in finding seeder peers due to demand because seeder peers limit their connections to not use excessive data. both of which can shrink/centralise the amount of full nodes available, or slowdown the speed of getting blockdata

That's a great way to put it but not quite what I meant


I was more thinking along the lines of the 3rd party node spoofing incoming transactions, but it looks like that's less of an issue than I initially thought:


Obviously the 3rd party node hiding transactions is not ideal either, but less exploitable than the other way round.

Oh, so you meant the safety of your physical integrity?

Yes, that's what I said. You have to trust the rest of the nodes.

You mean certainty? Validating blocks prior x or just starting from the genesis block adds certainty that the blocks are valid.

Where's devil's advocate? Judging by your post, I can only assume you agree with me.

A full (as in fully verifying) node needs two things in order to sync to the state of some current header.

1) the UTXO set

2) a proof that this UTXO set results from a valid block history from genesis to the current header

Part 1) only accounts for a small fraction of the 360GB; the vast majority is taken by part 2).

But in principle, part 2) could be replaced by a so-called Succinct Non-interactive Argument of Knowledge with recursive verification, that would be vastly smaller than part 1).

See this book for more details:

https://zcash.github.io/halo2/concepts/proofs.html

This technology is still in its infancy, and formalizing the entire Bitcoin consensus model into a SNARK would be a monumental effort, not to mention constructing the proof (which would have to be redone at regular intervals, e.g. every few months). But verification would be very efficient.

Playing devils advocate (and seeing if I can either disprove or validate your point):

If a chainstate was included in block x, and a node that didn’t validate any blocks prior to block x, would need to assume the block x they received is valid. If it is not, the chainstate in block x is invalid. Similarly, block x+1 would be invalid.

One part of validating a block is validating the total proof of work, that is the sum of the POW performed access all blocks to date, and that the current POW is beyond the threshold for the current difficulty. So the cost of producing a fake block x is not trivial and if a node is connected to multiple other random nodes, they should receive the valid blockchain. There are a variety of scenarios in which the total POW for blocks prior to block x could be faked to make it appear that an invalid block x has the highest total POW when it is not a valid blockchain.

Only validating blocks past block x would add uncertainty to the validation process. The chances of receiving an invalid long chain of blocks is low considering the cost of producing invalid blocks, however there are trivial “fixes” to this process that can reduce the chances of accepting an invalid blockchain to nearly zero.

If you are running a pruned node, you will validate the entire blockchain, but will not store more than z blocks on your hard drive. So you will validate all blocks but will not store all blocks. This means that, unless you are being subjected to a Sybil attack, you will not accept an invalid blockchain, but will only store a limited number of blocks. The current implementation of a pruned node is not ideal, for example if a new private key is imported, it will need to download the entire blockchain again in order to know the unspent outputs it can spend, even though it has the UTXO set stored.

Depends on your threat model! For a regular user it might make no difference (unlike e.g. storing your coins in a hot wallet vs using cold storage or a hardware wallet), but if you run a crypto-heavy business like an exchange I'd argue that running your own full node is safer over relying on someone elses.

No, you should see this in another way. Alright, let's say that Bob who runs his own node receives 1 BTC and the consolidation happens each time a new block is mined, for the 1,000th block. This means that sometime, his node will get rid of the block that contains that transaction and will mark his balance with 1 BTC.

Now, let's say that Alice wants to run her own node and starts receiving the consolidated balances of all the bitcoin addresses. What will prevent Bob from changing his mark that he owns 1 BTC? How can Alice verify that Bob, indeed, owns 1 BTC without trusting his sayings? The answer is:  By verifying every single transaction from the very first block.

There's no tradeoff in security when it comes to running a node or trusting an SPV server. There's only tradeoff in privacy for the latter.

Yeah, I'm pretty sure this was suggested before or some version of it, I was more curious about the objections, so thank you guys for helping me understand. 

In addition to what the others already said, keep in mind that Bitcoin is also about choice. With the way it works now, it's completely up to you how much you want to trust and how much additional effort you want to expend to stay trustless:

Want to remain trustless and are willing to keep some extra HD space for easier re-syncing if something goes wrong? Use a full node.

Want to remain trustless but don't want to use more HD space than necessary? Use a pruned node.

Want to spend as few resources as possible while keeping your own keys? Use a SPV wallet.


Each approach comes with a tradeoff of security vs convenience and while SPV wallet suffices for most people it's still important to allow for maximum security and trustlessness, where required.



On a sidenote, transaction cut-throughs may be interesting to you:
https://bitcointalksearch.org/topic/transaction-cut-through-281848

Some minor alts are using this approach, unfortunately implementation doesn't appear to be feasible for Bitcoin though.

The entire network does not matter, it's if A node can be fooled into accepting it.
And it's just not worth the risk.

This conversation keeps coming up on the forum about "ways to shrink the blockchain" and the answer is you can't. If you want to verify everything then you need everything.
If you don't want to verify everything, then there are plenty of light wallets that you don't need to download the entire blockchain for.

n0nce even started a thread on how you can run a full node for under $50
https://bitcointalksearch.org/topic/guide-how-to-run-a-bitcoin-core-full-node-for-under-50-bucks-5364742


-Dave

See, without such a consolidation and by instead verifying the whole blockchain, block by block starting from the genesis block, there is no previous state to verify or anything like that. That's why it's really the best way to do things. Also there have been ideas like yours in the past, such as inserting 'sync blocks' into the blockchain periodically, but all these changes even if just slightly, decrease the security, meanwhile the blockchain only grows slowly and storage gets cheaper and cheaper.

The motivation to change anything is really not strong enough. During Black Friday, there were 1TB SSDs for <80€, and it gets cheaper with HDDs. 1TB will last like 10 years-ish, if I remember correctly. I can save up another 80 bucks during 10 years if I'll then need to get a second drive lol.

Also, one thing you got wrong in your initial post:

Bitcoin is not balance-based, it's UTXO-based. This makes some things better, some things harder.

I see your point. I agree with you that trust would be less self-verifiable and delegated to an extent. The idea is that the hash would be verified against previous transactions up to the previous state, but the problem remains: how to verify the previous state?

Correct me if I'm wrong, but I understand trust here as being statistical, as in, "highly unlikely or computationally too demanding to be otherwise". By the same logic, how likely can the entire network be fooled into accepting a wrong but publicly verifiable state for a month or a year?

For the reason I told you. Due to the need for trust.

There'll never be enough time. Once the consolidation sets sail, every other person who will want to find out the previous transactions by running a node, will have to trust the other nodes.

I said 1,000 blocks, but anyway. Give or take whatever you want, it's a similar analogy.

Yeah, of course they can produce a hash. The problem is that you can't verify that hash if you don't have the content of the block. I may give you the following hash:

But, if you don't have the whole thing, which is this, how can you confirm that I've, indeed, found a valid proof of work and not written those hexadecimal characters by myself with zero effort?

BlackHatCoiner, very good point about "trust but verify", I agree.

However, what if this consolidation happened every month or year, leaving everybody plenty of time to publicly verify previous states and transactions?

Regarding your example, I see why syncing the node with a network that limits itself to the last 1000 transactions would be a problem, but since past nodes never change, can't a verification on past nodes produce some publicly verifiable hash that can just be checked and everybody can just move on?