Gold collapsing. Bitcoin UP. - page 402.

sidhujag

legendary

Activity: 2044

Merit: 1005

Quote from: sickpig on April 27, 2015, 04:37:54 AM

Quote from: sidhujag on April 27, 2015, 02:09:04 AM

Quote from: molecular on April 27, 2015, 12:23:54 AM

Quote from: sidhujag on April 26, 2015, 04:53:38 PM

Did you see the latest commit on pruning? Allows you to specify max disk usage and it prunes old blocks.. validates by reindexing which downloads all chain then prunes. You check it out? I wonder what the point is if youhave todownload the full chain anyway why prune? The main bottleneck is the lenghty sync time not storage space.
I also noticed that it has logic to stop sending blocks requested that have been pruned. So there is assumption that there are full nodes out there to give you the block that others have pruned.

There's also a plan to let you keep certain block ranges and network protocol addition to advertise which parts a node has. That way, full block data can be kept in a distributed fashion while nodes can still run (and contribute more meaningfully than just as a relay and utxo set provider) even with disk space limits.

In fact with that it would be theoretically possible to run the network (fully trustable) without any node having storage space for the whole blockchain.

Kinda like bittorrent ? This would just optimize the p2p layer of the protocol but need to
ensure its safe from attacks maybe thats why we didnt
do it to be on safe side.

I think that more than Bittorrent you should look at something along the line of DHT. There's a very informative thread in the Development & Technical Discussion
section of the forum... found:

"Using a DHT to reduce the resource requirements of full nodes" by DeathAndTaxes
https://bitcointalksearch.org/topic/m.7470866

Especially look at gmaxwell reply.

Another really helpful source of info about the approach adopted in the prune implementation is
this this 2014 thread on bitcoin core dev mailing list:

"[Bitcoin-development] Service bits for pruned nodes"
http://sourceforge.net/p/bitcoin/mailman/bitcoin-development/thread/CAPg%2BsBjSe23eADMxu-1mx0Kg2LGkN%2BBSNByq0PtZcMxAMh0uTg%40mail.gmail.com/#msg30779911

Yea gmaxwell pretty much said its not worth it for the risks you inherit. And DeathAndTaxes pretty much admitted himself that a more efficient p2p layer without DHT complexity would solve the problem in an effective manor.

cypherdoc

legendary

Activity: 1764

Merit: 1002

starting to roll:

sickpig

legendary

Activity: 1260

Merit: 1008

Quote from: sidhujag on April 27, 2015, 02:09:04 AM

Quote from: molecular on April 27, 2015, 12:23:54 AM

Quote from: sidhujag on April 26, 2015, 04:53:38 PM

Did you see the latest commit on pruning? Allows you to specify max disk usage and it prunes old blocks.. validates by reindexing which downloads all chain then prunes. You check it out? I wonder what the point is if youhave todownload the full chain anyway why prune? The main bottleneck is the lenghty sync time not storage space.
I also noticed that it has logic to stop sending blocks requested that have been pruned. So there is assumption that there are full nodes out there to give you the block that others have pruned.

There's also a plan to let you keep certain block ranges and network protocol addition to advertise which parts a node has. That way, full block data can be kept in a distributed fashion while nodes can still run (and contribute more meaningfully than just as a relay and utxo set provider) even with disk space limits.

In fact with that it would be theoretically possible to run the network (fully trustable) without any node having storage space for the whole blockchain.

Kinda like bittorrent ? This would just optimize the p2p layer of the protocol but need to
ensure its safe from attacks maybe thats why we didnt
do it to be on safe side.

I think that more than Bittorrent you should look at something along the line of DHT. There's a very informative thread in the Development & Technical Discussion
section of the forum... found:

"Using a DHT to reduce the resource requirements of full nodes" by DeathAndTaxes
https://bitcointalksearch.org/topic/m.7470866

Especially look at gmaxwell reply.

Another really helpful source of info about the approach adopted in the prune implementation is
this this 2014 thread on bitcoin core dev mailing list:

"[Bitcoin-development] Service bits for pruned nodes"
http://sourceforge.net/p/bitcoin/mailman/bitcoin-development/thread/CAPg%2BsBjSe23eADMxu-1mx0Kg2LGkN%2BBSNByq0PtZcMxAMh0uTg%40mail.gmail.com/#msg30779911

smooth

legendary

Activity: 2968

Merit: 1198

Quote from: sidhujag on April 27, 2015, 02:09:04 AM

Quote from: molecular on April 27, 2015, 12:23:54 AM

Quote from: sidhujag on April 26, 2015, 04:53:38 PM

Did you see the latest commit on pruning? Allows you to specify max disk usage and it prunes old blocks.. validates by reindexing which downloads all chain then prunes. You check it out? I wonder what the point is if youhave todownload the full chain anyway why prune? The main bottleneck is the lenghty sync time not storage space.
I also noticed that it has logic to stop sending blocks requested that have been pruned. So there is assumption that there are full nodes out there to give you the block that others have pruned.

There's also a plan to let you keep certain block ranges and network protocol addition to advertise which parts a node has. That way, full block data can be kept in a distributed fashion while nodes can still run (and contribute more meaningfully than just as a relay and utxo set provider) even with disk space limits.

In fact with that it would be theoretically possible to run the network (fully trustable) without any node having storage space for the whole blockchain.

Kinda like bittorrent ?

Not exactly. BitTorrent distributes the entire file to all peers and in fact it actively tries to distribute pieces with a low count more quickly. In this case, not every node would have every block by design. It's a bit more dangerous what is being proposed. An attacker can prevent new nodes from syncing by finding all of the nodes that have a particular piece of the blockchain (whichever piece is rarest) and attacking them.

Still, as long as there is one good copy of the blockchain in the world, it can always be sent out again. So I don't think we need to worry too much. Some people will still run nodes without pruning.

sidhujag

legendary

Activity: 2044

Merit: 1005

Quote from: molecular on April 27, 2015, 12:23:54 AM

Quote from: sidhujag on April 26, 2015, 04:53:38 PM

Did you see the latest commit on pruning? Allows you to specify max disk usage and it prunes old blocks.. validates by reindexing which downloads all chain then prunes. You check it out? I wonder what the point is if youhave todownload the full chain anyway why prune? The main bottleneck is the lenghty sync time not storage space.
I also noticed that it has logic to stop sending blocks requested that have been pruned. So there is assumption that there are full nodes out there to give you the block that others have pruned.

There's also a plan to let you keep certain block ranges and network protocol addition to advertise which parts a node has. That way, full block data can be kept in a distributed fashion while nodes can still run (and contribute more meaningfully than just as a relay and utxo set provider) even with disk space limits.

In fact with that it would be theoretically possible to run the network (fully trustable) without any node having storage space for the whole blockchain.

Kinda like bittorrent ? This would just optimize the p2p layer of the protocol but need to
ensure its safe from attacks maybe thats why we didnt
do it to be on safe side.

solex

legendary

Activity: 1078

Merit: 1006

100 satoshis -> ISO code

Quote from: molecular on April 27, 2015, 12:23:54 AM

Quote from: sidhujag on April 26, 2015, 04:53:38 PM

Did you see the latest commit on pruning? Allows you to specify max disk usage and it prunes old blocks.. validates by reindexing which downloads all chain then prunes. You check it out? I wonder what the point is if youhave todownload the full chain anyway why prune? The main bottleneck is the lenghty sync time not storage space.
I also noticed that it has logic to stop sending blocks requested that have been pruned. So there is assumption that there are full nodes out there to give you the block that others have pruned.

There's also a plan to let you keep certain block ranges and network protocol addition to advertise which parts a node has. That way, full block data can be kept in a distributed fashion while nodes can still run (and contribute more meaningfully than just as a relay and utxo set provider) even with disk space limits.

In fact with that it would be theoretically possible to run the network (fully trustable) without any node having storage space for the whole blockchain.

Yes, keeping whole block ranges (randomly determined before pruning) is an important feature of pruning which will hopefully be included in due course.

molecular

donator

Activity: 2772

Merit: 1019

Quote from: sidhujag on April 26, 2015, 04:53:38 PM

Did you see the latest commit on pruning? Allows you to specify max disk usage and it prunes old blocks.. validates by reindexing which downloads all chain then prunes. You check it out? I wonder what the point is if youhave todownload the full chain anyway why prune? The main bottleneck is the lenghty sync time not storage space.
I also noticed that it has logic to stop sending blocks requested that have been pruned. So there is assumption that there are full nodes out there to give you the block that others have pruned.

There's also a plan to let you keep certain block ranges and network protocol addition to advertise which parts a node has. That way, full block data can be kept in a distributed fashion while nodes can still run (and contribute more meaningfully than just as a relay and utxo set provider) even with disk space limits.

In fact with that it would be theoretically possible to run the network (fully trustable) without any node having storage space for the whole blockchain.

cypherdoc

legendary

Activity: 1764

Merit: 1002

Quote from: smooth on April 26, 2015, 08:51:09 PM

Quote from: sidhujag on April 26, 2015, 08:47:49 PM

cool thanks, but I thought unspent outputs would be derived from transactions which would no longer exist in the "backend" node. How does it know the unspent balance without tallying up all of the transactions it was involved in?

The way pruning works is the node still receives all transactions as before. The ones that get spent, it throws away. So the ones left are the ones it keeps in its database. Those are the unspent ones. This requires much less storage.

UTXO's have always worked this way; throwing away the spent ones. The UTXO set is very small compared to blocks because it only contains info on unspent data and doesn't contain any signatures or data, thus, this is not what results in less storage. it is the discarding of old blocks. in the case of saving the last 2016 blocks (minimum 288), 351874 of them as of now.

cypherdoc

legendary

Activity: 1764

Merit: 1002

Quote from: sidhujag on April 26, 2015, 08:31:01 PM

Quote from: smooth on April 26, 2015, 08:23:19 PM

Quote from: sidhujag on April 26, 2015, 08:18:40 PM

Quote from: smooth on April 26, 2015, 07:55:13 PM

Quote from: sidhujag on April 26, 2015, 07:53:21 PM

Quote from: smooth on April 26, 2015, 05:41:47 PM

Quote from: sidhujag on April 26, 2015, 05:39:10 PM

Quote from: cypherdoc on April 26, 2015, 05:33:01 PM

My understanding is you download the whole blockchain first for verificatio, then delete all but around the last 200 or so blocks. It works off the UTXO set.

In the case of the compute stick, you could use a separate SSD to initialize the blockchain, then disconnect it. Sticking these things in every nook and cranny could be a real problem for those against Bitcoin.

So for basically to allow for p2p tx and block relaying around the globe thru smaller mediums. The main use case I woulda thought is to reduce time for new nodes to sync up.. Since we know the price of hd is falling faster then the need for that space.

The pruning stuff does not reduce sync time at all, though the recent headers-first and syncing improvements do.

In order to prune syncing you have to relax the idea of independent verification and rely on someone else telling you that the part of the chain you are seeing is valid (as with SPV nodes). That's a major change to the Bitcoin trust model.

With spv you only get headers and maybe coinbase tx correct? So you wont be able to import your pvt key and get your txs in spv mode..

No you tell the server the addresses you are interested in and it gives you those tx. However, since you don't see the other tx you can't be sure that the tx you receive actually represent a valid chain of custody back to a coinbase (unless you trust the miners).

Ahh i see so essentialy those that dont want to sync fully can download a lite wallet which gets blocks via spv mode via a trusted node( ssl rpc).. so pruning doesnt really come into play here? If you prune then you wont be able to get your tx if trusted node pruned that block. So spv solves the blockchain sync time and pruning solves what? Being able to store on small flash disks running to relay transactions?

SPV solves being able to create and receive transactions without the bandwidth requirements of a node. Pruning solves being able to run a node that validates and relays transactions (and can even mine!) without storing the entire blockchain. They can even serve SPV clients, because they have all of the unspent transactions, and that's all SPV clients would ever want. These nodes just can't sync new nodes though (at least not by themselves) since they don't have all the old blocks to share.

By unspent I assume you mean the ones in mempool. But how would one import their wallet key into an spv thin client backed by a pruned node and get correct balance? It needs tx that have been pruned?

i think you are mixing up these concepts.

think of 3 types of nodes:

1. SPV clients: only store headers, no blockchain. has to trust other mining peers by measuring their POW. subject to Sybil attacks.
2. full nodes: full blockchain with all historical tx history. heavy data storage demands which is what primarily drives up the cost of running a full node.
3. Pruned full node: in between #1 & #2. highly cost effective as it trims the blockchain down to 1.3GB. initially downloads entire blockchain to verify tx history and build UTXO set and then deletes older blocks up to 2016 blocks back (i hear it can vary depending on a flag with the minimum being 288). pruned nodes won't be able to serve up the blockchain to new nodes but it can validate, verify, relay and mine. they will actually improve mining decentralization.

cypherdoc

legendary

Activity: 1764

Merit: 1002

Quote from: smooth on April 26, 2015, 08:23:19 PM

Quote from: sidhujag on April 26, 2015, 08:18:40 PM

Quote from: smooth on April 26, 2015, 07:55:13 PM

Quote from: sidhujag on April 26, 2015, 07:53:21 PM

Quote from: smooth on April 26, 2015, 05:41:47 PM

Quote from: sidhujag on April 26, 2015, 05:39:10 PM

Quote from: cypherdoc on April 26, 2015, 05:33:01 PM

My understanding is you download the whole blockchain first for verificatio, then delete all but around the last 200 or so blocks. It works off the UTXO set.

In the case of the compute stick, you could use a separate SSD to initialize the blockchain, then disconnect it. Sticking these things in every nook and cranny could be a real problem for those against Bitcoin.

So for basically to allow for p2p tx and block relaying around the globe thru smaller mediums. The main use case I woulda thought is to reduce time for new nodes to sync up.. Since we know the price of hd is falling faster then the need for that space.

The pruning stuff does not reduce sync time at all, though the recent headers-first and syncing improvements do.

In order to prune syncing you have to relax the idea of independent verification and rely on someone else telling you that the part of the chain you are seeing is valid (as with SPV nodes). That's a major change to the Bitcoin trust model.

With spv you only get headers and maybe coinbase tx correct? So you wont be able to import your pvt key and get your txs in spv mode..

No you tell the server the addresses you are interested in and it gives you those tx. However, since you don't see the other tx you can't be sure that the tx you receive actually represent a valid chain of custody back to a coinbase (unless you trust the miners).

Ahh i see so essentialy those that dont want to sync fully can download a lite wallet which gets blocks via spv mode via a trusted node( ssl rpc).. so pruning doesnt really come into play here? If you prune then you wont be able to get your tx if trusted node pruned that block. So spv solves the blockchain sync time and pruning solves what? Being able to store on small flash disks running to relay transactions?

SPV solves being able to create and receive transactions without the bandwidth requirements of a node. Pruning solves being able to run a node that validates and relays transactions (and can even mine!) without storing the entire blockchain. They can even serve SPV clients, because they have all of the unspent transactions, and that's all SPV clients would ever want. These nodes just can't sync new nodes though (at least not by themselves) since they don't have all the old blocks to share.

actually, pruned nodes can't interact with SPV clients since they won't have the blocks.

SPV clients use a bloom filter looking for the addresses and transactions they are interested in to the peer. The peer then only includes the relevant transactions in blocks it returns, together with their Merkle paths to prove that these transactions were indeed part of returned block.

cypherdoc

legendary

Activity: 1764

Merit: 1002

Quote from: smooth on April 26, 2015, 07:55:13 PM

Quote from: sidhujag on April 26, 2015, 07:53:21 PM

Quote from: smooth on April 26, 2015, 05:41:47 PM

Quote from: sidhujag on April 26, 2015, 05:39:10 PM

Quote from: cypherdoc on April 26, 2015, 05:33:01 PM

My understanding is you download the whole blockchain first for verificatio, then delete all but around the last 200 or so blocks. It works off the UTXO set.

In the case of the compute stick, you could use a separate SSD to initialize the blockchain, then disconnect it. Sticking these things in every nook and cranny could be a real problem for those against Bitcoin.

So for basically to allow for p2p tx and block relaying around the globe thru smaller mediums. The main use case I woulda thought is to reduce time for new nodes to sync up.. Since we know the price of hd is falling faster then the need for that space.

The pruning stuff does not reduce sync time at all, though the recent headers-first and syncing improvements do.

In order to prune syncing you have to relax the idea of independent verification and rely on someone else telling you that the part of the chain you are seeing is valid (as with SPV nodes). That's a major change to the Bitcoin trust model.

With spv you only get headers and maybe coinbase tx correct? So you wont be able to import your pvt key and get your txs in spv mode..

No you tell the server the addresses you are interested in and it gives you those tx. However, since you don't see the other tx you can't be sure that the tx you receive actually represent a valid chain of custody back to a coinbase (unless you trust the miners).

this is why spv clients will sample several different peers.

marcus_of_augustus

legendary

Activity: 3920

Merit: 2349

Eadem mutata resurgo

Quote from: sidhujag on April 26, 2015, 08:53:24 PM

Quote from: smooth on April 26, 2015, 08:51:09 PM

Quote from: sidhujag on April 26, 2015, 08:47:49 PM

cool thanks, but I thought unspent outputs would be derived from transactions which would no longer exist in the "backend" node. How does it know the unspent balance without tallying up all of the transactions it was involved in?

The way pruning works is the node still receives all transactions as before. The ones that get spent, it throws away. So the ones left are the ones it keeps in its database. Those are the unspent ones. This requires much less storage.

Makes total sense. Yea I even read that part of the code didn't clue in, duh.

The unspent transaction outputs (UTXO) is the actual distributed ledger. They are contained within the journal of all transactions seen on the network (the blockchain) which needs to be sorted through and filtered for unspent outputs.

You can tell all the "blockchain tech." was a bunch of marketing BS spouted by followers, not leaders, because the blockchain is already obsolete. UTXO is the future Wink

and where all the bleeding edge development is happening.

sidhujag

legendary

Activity: 2044

Merit: 1005

Quote from: smooth on April 26, 2015, 08:51:09 PM

Quote from: sidhujag on April 26, 2015, 08:47:49 PM

cool thanks, but I thought unspent outputs would be derived from transactions which would no longer exist in the "backend" node. How does it know the unspent balance without tallying up all of the transactions it was involved in?

The way pruning works is the node still receives all transactions as before. The ones that get spent, it throws away. So the ones left are the ones it keeps in its database. Those are the unspent ones. This requires much less storage.

Makes total sense. Yea I even read that part of the code didn't clue in, duh.

smooth

legendary

Activity: 2968

Merit: 1198

Quote from: sidhujag on April 26, 2015, 08:47:49 PM

cool thanks, but I thought unspent outputs would be derived from transactions which would no longer exist in the "backend" node. How does it know the unspent balance without tallying up all of the transactions it was involved in?

The way pruning works is the node still receives all transactions as before. The ones that get spent, it throws away. So the ones left are the ones it keeps in its database. Those are the unspent ones. This requires much less storage.

sidhujag

legendary

Activity: 2044

Merit: 1005

Quote from: smooth on April 26, 2015, 08:36:55 PM

Quote from: sidhujag on April 26, 2015, 08:31:01 PM

Quote from: smooth on April 26, 2015, 08:23:19 PM

Quote from: sidhujag on April 26, 2015, 08:18:40 PM

Quote from: smooth on April 26, 2015, 07:55:13 PM

Quote from: sidhujag on April 26, 2015, 07:53:21 PM

Quote from: smooth on April 26, 2015, 05:41:47 PM

Quote from: sidhujag on April 26, 2015, 05:39:10 PM

Quote from: cypherdoc on April 26, 2015, 05:33:01 PM

My understanding is you download the whole blockchain first for verificatio, then delete all but around the last 200 or so blocks. It works off the UTXO set.

In the case of the compute stick, you could use a separate SSD to initialize the blockchain, then disconnect it. Sticking these things in every nook and cranny could be a real problem for those against Bitcoin.

So for basically to allow for p2p tx and block relaying around the globe thru smaller mediums. The main use case I woulda thought is to reduce time for new nodes to sync up.. Since we know the price of hd is falling faster then the need for that space.

The pruning stuff does not reduce sync time at all, though the recent headers-first and syncing improvements do.

In order to prune syncing you have to relax the idea of independent verification and rely on someone else telling you that the part of the chain you are seeing is valid (as with SPV nodes). That's a major change to the Bitcoin trust model.

With spv you only get headers and maybe coinbase tx correct? So you wont be able to import your pvt key and get your txs in spv mode..

No you tell the server the addresses you are interested in and it gives you those tx. However, since you don't see the other tx you can't be sure that the tx you receive actually represent a valid chain of custody back to a coinbase (unless you trust the miners).

Ahh i see so essentialy those that dont want to sync fully can download a lite wallet which gets blocks via spv mode via a trusted node( ssl rpc).. so pruning doesnt really come into play here? If you prune then you wont be able to get your tx if trusted node pruned that block. So spv solves the blockchain sync time and pruning solves what? Being able to store on small flash disks running to relay transactions?

SPV solves being able to create and receive transactions without the bandwidth requirements of a node. Pruning solves being able to run a node that validates and relays transactions (and can even mine!) without storing the entire blockchain. They can even serve SPV clients, because they have all of the unspent transactions, and that's all SPV clients would ever want. These nodes just can't sync new nodes though (at least not by themselves) since they don't have all the old blocks to share.

By unspent I assume you mean the ones in mempool.

No mempool is unconfirmed. Unspent is means unspent. Meaning not already spent and you could therefore spend them. The total of your unspent outputs equals your balance.

Quote

But how would one import their wallet key into an spv thin client backed by a pruned node and get correct balance? It needs tx that have been pruned?

Wallet imports the private key and derives a public address. Address is sent to a node, which finds all the unspent transactions for that address and returns them.

You might not get old transactions (if pruned), but you will get any and all coins that are available for spending, which add up to the correct balance.