Pages:
Author

Topic: A new genesis block isn't just for Solidcoin... as Bitcoin has the same ailment (Read 2724 times)

hero member
Activity: 530
Merit: 500

I want to add that the amount of HDD "swapping" on new blocks is unreasonable. It sounds like the client performs defragmentation.

Can't this be optimized? Cached? It's very annoying and doesn't do any good for the disk.

I've seen constant write-to-disk of about 1MB (megabyte) per second on both old and fresh installs by multiple users. I thought naïvely this was a Mac specific 0.3.24 problem (edit: but 4.0rc2 seems to do it a lot less). Is this what you mean above?

Constant disk writing slows down a lot of systems, and I advice against running Bitcoin if you have either a laptop and/or an SSD because of it. I'd say it's actually more of a problem currently than the size of the blockchain (although the latter is almost a deal breaker when it comes to showing how Bitcoin works to new users: "Come back tomorrow, if you punch a hole in your firewall that is").

vip
Activity: 1386
Merit: 1141
The Casascius 1oz 10BTC Silver Round (w/ Gold B)
...but to keep it reasonable, you must keep the chain below a max of 2gigs, then id consider it "too big" to download for the average person.

That's the problem.  The chain grows without limit, and its growth rate increases exponentially.  It's not impossible that we could have a 2 gig chain this year.
sr. member
Activity: 321
Merit: 250
Firstbits: 1gyzhw
dial-up, get real, it was hardly useful back in 2000 where almost all pages were bare.
most hard drives in laptops are now ssds.

we are moving forward in time, not back, so why would we directly cater to people who refuse to upgrade? i don't think 1 or 2 gigs is too much bandwidth to ask a mobile connection for, sure it will be slow, but thats a 1 time thing. but to keep it reasonable, you must keep the chain below a max of 2gigs, then id consider it "too big" to download for the average person.
You're part of a tiny, privileged minority of people who live in well-connected countries where the cost of living is high and technology is relatively cheap. This isn't true of most of the world.

edit: http://www.netindex.com/value/
sr. member
Activity: 350
Merit: 251
This is a valid issue :

-adoption is difficult for people on dial-up or mobile broadband
-HDD gets very hot in laptops


dial-up, get real, it was hardly useful back in 2000 where almost all pages were bare.
most hard drives in laptops are now ssds.

we are moving forward in time, not back, so why would we directly cater to people who refuse to upgrade? i don't think 1 or 2 gigs is too much bandwidth to ask a mobile connection for, sure it will be slow, but thats a 1 time thing. but to keep it reasonable, you must keep the chain below a max of 2gigs, then id consider it "too big" to download for the average person.
hero member
Activity: 518
Merit: 500
This is a valid issue :

-adoption is difficult for people on dial-up or mobile broadband
-HDD gets very hot in laptops
legendary
Activity: 2128
Merit: 1074
Well, miners happen to also create the blocks that serve as the transaction record, so making sure that they can properly tell the difference between a valid transaction and an invalid one unassisted is relatively important.
Again, long term miners just don't have place in the bitcoin ecosystem. Only refiners do the job you describe. Mining was sort of a temporary workaround for bootstraping the project. Satoshi was just very polite in programming an exponential decay of mining and leaving refining revenue to further modifications.

I used the word "peons" because in my experience it will force people to declare their intention and their psychological membership in the bitcoin caste system.
hero member
Activity: 868
Merit: 1008
Steve's comment of putting in just a hash of all the unspent transactions got me thinking...

What if that was just one more thing that all miners verified?  If the txid's of all the unspent transactions were arranged into a (very large) merkle tree and the 32-byte root hash were saved - PER BLOCK - then this validation would be network-wide.

If that were done, clients could receive pruned blocks.  They would not necessarily know what was missing, but if they were able to calculate the same unspent merkle root hash with what remained, they would at least know that they had all the transactions anybody cared about.

That's exactly what I was saying (though perhaps not so clearly).
vip
Activity: 1386
Merit: 1141
The Casascius 1oz 10BTC Silver Round (w/ Gold B)
The only participants of the bitcoin network that require trust are the exchangers. They will always have a cash flow to support the ancillary requirements of the network. The sooner you guys stop thinking about miners (and mining pools) as a centerpiece of the network, the quicker you'll arrive at a workable solution to this problem. Miners (of fresh bitcoins) need to be relegated to the status of peons. The actual value added is in the refining (collecting transaction fees) and the currency exchanging. Only exchangers have the incentive to maintain long-term business presence in the ecosystems. Pretty much everyone else is better served by being fly-by-night than the actual trusted peer. Only exchangers will submit themselves to a normal accounting audit.


Well, miners happen to also create the blocks that serve as the transaction record, so making sure that they can properly tell the difference between a valid transaction and an invalid one unassisted is relatively important.
vip
Activity: 1386
Merit: 1141
The Casascius 1oz 10BTC Silver Round (w/ Gold B)
Steve's comment of putting in just a hash of all the unspent transactions got me thinking...

What if that was just one more thing that all miners verified?  If the txid's of all the unspent transactions were arranged into a (very large) merkle tree and the 32-byte root hash were saved - PER BLOCK - then this validation would be network-wide.

If that were done, clients could receive pruned blocks.  They would not necessarily know what was missing, but if they were able to calculate the same unspent merkle root hash with what remained, they would at least know that they had all the transactions anybody cared about.
legendary
Activity: 2128
Merit: 1074
I kinda don't care about the specific algorithm to prune the block chain. But the block marking the pruning need to be called "jubilee blocks" or "Leviticus blocks", in the same way that the origin block is called "Genesis block".

http://en.wikipedia.org/wiki/Jubilee_(biblical)

But seriously:

The only participants of the bitcoin network that require trust are the exchangers. They will always have a cash flow to support the ancillary requirements of the network. The sooner you guys stop thinking about miners (and mining pools) as a centerpiece of the network, the quicker you'll arrive at a workable solution to this problem. Miners (of fresh bitcoins) need to be relegated to the status of peons. The actual value added is in the refining (collecting transaction fees) and the currency exchanging. Only exchangers have the incentive to maintain long-term business presence in the ecosystems. Pretty much everyone else is better served by being fly-by-night than the actual trusted peer. Only exchangers will submit themselves to a normal accounting audit.
sr. member
Activity: 350
Merit: 251
the main problem is picking somebody trusted to do it. you would have to have hundreds of people run the same calculations, and audit the code just to be 99.9999999% sure that nobody lost/gained any coins and such. the network could be in ruins for a week while this happened. while we are at it, we might as well recycle all the very small amounts just wasting disk space, like anything lower than .001 will get added back into the pile of coins yet to be claimed.

i think it should be done every gigabyte or so. right now that would be off the top of my head, about every 1.5-2 years.
hero member
Activity: 868
Merit: 1008
So you receive an incoming transaction... How do you know if it's a double spend if all you have is block headers?  If it has to ask the network about transactions, what happens if the network lies or the node being asked doesn't have a complete picture?  The block headers allow you to prove that something is on the block chain, but in this case, you are needing proof that something invalidating it is not on the block chain to know it is any good.  You can't do that without a complete copy, a trusted peer, or a trusted digest.
You don't have to trust any peers...transactions are identified by their hash...when you ask a peer for the transaction associated with a txid (i.e. you need to retrieve a transaction input), you validate what it returns by hashing it and verifying that it matches the txid.  If that transaction is part of an earlier block, then to the extent you trust the block headers that you downloaded, you can trust that the input transaction was valid (you could make a rule that you only trust the headers that pre-date one of the hard coded blocks in the client...probably hundreds of other possibilities as well).

I agree you can easily determine that the transaction exists, but what part of this validation process will allow you to confirm that the transaction has not been spent by a later transaction?
If a transaction appears in a block, you know that the network has agreed that it is the authoritative transaction to the extent that the network is able to agree to such a thing (reorgs, >50% attacks, etc).  If it doesn't appear in a block, well, you need to ask your peers if they think it's valid (if they've seen a conflicting transaction, they won't) or you wait for it to appear in a block (and both still carry degrees of risk when it comes to double spends).  A headers only client doesn't mean that you never pull the body of the block from the network...if you need to check that a transaction appeared in a block, you can ask a peer that question...the peer can respond with the portion of the merkle tree that proves the transaction is in the block.

Quote
The real benefit is simple: much less for users to download.
My point is that with a lobotomized client, you still have to place trust in network peers...so you may as well go as thin as possible and not bother with the block chain at all.  Just a very thin client connecting to one (or several if you want a little bit of distributed trust) full network nodes.
hero member
Activity: 868
Merit: 1008
Getting back to the topic of the OP...if you believe that bitcoin will flourish and be used for generations to come, then eventually this does become a problem.  I'm also interested in ways to allow the network to forget old transactions just out of principle (although, you should never assume old transactions are ever forgotten).  I think you can do this in a couple steps.  The first step is to establish a block depth, beyond which, the network will never accept a re-org.  This formalizes the current practice of hard coding recent blocks into the client.  It should be set deep enough that it doesn't present any real risk of block chain splits.  The second step is to add the hash of the unspent transactions to the block.  Nodes would have to validate this hash before considering the block to be valid.

New nodes joining the network could request the current block along with its transactions.  The node can then work backward, fetching each block and the transactions until it reaches a block that is old enough that it would never be subject to reorg (plus a few extra for good measure in case some of the rest of the network hasn't seen some of the more recent blocks).  It fetches all the unspent transactions that go with that block.  It can then validate going forward all of the transactions and blocks.  This new node will be trusting that other nodes have validated history up to the point in the past where it is starting from.  Care could be taken to ensure that trust is distributed among a large and diverse set of nodes by confirming everything with a large number of nodes (perhaps seeded by a group of nodes widely acknowledged to be trustworthy).  Another approach to this node bootstrapping would be to view it as a kind of cloning of existing nodes.  If Alice trust Bob, John, and Bill, then Alice would be confident in bootstrapping a new node from Bob, John and Bill and not have to place trust in the broader network.

If I'm not missing anything, nodes would only need to keep block headers and unspent transactions that are older than the block at which reorgs are prohibited.  Spent transactions that are older than the reorg boundary (plus some buffer) could be safely discarded.  The older blocks themselves could possibly be discarded as well (remember, all the unspent transactions can be verified to be in the chain by virtue of this new hash in the header).  I think for good measure you would want to keep a healthy amount of history beyond the reorg boundary, though I'm not sure it would be strictly necessary.  I even wonder whether this would work and be safe even on very short time scales if you retained a relatively long tail of block headers (i.e. a few hours for the reorg boundary).
vip
Activity: 1386
Merit: 1141
The Casascius 1oz 10BTC Silver Round (w/ Gold B)
So you receive an incoming transaction... How do you know if it's a double spend if all you have is block headers?  If it has to ask the network about transactions, what happens if the network lies or the node being asked doesn't have a complete picture?  The block headers allow you to prove that something is on the block chain, but in this case, you are needing proof that something invalidating it is not on the block chain to know it is any good.  You can't do that without a complete copy, a trusted peer, or a trusted digest.
You don't have to trust any peers...transactions are identified by their hash...when you ask a peer for the transaction associated with a txid (i.e. you need to retrieve a transaction input), you validate what it returns by hashing it and verifying that it matches the txid.  If that transaction is part of an earlier block, then to the extent you trust the block headers that you downloaded, you can trust that the input transaction was valid (you could make a rule that you only trust the headers that pre-date one of the hard coded blocks in the client...probably hundreds of other possibilities as well).

I agree you can easily determine that the transaction exists, but what part of this validation process will allow you to confirm that the transaction has not been spent by a later transaction?  

Still, I'm struggling to think of a real benefit of such digests (it is redundant information after all)...using digests would require placing some trust in the information provided by peers (full validation of the transaction history and block chain is required to avoid trusting peers).  It's possible this could provide some benefit to lighter weight peers (that don't have the storage, CPU or bandwidth needed for full validation), but then I think that problem is better solved by a lightweight client that just communicates with a full node that it trusts.  In fact, as I write this, I'm wondering whether a "headers only" client fills an real need either.  

Either you're trusting other nodes or your not...if you're not, then you need the full transaction history...I don't think you can escape that.  If you are trusting other nodes, then just pull only the information you need from one or more trusted nodes when you need it and don't even try to connect directly to the p2p network.

The real benefit is simple: much less for users to download.

Imagine block 150000 was a superblock that contained a digest of all the unspent transactions before it, and let's say it weighed in at 50 MB.  A client would only need to download the 80-byte block headers of blocks 1 thru 149999, which would be only 149999*80bytes, or 12 megabytes.  Total download burden to start using Bitcoin at block 150000: 62 megabytes.

Now let's say at block 750000, the block chain is 50 terabytes.  But as a superblock, it lists all unspent transactions in 1 GB.  Total download burden: 1 GB + 750000*80 bytes (60 MB) = 1.06 GB, versus 50 terabytes.

I am sure you would agree that having less to download is universally desirable.
legendary
Activity: 2128
Merit: 1074
In fact, as I write this, I'm wondering whether a "headers only" client fills an real need either.  
It fulfills psychological needs of some developers, and transitively some bitcoin stakeholders and promoters.

Whether those needs are "real" or not is a different story.

In my opinion they are very real, although sometimes subconscious.
kjj
legendary
Activity: 1302
Merit: 1026
i am not really a crypto guy: but wouldnt it be possible to bootstrap the thin client from a trusted node (maybe from home-pc) so it does have the latest 10 block digests.

as far as i understand you that would be enough to verify further transactions.

Sorta.  If you assume a really trusted node, you don't even need 10 digests, or even one.  But it really depends on how thin the thin client is, and what it is planning to do.

Consider a wallet that only holds keys and signs transactions.  In theory, it doesn't need to know anything at all, because it can ask for the transactions it is about to spend at when it is ready to spend.  The worst a hostile node could do at that point is present bogus transactions for the wallet to use as inputs, but that doesn't hurt the wallet any, just the recipient.  And the hostile node will almost always belong to either the spender or the receiver, and one of them would notice the problem quickly anyway.

As you ask the client to do more things, do more extensive verification, or trust other nodes less, it needs more data and needs to do more processing itself.  There is a full spectrum between the full nodes that everyone has now, and the superthin wallets that I think people will eventually be carrying around.
legendary
Activity: 1428
Merit: 1000
i am not really a crypto guy: but wouldnt it be possible to bootstrap the thin client from a trusted node (maybe from home-pc) so it does have the latest 10 block digests.

as far as i understand you that would be enough to verify further transactions.
hero member
Activity: 868
Merit: 1008
I think this has been discussed many times before.  I thought a "block headers only" solution was in the works.  In this case, a client only needs to download the block headers instead of all transactions.  As new transactions are received, it would pull input transactions lazily out of the network to do verification.  You need to take steps to ensure that there are still clients that maintain the full transaction history however.  

So you receive an incoming transaction... How do you know if it's a double spend if all you have is block headers?  If it has to ask the network about transactions, what happens if the network lies or the node being asked doesn't have a complete picture?  The block headers allow you to prove that something is on the block chain, but in this case, you are needing proof that something invalidating it is not on the block chain to know it is any good.  You can't do that without a complete copy, a trusted peer, or a trusted digest.
You don't have to trust any peers...transactions are identified by their hash...when you ask a peer for the transaction associated with a txid (i.e. you need to retrieve a transaction input), you validate what it returns by hashing it and verifying that it matches the txid.  If that transaction is part of an earlier block, then to the extent you trust the block headers that you downloaded, you can trust that the input transaction was valid (you could make a rule that you only trust the headers that pre-date one of the hard coded blocks in the client...probably hundreds of other possibilities as well).

Creating a digest of current account balances is still a good idea though...in fact, it might be a good idea to include a full digest of all account balances with every block that is created (the block header would have a merkle hash that points to the digest and only addresses with a nonzero balance would be included).  This way, no signatures or trust network is needed...the digest gets created as part of normal block creation.  In fact, clients could choose not to verify the entire history, but simply start with the current block in the network and validate forward from there.  They could even discard old transaction history this way (as long as you set a limit on how deep reorgs are allowed to happen).  The validation might need to reach back into history prior to the earliest digest that the client is using, but it could pull those earlier transactions as needed from the network.  I think you would just need to be verifying that input transactions did appear in an earlier block.

If these were done on an automated basis (e.g. every 1000 or 2016 blocks), then a client could feasibly only need the block headers from genesis to the last trusted digest.  Miners would probably need to be forced not to "trust" the digest and would need to verify it in its entirety before building on it, until it's at least a certain age (e.g. 1000 blocks), and then clients would only treat a digest as trusted if it's more than 1000 blocks old.  The digest is going to be huge (tens of MB or more) and can't reasonably be done every block of course, and the way I see it, is still probably a viable solution.

The digest for any block could be produced on demand by any node that has the full transaction history.  If you did something like this, keep it simple...define a standard format for the digest and have miners always include the hash of the digest in every block.  When a new block is found, other nodes would update their running digest with that block's transaction, then compute the hash of the digest and verify that it matches what is in the block.

Still, I'm struggling to think of a real benefit of such digests (it is redundant information after all)...using digests would require placing some trust in the information provided by peers (full validation of the transaction history and block chain is required to avoid trusting peers).  It's possible this could provide some benefit to lighter weight peers (that don't have the storage, CPU or bandwidth needed for full validation), but then I think that problem is better solved by a lightweight client that just communicates with a full node that it trusts.  In fact, as I write this, I'm wondering whether a "headers only" client fills an real need either.  

Either you're trusting other nodes or your not...if you're not, then you need the full transaction history...I don't think you can escape that.  If you are trusting other nodes, then just pull only the information you need from one or more trusted nodes when you need it and don't even try to connect directly to the p2p network.
administrator
Activity: 5222
Merit: 13032
Solo miners and pool servers, which need all unspent transactions, will probably eventually figure out some system of downloading only unspent transactions. A list of spent transactions will become "common knowledge", perhaps.

Clients are expected to mostly use headers-only mode, which is safe as long as an attacker doesn't have >50% of the network. Clients could also rely on the network only to verify spends of very old transactions, and verify new blocks themselves.

If it really becomes a serious problem, it's possible for >50% of the network to implement something like demurrage in a backward-compatible way, which would allow the network to forget really old transactions, even if they are unspent.
legendary
Activity: 1666
Merit: 1057
Marketing manager - GO MP
I don't think this will ever be a real issue, i seriously doubt the size of the blockchain will ever outgrow the size of the average harddrive.
Pages:
Jump to: