Topic: Block Chain Summary or Ledger or Balance (Read 1727 times)

After re-reading this thread more thoroughly, I came to some other conclusions.


An estimate I made earlier:


AFAIK the only things dropping tx history prevent you from doing are
A: Using the blockchain as a permanent timestamp server or as a file server, both of which are outside the scope of bitcoin anyway, not to mention would contribute considerably to size.

and

B: Long term data mining of the blockchain, which would only be used by law enforcement (who could still retain as much as they wanted) and identity thieves.

As far as long term goes, I think it's relevant to compare BTC to say, VISA. VISA does, on average, the equivalent of 1.2 million tx per block, or around 360MB per block, or 52GB per day. For this they have one to a few large centralized datacenters, which are in turn served by one to a few large digital clearing houses. Each of these has some absurd internet hookup for handling all of the necessary dataflow, paid for by near-draconian fees which are usually paid by merchants and thus are paid for by, although opaque to, consumers.

BTC, on the other hand, is extremely extremely redundant redundant redundant. Every independent miner and every full client carries the full weight of documenting the transaction history as well as the full network load for outstanding tx. In order for the BTC network to handle the same sort of tx volume as VISA, every miner must then have the full network capacity of VISA as well as the storage capacity to maintain an indefinitely large transaction history.

Ultimately this is unavoidable for any secure payment system, however for BTC there are other considerations that are quite relevant even right now. For example, due to the size of the blockchain, most miners (or at least most of the hashing power) would rather have DeepBit take a big chunk of fees and do all the tx verification and blockchain storage for them, rather than store the whole thing and get no-fees-plus-donations on P2Pool.

The pruned blockchain model doesn't fix this very well, and ultimately the idea pushes tx verification onto centralized mining pools, with clients being forced to request merkle tree data from tx sent to their accounts in order to verify it. That would only add further redundancy and strain the network capacity of the centralized nodes moreso than VISA's servers deal with currently.

More importantly, the only thing resisting the tendency of increased centralization of BTC and direct reliance on cartels is Moore's Law. Quite obviously, BTC user demand has the capacity to grow much faster than Moore's law. Reducing the per-tx network requirements moderately and reducing the per-tx-per-block storage requirements by more than an (computing) order of magnitude would be a big step towards weighting the trend back towards decentralization. Fully (or at least mostly) solving the centralization problem can only be done through the block reward, since tx volume tends to scale with market cap, such that the value of the block reward scales with the BTC economy. IMO fees are incapable of supporting the network due to the level of redundancy present, and will only tend to reduce its use value.

I had that same idea. By itself it wouldn't save blockchain space by much, but condensed into a ledger it could allow you to keep much less blockchain for basically the same security. It would also work well combined with your proof of stake concept, using ledger hashes every, say 138 out of 144 blocks, with every 144th being signed.

I have been schooled. Thank you. I am just beginning to understand how bitcoin works, so my ideas maybe were borne out of frustration. Incidentally, I have looked at alternative light clients, and they might do what I want already.

As for the summary, we take the most trusted blocks, meaning, the most recent blocks are not included in the summary. We put an arbitrary number on what is considered recent (1 to 6 blocks ago) and what is considered secure and stable (100 to 1000 blocks ago.) I just thought that the bitcoin community would consider that 6 blocks ago is fairly secure, and anything older is exponentially more secure. But after a certain point, how much more secure an older block is compared to a not so old block is just a large number.

On an unrelated note, the world is flat. There's even a book on it. The cliff's notes / baron's notes version of that is the world is smaller, we can connect to more people than ever, so it's as if the world is flat. But only figuratively. The world is still physically as round as it has been for the past billion years. We just have wires and satellites crossing every country and continent now.

To me, the idea of a server acting like a gateway, similar to Visa/MasterCard, is our best bet for fluid and easy transmission of the updated blockchain to your mobile device.

Good point. If summary disagrees with full history, which one is authoritative?

In current system, transactions only get into block chain once and keep ageing. Old transactions thousands of blocks ago don't need to worry much about current block chain attacks. With summary ledger, they can be messed with at any time.

Current system, only a previous owner of the coin can mess with it. Even under 51% attack, only previous owners of coin can try to double spend... With summary ledger, we take some miner's word for it.

Don't look at the source code, look at BlockExplorer, that's the best way to understand how it works.

Again, this can be included in a new protocol (which allows forcibly merging several outputs of the same address), but in the current protocol, knowing the balance of every address is useless.

No once again.  Bitcoin has absolutely no concept of balances.  It only knows about outputs.  The input of every tx is the output of one or more prior txs.  You can't combine outputs only the private keyholder can and that would simply be a new tx.

A->B 1 BTC
C->D 1 BTC

these are discrete and immutable constructs.  Nobody can change them.

Now the owner of B & D could make a new tx which
B & D -> E 2BTC

at which point A, B, C, and D could be pruned however the only entity which could create that tx is the private key holder of B & D.

You can prune dead end outputs (ones that have already been uses as the input in a subsequent tx) but you can do that by pruning the blockchain and making a summary provides no significant space or bandwidth savings over the sum of pruned blocks.


That is obvious and looking at source code isn't a requirement however when someone says the network doesn't track the balances of addresses and you keep going back to that flawed thinking well it kinda makes further discussion pointless.

It would be like
You: "see the world is flat"
Me: "no the world is round"
You: "ah I get the world is round, but lets say the world is flat then you can't sail around the world"
Me: "but you can because the world IS round"
You: "I know I haven't looked at the source code of the world but say you modified the world so it was round"
Me: ...


The blockchain can be pruned to achieve that without creating summaries.  In the future it would be possible to have nodes requested and provide pruned blocks.  Still the savings is not as much as you think.  Roughly 25% of all outputs haven't been used in a subsequent input so even pruning all redundant information would only reduce the block chain by a factor of 4 not a factor of 33 as you incorrectly indicated.

Oh, okay. I will study more about how this output works as opposed to just plain balances.

But allow me to add another thought to the main idea.

The summary block will be computed just like any other block (that is, secured by the hashing power of miners). This summary will only be computed for all the previous blocks except the most recent 6 (or most recent hour, or maybe day, 2 days, week, month, whatever is acceptable as safe and secure by everyone.)

On the other hand, maybe I understand this wrong, but since balances are not really balances but outputs, the general idea is the same, to compress all these outputs.

Working on a your local copy of the block chain up to 6 blocks ago, you can get all the outputs, then summarize them into 1 output per address, and assuming all addresses are reset to zero.

Let's say this represents the block chain for one address:
1 + 1 + 2 + 2 + 3 + 3 = 12

This should represent the same final result:
0 + 12 = 12

Ok, I am also a newbie, so if this is all wrong, then maybe I don't really understand how bitcoin's internal system works. I have never looked at the source code, I just read everything on wiki that a reasonable normal person would read over the course of 2 or 3 weeks.

The space saving I had in mind is precisely what people think it is, I am just storing address balances. It's like I have a modified bitcoin client that has all the known public keys with non-zero balances. And again, this is all based on the block chain.

Well you can't forget outputs because they are what is used in new txs.  You don't send an amount you transfer outputs thus to be able to verify future outputs you need to have a record of all outputs with value not just balances of addresses (which is completely useless).

So while you could prune off all dead end tx chains you need to save every output which has been unspent.  Combining that into a summary wouldn't save significantly more space that simply providing the pruned blocks.

Lastly your "space savings" of 1GB -> 30MB is not even in the right magnitude.  It makes people think you are just storing address balances.  There have been analysis done which estimate the space/bandwidth savings of to be more like a 75% reduction not a 97% reduction.

The reason, it seems, most people want the blockchain to "shrink" is initial use, and underlying database processing bottleneck. databases can be made to go faster.

1-2GB is a lot to download now, and 10GB a good amount in the near future, but I really don't think that with the addition of bigger, better, faster, more high speed links, this is an issue. I don't think it's really a storage issue long term either (ok, maybe at terabytes, but we can burn that bridge when we come to it )

Changing protocols (thus the network) is not (ever) an issue to be taken lightly, IMO, and should only be done when absolutely necessary. While this applies to all major network protocols, in the case of bitcoin, changing the protocol means that you have changed the agreement that people have made about how their money works - that is not an act to be taken without great thought.

If there's no impending doom from blockchain size, it's not clear that it needs to be fixed. If there is, I guess we'll have to fix it, but on the time schedule requested by the doom, not any more hastily.

There's no such thing as a system with no bugs, I think, and any addition to code is likely to introduce new bugs. Bugs in my web browser can be exploited to steal my money. Imagine what bugs in my money expose me to  (is runaway inflation a bug? we got that in USD lol)

I'd rather see things fixed in clients, services, and add-ons that enhance usage of and experience of the core network functionality without changing it. Then, when no other options remain to deal with issues of true necessity, we can change the network. The further along we go, IMO, the more we will have to put into place more formal means of achieving consensus on such changes...but that's another post.

What you can do, is forget the used outputs.
Only problem with that is you don't have the complete blocks anymore for new clients that want to download the blockchain.

I think the fact that Bitcoin works on outputs, rather than addresses, is lost on you. An address is merely a representation of the terms under which an output can be spent.

If your client wants to work with the existing network, it needs to understand the language of outputs, and it needs to only accept a transaction if the network will agree that its inputs are valid, unspent outputs.

If your client receives an incoming transaction, even if it knows that the sending address has a positive balance, if it doesn't know where the balance came from, it can't tell if the new transaction is really a re-spending of an old output which was spent long ago. If it is the network will not accept the transaction.

As far as I can tell this is only possible with a fundamental protocol break, which may have many unintended consequences, and is of questionable utility. I wouldn't bet on it being implemented in Bitcoin ever, and definitely not in the next few years. It might work for an alt though.

FWIW, I myself am a proponent of some fundamental protocol breaks which I hope will be made one day, but for those I can make a very strong case for their necessity.

The summary is based on all the previous blocks. Then it is protected and broadcast across the network (assuming this idea becomes part of the protocol; or if it can be an addition to the protocol without breaking it.)

My client isn't the same as other lightweight clients in that it knows how much are in every address that has a positive balance. The double spending is prevented because I'm getting the exact same data from the block chain up to that point in time, because for all intents and purposes, it's just another bitcoin client with the latest blocks, just like any other normal client.

The difference is, my client doesn't know the history of how the bitcoins ended up where they are now, or said another way, my client has forgotten what happened 3000 blocks ago, but knows for sure what all other clients know about transactions and addresses and what balances they contain presently.

For example: in Block 20, 1aaa with a total bitcoin of 50 sent 50 BTC to 1zzz. If nothing changes between then and when the summary block gets created, in Block 170000, 1aaa has 0 BTC and 1zzz has 50 BTC. How that happened, the client doesn't know, but the client knows that's how much these addresses should have with the same certainty that 6 blocks confirms your transaction 1 hour ago.

I don't know if the idea breaks protocol or if it can be inserted while making it still compatible with regular clients. All I know is I want the block chain to shrink, while maintaining the security of bitcoin, and simultaneously adding a little measure of privacy or anonymity.

@Dabs: If you're not changing the protocol, can you specify exactly what can your client do that a standard lightweight client can't? In case the answer is "verify that incoming transactions are valid", how exactly does it do that if it doesn't know what are the outputs contributing to a summary, and thus can't know if the same output is spent twice?

To make a summary, you need the previous blocks. The summary is then hashed and protected with the same protocol as all other blocks. After 6 blocks, the summary can be considered confirmed and safe to use. The previous blocks can then be discarded or become optional. Basically, the summary block can be a giant block of transactions that start from zero and puts the current balances of each address as a transfer? Just another idea. The summary is or can be part of the block chain, simply making previous blocks optional. Transactions will still be verified or confirmed by clients as it does now.

The next summary will rely on the previous summary and the succeeding blocks.

Right now, I can probably make (or someone who knows programming) a client that does this without affecting the network. The client will accept new transfers to me, and the client will broadcast transfers from me to another address. Invalid transactions will be rejected by the network.

The alternative is a mobile device (such as a cell phone) which gets updates from a server (either at home or a third party.)

For the home version, I would be running the regular bitcoin client on my home desktop machine (or office, whatever) and it has another program (or this can be all integrated) that receives and sends SMS (text messages) or BBMs (Black Berry) to my mobile device. The home version can be trusted since it's my machine, with my security and my passwords etc etc.

The third party, well, you'd have to trust them.

If bitcoin becomes big and we have a billion addresses, the summary will still be a lot smaller than all the transactions for those billion addresses, basically shrinking to 1 transaction per address.

Does that make sense?

Two issues. 

The first is that pruning results in almost the exact same savings and is optional.

The second is how do you ensure the summary is accurate?  To verify the summary requires having all prior blocks anyways.

The reality is that over time MOST users will use lightweight clients which don't have the blockchain at all.  The nodes which do keep the blockchain (miners, paranoid users, banks, exchanges, merchants, service providers, etc) will want full functionality.

If the client doesn't have the blockchain, it can't verify transactions. Having just the "balance" is useless because there is no protection against using the same output twice when there is additional balance available.

If it is a lightweight client that doesn't verify transactions, and relies on someone else to do it, then it doesn't need to download blocks in the first place, or to know balances. It just needs block headers and relevant Merkle branches.

With a hard protocol change you could do something like promoting the summary balance to an effective output which can be spent.

I believe this can be done on the client (or any alternate client) without modifying the network.
As it fetches new blocks it simply updates its ledger, instead of storing the entire blockchain (although it should store say X number of the latest blocks for security).
If the client is not mining, why would it need the entire blockchain?

1. Going forward bandwidth is likely to be as much of a problem as storage. A client working with summaries will still have to download all new transactions, and thus will be unsuitable for cellular internet connection.

2. If Bitcoin becomes ubiquitous there will probably be billions of used addresses, so summary storage will still requires tens of gigabytes.

3. Rapidly dropping transaction history will limit what you can do with the Bitcoin system.

4. Gains are likely not to be much above what pruning can do.

I don't see how this improves on the (future planned) status quo of full or pruned blockchains on servers, lightweight clients on devices.

Reference threads that I can not post to yet.

Proposal for reduction of storage space on all clients
https://bitcointalksearch.org/topic/proposal-for-reduction-of-storage-space-on-all-clients-12376

With "Balance sheets" most of the block chain can be forgotten.
https://bitcointalksearch.org/topic/with-balance-sheets-most-of-the-block-chain-can-be-forgotten-505

Transaction pruning details
https://bitcointalksearch.org/topic/transaction-pruning-details-10663

A proposal for a scalable blockchain.
https://bitcointalksearch.org/topic/a-proposal-for-a-scalable-blockchain-52859

My Idea on the bitcoin block chain summary.

The block chain is big. 1 gigabyte big. In about 8 years it can become about 4 gigs.

The block chain is basically a ledger of all transactions since the beginning. If we follow the analogy of accounting metaphors, we should also have bitcoin summaries, or ending balances, or closing entries for the previous periods or blocks. This can be daily, weekly, or monthly, for example.

This summary or closing balance block will only contain bitcoin addresses with positive values as of that block, or as of that point in time, thus compressing the block chain from the 1 gigabyte it is today, to maybe 30 megabytes, or some number significantly smaller. The number of bitcoin addresses with something in them (meaning, at least 1 Satoshi) is about 600,000. It takes about 50 bytes (or less) to store an address and its balance. That's how I came up with 30 megabytes. Half a million addresses with balances might be 250 megabytes. This is not using compression or optimization of how the data is stored. I mean, an address right now is 160 bits or 20 bytes and a balance of 500000.00000001 bitcoins can be stored as a 64 bit number or 8 bytes. You can now store 10 million addresses with balances in 133 megabytes. The 600,000 addresses now can be stored in 16 megabytes.

Addresses that have zero balances will simply not be included. And I think there are no such thing as negative balances.

As it is today, a modified bitcoin client can do this block chain summary for itself, and use that as a starting point. Sort of a selfish client. The difference is that it has the balances of the entire block chain, not just of its own public keys or addresses. Adding a key to the wallet file and doing a -rescan will yield identical results for the original client and this modified client.

The client will lose the ability to look at the previous transactions prior to the summary, or perhaps two or three summaries ago. But for most people this might be acceptable.

With a smaller combined block chain and summary, this will bring bitcoin to be more accessible to small devices, cell phones, ipods or ipads, androids, netbooks, etc.

People who like to consolidate their bitcoins into a few addresses will do the world a favor.

The client will of course compute if this block summary is valid, then hash it or whatever it is that miners do.

Desktop computers will be encouraged to retain the entire block chain. Small laptops or small devices will use the block chain summary.

We can extend this further, by having both the monthly block chain summaries and annual summaries, where the annual summary takes over the block chain. Or some other longer length of time, most probably determined by how large the block chain will be. This way, we don't get uber giant terabyte block chains in 20 years and bitcoin will survive into the distant future.

What I am proposing will also effectively put a hard limit on transaction history to say, 5 years worth. Meaning, we will never see transactions going back 6 years ago. But only for those clients that discard the old data.

Attempts to produce a fake summary will of course be rejected by the network, but at least the network is spared from giant block chains, and even with a limit of 1 year, the current block chain is still very secure.

If we like to assume that 6 blocks or 1 hour confirms our transactions, then 4000 blocks or 1 month really confirms our transactions. The difficulty now is 1626553.4813289. The same difficulty can be applied to the miner computing the summary. 6 blocks later, the summary is confirmed and all the previous blocks can be discarded or deleted.

Incidentally, the difficulty gets adjusted every 2016 blocks. That's about 2 weeks worth.

2000 blocks takes up a lot smaller space than 173386 blocks.

So, in theory, we need never keep transaction histories longer than 1 year, or 6 months, or even 1 month. Thus keeping the total block chain and summaries to a reasonable size.

To visualize this, the current system is something like this:

Block 1, Block 2, Block 3, Block 4. . . . . Block 173386.

or to make it even easier to visualize, I will abbreviate it to something like this:

B1, B2, B3, B4, ... B173386. Where B is a Block.

The new system or protocol or implementation would be something like this:

B1, B2, B3, S1, B4, B5, B6, S2, B7 ... S99, B173384, B173385, B173386. Where S is the Summary block.

for something that computes the balances every 3 blocks. The client can then keep the latest 144 blocks with the 48 summaries, and discard everything else, essentially keeping track of the current state of the network as of 1 day.

Clients that are new to the network will not need to download the entire block chain, but the latest summary and blocks. These blocks and summaries will be considered confirmed (and safe and secure) if they go back at least 6 blocks long, isn't it?

Currently, if someone wanted to attack the bitcoin network, he'd need 51 percent of the hashing power, and he'd need it for at least more than 6 blocks worth, and keeping at the attack for a few days. That or someone simply finds a way to compute private keys faster than brute force and steals all our bitcoins.

There are at least two side effects of implementing this block chain summary.

1. clients will be unable to keep track of very old transactions unless the client saves this information.
2. since summaries no longer contain transactions, better anonymity is preserved for older transactions that are no longer being retained in the current block chain.

I say better, because there will be people or entities out there who will still keep the entire block chain, for whatever purpose it will serve them, or simply because they have terabytes of space for it.

But most people will "forget" about old transactions. And depending on consensus, I think that is fine, and some people will probably even like it.

Take note that this suggestion or idea is separate from the Merkle Tree for selfish clients. So you have two methods to shrink the block chain, but mine still retains the records of all address balances. You can even combine the two to have really small selfish clients.

Summary: secure small block chains with the side effect of slightly better anonymity.

If you've read this far and like this idea or think it has merit, you may tip or donate to 1CAnhNTL7n78U9rRHyC1LqbxVbFBWFfSTU any amount will do, no matter how small.

If this can't or won't work now, then I propose this for the next version of bitcoin. But I'm thinking there is a way to do this, especially when you have everyone upgrade their bitcoin client to version 0.5.4. (or whatever version includes it.)