Topic: on average, how much HD space does bitcoin-qt consume per day (Read 13991 times)

I am sorry to say, but it appears that my suggestions, changes, GitHub commits seem to be not even worthy of comment

See
https://bitcointalksearch.org/topic/m.2708118
from July, 2013

The only thing I see is a new argument in bitcoin 0.8.5, to the command line (or bitcoin.conf) of -wallet

I see no document referenced, and have heard no mention of it, and looking at the code, in init.cpp I can't even be sure what it does.

My code, documents explicitly what it does and how. BTW, I have even more (and better) code that completely separates all the Berkeley DB wallet files so that they may be moved, renamed, etc. etc. But there seems to be more chatter about bitcoind (bitcoin-qt?) having the miner and wallet code removed in version 0.9? Perhaps a second and third separate program to recreate the original functioning bitcoind?

I posted this message, on the bitcoin-development list on sourceforgr.net:

No comments so far. And note that 2917 github link. Gordon wants to remove the wallet too! For my money, that would just about strip bitcoin of all of its power as a single program that is a full network "node" a wallet if need be and a miner if need be, and a server if need be. I should think that Satoshi would be spinning in his grave like a lathe, if he/she they were dead

After seeing (the first part of) Ted Nelson's piece,
http://www.youtube.com/watch?v=emDJTGTrEm0&feature=player_embedded

I don't see how one could think of striping the wallet and the miner code out of bitcoind or bitcoin-qt.

Just my $0.02

Ron

Or the issue will be pedantic because ASICs centralize mining hardware production into the hands of 2-3 chip fabrication companies... but lets assume not for the sake of argument, and anyway an alt-coin - perhaps one even duplicating the existing blockchain - can always change the PoW algorithm when it comes to that. There do appear to exist algorithms where the ratios of profitability between totally custom, "decentralized cottage industry electronics", and off-the-shelf aren't so extreme, but that needs further research.


The problem is more that there's no known way to create darknets with high-bandwidth individual connections, relative to the bandwidth possible on opennet connections, and there's no reason to think improvements in technology will do anything to close that gap. (if anything, the gap will probably widen due to the limits of stenography - there's a limit to how much bandwidth individual HD video streams plausibly need for instance) Hence my focus on determining if and how mining can be done co-operatively in such a way that any individual participant needs only a subset of the total bandwidth required for the system to operate, while still allowing participants to use pseudonyms and remaining resistant to attack by participants, and on top of all those requirements keeping the profitability for such participants as similar as possible to the profitability more centralized operators enjoy. Not an easy problem.

Incidentally, the fact that pools make any sense at all is something I'd like to see fixed.

Indeed I am... I can barely follow the most discussed topics here and on reddit. The devlist I barely read at all.

Thanks for the update anyways!


Perhaps the most authoritarian ones will attempt, but I don't think they'll manage. They can't even stop file-sharing. There will always be jurisdictions in which mining will remain an unregulated activity, and block generators would just put their servers there. I don't think that'll ever be a serious problem.


If you're really worried about this - IMHO you shouldn't worry that much - perhaps it'd be better to focus on more scalable darknet protocols instead. Perhaps using bitcoin for funding darknet nodes, I don't know... Anyways, it looks to me your problem here isn't simply "keep block generation decentralized", but more like "being able to communicate without significant latency and without revealing your physical location". In other words, a way for dissident pool operators to hide themselves in a highly totalitarian world. The solution is probably outside the Bitcoin protocol.

You're more than a little out of date... heck, I recall Gavin saying in #bitcoin-dev last Febuary or so that he personally considered a working fraud proof implementation to be a reasonable pre-condition to raising the blocksize. That's why I pointed out the censorship problem and asked people for ideas on how to solve it. (I haven't gotten any) Of course, lots of people don't see that as a problem at all: Mike Hearn has said he expects mining to be a government regulated activity in the future for example.

As for solving that problem - that is allowing for mining to be decentralized with a large blocksize - I've got a few solutions but they all have the really ugly problem that it's hard to be sure any "sharded" blockchain data is actually in the hands of more than one individual. Put another way, an effective attack is to ensure that you are the only entity with a copy of part of a block (perhaps by sybiling the network) and then keep that data for yourself. Without the data no-one can prove a later spend from that part of the UTXO set is valid or not, rendering those coins unspendable. Possible motivations include freezing someone's funds, or even charging high fees to spend those funds.

The best counter-measure I can think of is using proof-of-stake to ensure that any given part of a block was actually seen by some threshold percentage of the owners of those coins. But lost coins and apathy complicates this, and getting such systems working right would, as far as I can tell, require changes to the basic economic model of Bitcoin. Anyway without a ASIC-hard PoW algorithm those changes aren't necessarily all that useful anyway.

Do these changes make it upstream into the main repository?

Then for the sanity of the developers, and to "lower the learning curve" and to the benefit of people who would like to read the code and understand it, why don't we make this "complex C++ code"  more understandable?

Adding comments or a separate comment file, sort of like man pages for as many classes, methods, functions, variables, etc. etc. that we can.

We (developers) may actually find a few "loose ends"!

I for one find functions/methods that return a boolean true or false, that usually signifies success or failure, should really have a comment as to which return is which! Especially since all I see are magic numbers, magic boolean expressions, i.e. return 0, return 1, return (ret != 0) and other such vaguaries. One can discover what a variable, function, class does by studying the code. But is this what was intended? Does what it does match the names of the things that are doing the action? Sometimes one can infer what the action or the intent is/was of a function/method by noting what else happens, error  messages, etc. But intent is always a guess when there is no documentation.

But one should not have to infer how something works. It should be specified. Sometimes the intent is not realized by the function, or is the opposite, or is ignored and shouldn't be! The code doesn't know what you "meant" to do, only what is written. So I am guessing at the intent when I look at undocumented code. And I may be wrong.

I have taken to replacing the magic numbers: 0, 1, 144, etc. with more meaningful constants: Failure, Success, AverageBlocksPerDay, etc. where they are or seem appropriate (to me). I could go on & on about meaningful names for variables, etc. etc. I hate having to keep a symbol table in my head about "this' really means "that" when reading code. The variables, functions, properties, methods, classes, filenames etc. etc. can easily be renamed. I usually just do it myself!

I think that C++ code can be and usually is made so obscure, that one statement can literally need a page of comments to explain it. I think that C++ code should be mostly comments and every now and then a statement! Some say it is next to Forth in being a write only language

I think that a consistent indenting style (source code formatting) is also important in understanding.  I do see a coding standard (coding.md), but the code seems to still be a mixture of different styles?


Just my $0.02 on the understandability issue.

Ron

Wait, so you've managed to come up with a way for SPV nodes to satisfyingly verify that an "evil cartel of block generators" isn't defrauding everybody? The debate of whether to keep the crippling 1Mb limit for blocks or to let it grow by removing said limit is finally over?

That's brilliant! It's great to see you've managed to found some common ground. Seriously, no irony, that's a relief to me. I was confident an actual fork (with two different blockchains surviving) was inevitably ahead of us.

Point is, if you can prove fraud and publish your findings many people auditing parts can audit the whole 100% in an effective way.


I outline one such required modification here, and the fraud proofs system to make good use of it: https://bitcointalksearch.org/topic/inflation-proofing-via-fraud-notices-137933

UTXO set commitments are the other required modification; without them you can't create a compact proof that a transaction spent a txout that wasn't real. Unfortunately while maaku is working on UTXO set commitments he doesn't seem to understand the value of fraud proofs and partial validation. I've also got my doubts that he understands comp-sci and systems engineering well enough to take on the task for a variety of reasons: https://bitcointalksearch.org/topic/inflation-proofing-via-fraud-notices-137933 Hopefully what code he writes will be written in such a way as to make fixing the problems later possible.


New network messages will be needed yes.

What if the backdoor is in a piece you chose not to validate?
(just so this is crystal clear: I don't believe this is a serious danger, as I do believe in open source.. my point is just to show that the same applies to the blockchain, which is also open)


I'm not sure which modifications you're talking about (that long thread from the author of Armory perhaps?), but with the current Merkle Tree I believe this is already possible to some extent. I can validate the coinbase transaction only to certify no miner is inflating the money supply more than what's allowed by the protocol. I could perhaps also request past transactions in order to validate that the output crediting my address really does have the money it claims to have, all the way until a coinbase. By only doing that to transactions that concern me - and always checking the headers, PoW etc, ain't I doing a "validation in pieces"? If every user´s client does at least that, doesn't the chain gets entirely validated in the end?

No structural changes would be needed, AFAIK. Unless by "protocol changes" you mean new network messages in order to exchange such partial data, in case they don't exist already.


It's done continuously by a machine, not by human beings. The actual work of sporadically auditing the code is probably bigger than the work of letting the full node running.

caveden: You can audit the sourcecode in pieces, and there are plenty of people doing just that.

Similarly with some changes to the Bitcoin protocol you'll be able to audit the blockchain in pieces, which could lead to the elimination of a strict separation between full and SPV nodes in exchange for a model where nodes do more or less verification work.

In any case the source code doesn't change much, so auditing work once done remains valid for a fairly long time; the auditing that full nodes do needs to be done continuously. There's also issues of attacks and capacity - if we continue to lose full nodes you'll eventually find it hard to use SPV clients, and the network as a whole becomes more and more vulnerable to DoS attacks.

When you don't read and fully understand the entire source code of your bitcoin client, you are trusting developers to always follow the rules faithfully - to not steal/spy from/on users by inserting a backdoor etc. But if users are not reading and fully understanding the source code, because they're incapable or not willing to do so, what incentive is there for developers to behave quite so faithfully?


I'm repeating myself, but it seems important to do so: there will always be more honest people willing and capable of running a full node than there will be honest people willingly and capable of reading and understanding such a complex C++ code as that of bitcoin. The barriers are incomparable. Running a full node is simple, and will be always cheaper than either hiring a C++ expert to do the code-parsing for you, or going through the huge learning curve yourself. Actually the option of hiring someone only transfers the trust, so you have to go through the learning curve if you want to be 100% sure. There's no multiple years learning curve necessary for someone to run a full node.

So, please, let's quit the drama. If the "everybody can validate the source because it's open" sentence is really true and justifies the trust we have on open source software like Bitcoin, then there's no reason for the sentence "everybody can validate the blockchain because the data is open/public" not to be equally true, and an equivalent justification for SPV clients to trust in its integrity.

They do— they trust the peers they speak to more than a regular node does. E.g. if they show unconfirmed transactions they trust their peers to only tell them about valid unconfirmed transactions. When they receive blocks, they trust their peers to only tell them about valid blocks (and failing that, miners to only produce valid blocks. Especially when they display 1-confirmed transactions as "confirmed"). I don't believe _any_ SPV node software has a secure peer discovery implementation either, which is frustrating because they are somewhat more vulnerable to naughty peers.

IIRC electrum connects only to a single server at a time. It's SPV from there out but because it makes a single point of attachment it's pretty vulnerable. It also does no SSL certificate validation.   BitcoinJ wallets are differently bad: They connect to multiple, but they're only what get returned from DNS. Of course, the Bitcoin protocol itself is unauthenticated, so a network attacker near the node can isolate it easily.

Obviously SPV wallets will improve in the future. ... But they are no replacement for full nodes. If it wasn't important for many regular users to run full nodes then we wouldn't need full nodes at all.  When you don't run full nodes you are trusting miners to always follow the rules faithfully— to not steal from the users by inflating the coin, etc.  But if users are not rejecting unfaithful blocks, because they are SPV and can't validate them, what incentive is there for miners to behave quite so faithfully?  Will they watch themselves, mutually distrutful?  Not so clear, because of cloud mining and pooling the majority of hashpower is under the control of approximately 3 people right now. ... and increasing their income would be uniformly beneficial to them.

In classical currencies and banking the people with equivalent trust are the central banks and governments. There are many reasons the behavior of fiat currencies and banking in democratic regimes trustworthy: They are highly regulated, privileged institutions— not anonymous entities like mining— arguably controlled by elected officials.  And yet they are observably not trustworthy, and their lack of trustworthness— the lack of any institution depending on human fallibility— was a motivation for Bitcoin's creation.

But if you leave network security to a handful of self-appointed anonymous miners without the check of tens of thousands of regular users with uncorrelated interests outside of Bitcoin's integrity then the result is something even less trustworthy. I think that Bitcoin's value argument is too weak to be worth keeping it around if it doesn't offer an integrity improvement over the alternatives.

Then perhaps you should read it? It's eight pages long and written in plain English.

At times I want to start banning people from this sub-forum who haven't read it. It is far from answering all questions, but there really is no excuse not to have read it.

A good marketing campaign...if not a modestly technical wiki page with some appropriately designed jargon...can do wonders for turning a bug into a feature.  In a similar vein, one man's weakness is another man's strength.  There are plenty in the community who are probably completely happy to consolidate the infrastructure to a comfortable set of operators and get on with the business of running an economy with a proper level of oversight, control, and in some cases with a fair level of value extraction.

I beg to differ. The growth according to the graph of the link given, is more or less linear at about 10-11 GB /year (currently), so ~100GB/decade, ~1TB /century.

There will be spurts of growth in places like Cyprus, Argentina, and for the opposite reason in places such as Hungary, Iceland...Then plateaus of "tranquility"

As to how long to download, not too long. With a "good" internet connection it is easy to get 1 new block in ~<10 seconds. Now there are ~6 blocks created /hr, i.e 144/day. So to catch up for one day, you need 144 * 10 seconds = 1440 seconds = 1440/60 = 24  minutes. Notice about one minute for an hour of catch up. So a week of catch up ~ 24*7 minutes = 168 minutes  which is < 3 hrs. So a month catch up is about 3 * 4 or 12 hours, 2 months catch up will take ~ a day,  ~a year ~ 6 days,  and you can estimate further if you like. It's not that bad. It's actually better! The first blocks being small come really fast, so it takes about 3 days, more or less to get the whole block chain.

What this should tell you is that a "good" block chain is a lot of time saved if you need to start up bitcoin on a new machine! So save the files in those 3 directories:  [datadir]/blocks/*.*
[datadir]/blocks/index/*.*
[datadir]/chainstate/*.*
that comprise the levelDB database! It is time well saved. Almost worth backing up too. Since time is money for some, the block chain may be worth more to you than what's in your BTC (or is it now XBT) wallet!

Ron

I don't think he "envisioned" that as much as "saw" that it was the only way it would work. It does not scale particularly well, and this is a weakness, not a feature. How weak a weakness is certainly debatable.

Here, read this https://en.bitcoin.it/wiki/Scalability

In short, you don't need the full copy of a blockchain to use bitcoin. Plenty of light clients just ping random nodes and request only the address balances that are relevant to them. Also, the blockchain can be pruned to only the addresses that have balances in them. If that was done today, the 12gig blockchain file would shrink to only something like 200 megs.

Satoshi envisioned that eventually only few copies of the entire blockchain will be stored for archive purposes, and large miners and pool operators (and wealthy hobbyists/supporters) will be the only ones keeping copies of the blockchain, with the general users just relying on the light clients that only request address balances, and sign/transmit transfers to the network.

"Depends"??   I don't think Satoshi left this to a random uncertainty. What's the existing architecture designed for? Each node containing all blockchain or is there an alternative option where larger miners would contain a larger section of the blockchain in the future for example?

I can't think of a different solution to deal with the infinite storage issue.  For example, does it make sense to purge the blockchain? or dynamically compress it? I'm just wondering what Satoshi actually envisioned in his paper.

Depends, each day it grows ever so quickly!

I don't think I've found the answer before but is it scalable to expect that the blockchain is downloaded to personal computers??

Today, the total volume of transactions is immaterial relative to other currencies, but say 5 years from now it represents a very large percentage. In that case, it would be ridiculous to have to download the entire history of transactions from day 0 in all computers. Unless there's a revolutionary technology that will allow the convenient storage of such info without requiring massive storage devices, it just doesn't seem practical to do so.

Perhaps I'm missing something but what did Satoshi think about this? How is this part of Bitcoin efficient? Thanks.

If you think that's a problem, you are free to join github and add/critique all the suggestions on there. You are also free to come up with ideas and try to convince others that they are good, so that some developer somewhere can implement it.
I think you are under the impression that Bitcoin development consists of Gavin and maybe 3 other people. That's not how this project works. There are tons of suggested features and actual code suggested by a whole slew of people from around the world, and those features and code gets reviewed by others, and if proven to work right, get implemented in various clients. The core developers just check and fix the code and sort of herd the cats, in some attempt to keep things organized (plus write their own code, too, of course).