Topic: Bitcoin-QT 0.83 blockchain download / wallet management (Read 1436 times)

Thanks for the Satoshi paper download suggestion. In fact, I did just download it a little earlier and was surprised it wasn't too many pages. But right now I think you can understand it's a bit of information overload for me as I'm trying to get some solid coverage of the basics in a few important areas (wallet, transactions, blockchain, etc) just so I feel that I have a good feel for what's going on. But I am aiming to get to the paper and read most of the parts that I would understand. Your input so far has been very helpful to reaffirm various points I've picked up from many different articles and reference pages, but sometimes when I'm new to something I just can't be absolutely sure I'm interpreting the materials I'm reading correctly and I guess that's why we have forums like this.

Generally, yes.


If you haven't done so yet, you should really consider reading the Satoshi Whitepaper.  Is answers so many of the questions you've already asked. In this case, it answers your question about the reduced security of a thin SPV client:


Correct.


Correct.

Thanks for the detailed explanations of the transaction process.


I noticed that other wallet thin clients don't need the entire block chain locally on disk. Do these clients access blockchain data as needed on the fly via other wallet client peers? What are the (potential) disadvantages of thin clients compared to thick clients in terms of security and integrity? Eventually, I would like to use Armory when they release the forthcoming new Windows version that will run on systems with lower RAM (present Armory version requires 6 GB RAM or higher). A feature I like about it is multiple wallet management. From what I understand, Armory runs ON TOP OF Bitcoin-QT (perhaps as a UI shell and also adding its own Armory functionality?). I'm assuming that QT client must be present for Armory to work.



Right, so overtime, the wallet.dat file will gradually accumulate all used key pairs and unused new key pairs generated by the wallet client, and there are no key pairs ever deleted from the wallet.dat file key history, which is accumulated from first use and thus the wallet.dat file size will continue to slowly grow, if I understand this correctly.



As the wallet.dat file integrity is so critical, I personally would plan to use a rotating back up scheme with multiple sets of back ups. Of course, the back up frequency will need to be adjusted to be appropriate for the rate of which the number of key pairs are being used up, so that the number of newly used / consumed key pairs that occured since the most recent back up, remains only a fraction of the keypool size.

A transaction is supplied value from one or more "inputs", which are just a list of previously unspent "outputs".  A transaction assigns that value to one or more outputs.  Each output is an address and a value assigned to that address.  The rules of the protocol (that are enforced by every peer that relays the transaction as well as every miner that attempts to add the transaction to a block) require that the total value of all inputs in a transaction must be greater than or equal to the total value of all outputs in the transaction.

That tree view you are looking at is an abstraction that tries to help you visualize the movement of value as transactions occur.  It doesn't show you exactly what is going on in each transaction.  If a transaction receives value from multiple inputs that were all associated with the same address, then the "tree view" totals them up and shows them as a single "node" even though they are actually separated in the blockchain.  If a transaction receives value from multiple inputs that are associated with
different addresses, then the "tree view" only displays the value provided to the transaction from the address you are viewing and ignores the value from the other addresses.

So while it is a useful tool for tracking the movement of value, don't rely on that tree view to understand what is actually happening within transactions or within the blockchain.

Note that I stated earlier that a transaction assigns value to one ore more outputs.  In Bitcoin-Qt you'll see, in the "Send coins" window, an "Add Recipient" button.  This button allows you to add additional outputs to your transaction. Doing so would cause the tree view to split into more nodes, since the transaction would assign value to each of the recipients as well as the change back into the wallet at a new address.

Even the things I'm saying so far are an abstraction. While useful for understanding how the blockchain works, my description of transactions outputs as "a value assigned to an address" is an oversimplification.  In reality, a transaction output is a value and a set of instructions (known as a script) describing what criteria must be met to include the output as an input in another transaction.  In the typical case, these instructions set up a requirement that a signature with a private key associated with a particular address must be provided in order to include the output as an input to another transaction.  As an abstraction, we say that the output assigned the value to that address.


There are a variety of things the Bitcoin-Qt application does as it starts up to verify the integrity of the blockchain, and the wallet.dat file. Bitcoin-Qt makes sure that the data haven't been corrupted or manipulated while it wasn't running. Much of the start up time was spent on that verification.


It is clear that the priority for the developers of Bitcoin-Qt as a reference client is maintaining the security and reliability of the protocol and not user-friendliness. There appears to be an expectation that other wallets will fill the user-friendliness gap.


I'm not sure I understand your question correctly, but unless you manipulate the contents of the wallet.dat file, it will accumulate all the keys that are created by the wallet program while connected to that wallet.dat file, regardless of whether you use those keys or not.  If you click the "New Address" button in the "Receive coins" window 30 times, then the wallet.dat file will accumulate an additional 30 addresses. If you send another 10 transactions that each have change, then the wallet.dat will accumulate another 10 addresses.


Bitcoin-Qt does not fill the key pool in batches.  It does it immediately as keys are removed from the pool.  So if you click the "New Address" button, the wallet program pulls the next key from the key pool queue, and at the same time it generates a new random key and adds it to the key pool queue, so that the number of keys in the pool remains constant.  Change addresses are handled the same way (pull the change key from the front of the queue, and simultaneously generate a new random key to add to the back of the queue).


Correct.  It can be reduced by either backing up more frequently, or by increasing the keypool size.  I try to keep a general idea of how many addresses have been used (the sum of how many times the "New Address" button is clicked and the number of outgoing transactions) since my last backup and then create a new backup when the number of addresses used is about one fourth of the keypool size.  Then I keep the 3 most recent backups.  This allows me to recover all of my currently used keys from any of my three backups if one of them becomes damaged or otherwise unusable.

Thanks for this, which was a useful summary. Looking at the tree structure of transactions posted on blockchain.info, many of them branch into two and sometimes more branches from the original value (what is called the transaction input value I suppose), and often in a two branch tree one branch will be spent and another is unspent. I would suspect the unspent new child node is often the "change" value that is returned to the payer / sender of funds under a newly generated identifier / address, if I understand this process correctly. I'm not sure under what circumstances a parent node can split into more than three child nodes however.

Today I started up the Bitcoin-QT client manually and it took about a good 3 or 4 minutes for the client to (appear to) search through the entire downloade block chain (which is about 10.5 GB at present). I was surprised it needed to do this has I currently have an empty wallet which should mean a minimum number of used keys in the client's default keypool count of 100 keys. It then proceeded to update the less than one day of blockchain the client was behind on. Because of this long wait, it would seem that the QT client isn't quite a quick starting application in terms of user friendliness (I'm aware of other newer thin clients that don't fetch the entire blockchain database) and I would surmise the client start up wait time will only get worse as the BTC currency will become more active in the future and the blockchain inevitably gets much larger. Based on your description of a BTC wallet structure and the transaction format of BTC, it would appear that the WALLET.DAT file would also increase in sizet gradually as more user transactions occur and the number of identifiers increases in the wallet. Do I understand it correctly that a wallet.dat file will eventually accumulate all used keys since the start of transaction with that particular wallet.dat file?

I'm also wondering whether there is an advantage to increase the QT client KEY POOL size value beyond the default of 100, by using the -keypool parameter within the BITCOIN.CONF file. From what I've read up so far, the client will attempt to maintain a constant pool of (new unused) key pairs which totals to the number specified by the -keypool parameter (or just 100 if parameter is not specified). But I'm a bit confused on new key generation for the keypool after the current keypool's new key pairs are used up. When that occurs, does the QT client generate one new key at a time as needed when new keys are required / requested by the user, or will the client immediately generate a new batch of new keys in order to maintain the available number of new key pairs in the key pool at the amount specified by -keypool parameter? I understand that depending on the keypool size, occurance of key use frequency and also the frequency of WALLET.DAT back up by the user, the degree of risk of potential new keys lost when restoring from a recent back up will be affected. I guess that will depend on how recent the back up was made and how many new keys have been generated after the most recent WALLET.DAT back up took place. Based on this, it would seem that new key loss risks from WALLET.DAT restoration from a recent back up could be reduced by increasing the key pool size. Would this be correct?

It is best to think of the wallet storing a set of passwords (private keys) that are used to sign messages (transactions).  Each password (private key)  is mathematically linked to exactly one identifier (address).

So the blockchain is a set of messages that essentially say "Reassign the value that was previously assigned to identifier A as identified in this previous message: (transactionID here) Assign that value to the following identifiers in the following quantities: (addresses and values here)" The message is then signed with the password (private key) of identifier A.

The wallet.dat file is just a collection of those passwords (private keys), and the associated identifiers (addresses).  The wallet software searches through the local copy of the blockchain for every occurrence where value is assigned to any of the identifiers (addresses) from the wallet and isn't reassigned via a signed message later.  Each such occurrence of a message assigning value to an identifier (address) that isn't reassigned later is a UTXO (unspent transaction output).  The wallet software keeps a list of the UTXO that it finds, and totals up all the UTXO associated with identifiers (addresses) from the wallet.dat file.  This total is displayed by the wallet software as the total "bitcoins" that you have.

You can import private keys into a wallet.dat file and the wallet software will compute the associated address and add it to the list as well.  So multiple copies of a wallet.dat doesn't mean multiple copies of bitcoins, it's just multiple copies of the passwords that are used to sign messages to be added to the blockchain.  Loss of the information in the wallet.dat means you can never sign a message to reassign the value from UTXO currently associated with the addresses, so you can never create transactions to transfer that value to anyone else.


It could be either one, but not necessarily.

Bitcoin is a peer to peer network.  When you broadcast a transaction, you only broadcast it to the few peers that your client happens to be connected directly to.  Those peers each validate that the transaction meets all the requirements of the protocol, and then they relay it to the peers that they are connected directly to.  Those peers then do the same, and so on until the entire network (including blockchain and the recipient) has heard about the transaction. The ip address stored by blockchain.info is the ip address of the peer that they were connected directly to that first relayed the transaction to them.  This could be you, could be the recipient, or could be someone else entirely.

Yes, it's like using Quicken and giving it access to all your bank accounts, brokerage, etc and it displays your total net worth.

Okay, so each bitcoin address is like an independent "sub-wallet" that is all collected together by a single wallet.dat file then, even though the wallet client will display the wallet grand total as the sum of all the addresses. I noticed that on sites such as blockchain.info, transaction data includes an IP address. Would this be that of the sender or recipient of the funds? It would seem that it would be possible to see all transactions related to one user if you knew their IP address, if I understand this correctly. So I wonder about the implication of privacy / anonymity of the system.



That sounds strange. I've never had a Windows shortcut misbehave like that. I personally would not prefer to have the Bitcoin-QT client in the start up program group, I plan to only run it manually when needed and don't mind that I have to wait a bit for the blockchain to be updated, as I prefer not to keep the QT client connected to the internet at all times unnecessarily. The -datadir parameter is working very well for me at the moment and I may use it to manage multiple wallets in the future by creating additional Windows shortcuts that point to a different datadir path.

Everything is in the blockchain, so you need to have a fully downloaded blockchain to see how much each address holds.
Addresses are never checked to see "if they are in use".  If you find someone else's address -- congrats, you can take all their money.
Change addresses (all receiving addresses) are generated by your wallet client.  There's no other way to do it.
Don't think of the addresses as pointing to the same wallet.  All the addresses are independent.  The wallet is just a list of addresses & their associated private keys, nothing more.

Thanks for your detailed explanation and supplying the CHANGE Wiki page. If I understand what I just learnt correctly, Bitcoin addresses are just like e-mail aliases that all point to the same e-mail account (i.e. wallet), you can use multiple Bitcoin addresses for higher privacy protection, but incoming funds will all end up in the same given wallet. From what I read in the CHANGE wiki page, the grand total of a wallet would be calculated by the wallet client software which would obtain the various bitcoin address balances to arrive at a wallet grand total. Does the wallet client actually look into the downloaded blockchain database to obtain the latest current balance for each of the wallet's Bitcoin addresses or would the client look this up online? If the former, then that would imply the wallet grand total may not be determinable by the wallet client until the entire blockchain has been downloaded, would that be correct?

Regarding the newly generated change address when a user (sender) sends a payment and receives change back, under most clients a new bitcoin address will be created to hold the change. Is the new "change" bitcoin address generated from the sender's (payer's) side client software or is it generated by the bitcoin transaction process and then then new address is returned to the payer's client software? It seems to me that there is a process somewhere where a new bitcoin address is checked to see if it was previously already used but I'm unsure where / how this process occurs, whether it is the role of the client software, the payee's client software, or the transaction process in the blockchain that generates it.

I had surmised that the wallet bitcoin values were not stored in the WALLET.DAT file as I understand from reading elsewhere that with the proper keys a lost wallet file's value can be recovered in a new WALLET.DAT file so this implied to me that the data would have been storeed in the blockchain. During the recreation of a wallet, would it take some time while the client software has to search through the indices of the downloaded blockchain database, and would it mean that if I had a new QT client installation which did not yet have the blockchain database fully downloaded, that it would be impossible to recreate a wallet with its keys?

I had problems with the shortcut in my startup directory not keeping the --datadir argument.  When I'd restart my computer it would start downloading to my appdata folder instead of using my specified datadir.  I deleted the shortcut from my startup directory and just run it manually now and it's fine.

You have a good point about different data directories when logging into Windows as different users (something I hadn't thought about), although in my case, the data directory hasn't changed and I'm using the same Windows user account each time I'm running Bitcoin-QT. In fact, before I even installed QT, I was concerned about the issue of finding out how to control and specify the data directory path for the QT client and subsequently found out that I could use the -datadir command line parameter in the Windows shortcut to the QT client in order to explictly specify the target QT client data folder path as I do not want the large block chain file to hog the boot drive C:. I have the data folder on a different drive path.

I'm not sure why the QT client decided that the fully downloaded block chain last night was invalid. As soon as I noticed it was completely re-downloading despite ALL the downloaded blockchain files were still present, I quit QT, deleted all data folder files including my empty wallet file and restarted the QT client. Strangely, tonight the download is significantly faster and it is already close to finishing after just 4 to 5 hours compared to about 8 hours last night and also the CPU load consumption on my multi-core processor is much lower tonight.

Looking down the road, if when I will be updating the QT client to future versions, I'm assuming that with my data folder in a completely separate drive path, I should just need to completely uninstall the current QT client version and then install the new version I'm upgrading to, add the -datadir parameter to the Windows shortcut, and I should be able to pick right up where I left off with the old version, yes?

I'll have a look at the Electrum client, though I think as a study and learning experience, it might be useful for me to start with the original Bitcoin-QT client first as a base experience so I can relate to other new / different features in other clients and understand how it compares in pros and cons of the original BTC client.

You can do without change addresses in electrum, but I haven't seen that in bitcoin-qt.  Your client shouldn't download the blockchain each time.  Make sure you're logging on as the same user as it puts it in your home directory.  If you run it as admin and then as a regular user your home directories will be different.

A wallet is a collection of one or more bitcoin addresses.  Those aren't "wallet addresses", they are "bitcoin addresses".  There is no such thing as a "wallet address" (although some of the wallet programs out there may limit you to just one bitcoin address in the wallet).  Bitcoin-Qt will allow you to have as many bitcoin addresses in the wallet as you like.  Just click the "New Address" button on the "Receive Coins" screen to get another address added to the wallet.

Now, if you want to isolate bitcoin addresses into separate wallets, you'll have to be careful.  Bitcoin-Qt doesn't have such functionality built in (you might want to look into Armory, which can do that).  It can be done with Bitcoin-Qt, by carefully swapping out the wallet.dat files after the Bitcoin-Qt program is fully shut down.  Don't do it while the program is still shutting down, you'll risk damaging the wallet file.  Each wallet.dat can then contain access to a set of bitcoin addresses.

Note that the bitcoins are not stored in your wallet.dat file, nor are they stored in your Bitcoin-Qt program at all.  The record of how many bitcoins each address has access to is permanently stored in the blockchain that EVERY full node has a complete copy of.  When you send bitcoins to an address from a wallet that you don't have loaded, the transaction is relayed out to the network and eventually added to a block in the blockchain.  Then EVERY full node knows about it.  Later when you start up Bitcoin-Qt with the wallet.dat that contains the address, the Bitcoin-Qt sees the record in the blockchain and updates the balance it displays to you.


When sending bitcoins.


Read this:
https://en.bitcoin.it/wiki/Change


You can't.

I just started my learning curve from scratch with BTC and installed the Bitcoin-QT 0.83 Windows client on my system. Eventually I would like to also use Armory on top of this client though at this time I won't be using Armory as I read that recent updates of Armory had caused difficulties with Windows systems below a specific level of available RAM so I plan to wait for later updates.

In the meantime, I have just started with the QT client and it installed fine and completed download of the entire blockchain (about 10 GB data) in about 6 to 7 hours and everything was fine. However, today when I restarted the QT client, for some reason it started to re-download the entire blockchain as I noticed the progress bar at the bottom of the client window saying that the blockchain on my system was several hundred days behind current. I ended up quitting the client and went into the data folder to delete all the files (I had an empty wallet) and then restarted the QT client which is now re-downloading the blockchain (though seemingly at a somewhat faster pace). I'm wondering if it is normal for this sort of thing to happen with this client and if anyone else had similar experiences, what the cause of this might be. From what I've read, if the QT client is quit and goes off line, next time I restart the client it should just download new portions of the blockchain that it hasn't yet downloaded previously, so I had figured a one day re-sync should be fairly quick.

I have read up as much as I can within recent days on various articles about wallet and wallet management but am confused about a number of points.

1. I understand that the WALLET.DAT file contains keys and other security control information to my wallet. But is WALLET.DAT limited to holding one wallet address only? I searched other threads on the forum about managing multiple wallets (which eventually is my goal before having any serious BTC transactions), and I understand that one can transfer between multiple wallets. But how are multiple wallets managed - does it require multiple wallet.dat files and what exactly happens during funds transfers between wallets? Does this mean that funds transfers have to go out as a transaction into the master block chain in the BTC network and then somehow return to a target wallet of mine that I specify?

2. I also seem to have somewhere read that in the QT client, whenever a transaction occurs (I cannot remember if it is sending or receiving BTCs), that the client will automatically generate a new address. Can someone clarify this for me? I read that the reason for generating new addresses is to increase security, but how / where in the QT client could the user configure the client so that the user can manually select whether a new address would be automatically generated or not after a transaction (Perhaps configured via the BITCOIN.CONF file start up parameters or command line parameters?), and what the pros and cons of this would be in terms of security and anonymity.

I have read up on a lot of various cases of losing funds in from a wallet so I'm trying to study / understand how the wallet works to get a good feel for BTC wallet management before diving into any significant transactions. Thanks for any input.