Topic: Lost large number of bitcoins (Read 43742 times)

Yeah, I love this and want to amplify the thought. Primary devs take note.

I understood roughly what the official client does with the wallet.dat when I began experimenting with it, but learning "specifically" what it does took concerted research effort trolling the forums on my part. It is not clearly written on the box, so to speak. So, because I'm curious, and I understand that I'm working with "experimental" software and need to be careful, I am aware of the need to have an ongoing backup strategy tailored to the needs of the wallet.dat and my spending habits when using the client.

Some of we tinkerers, e.g. myself, come from a background of experience using OpenPGP and X.509 to accomplish secure/authenticated communications. In this world, you, the user, have and must have total control of the creation and maintenance of your private key. Learning these systems taught me what I know so far about public-key crypto. I learned it would be a pain if you lost control over your personal private key and had to revoke it, then go out and accumulate signatures on a new key. Because of this we tend to attach ourselves mostly to just one key, and keep it carefully secure. It takes on a meaning then where it comes represent our official self digitally.

Coming into Bitcoin with that attitude is kinda dangerous without some more education. As it stands in the client currently, key management is automated and effectively comes with the label, "no user-serviceable parts inside."

Wallet.dat is a black-box to most of us. We can ask the client to generate new addresses (which causes new keypair generation within the wallet.dat), but this is all the control we have.

To avoid value destroying mistakes of the one this thread is all about, I think key management should be reconsidered. I'm going to vote for a more OpenPGP-style default design. This is essentially what you would start to get within wallet.dat if Joan's ideas (quoted above) were implemented as defaults.

I understand the benefits of using new keys as fluidly as they are in the client right now, it massively bolsters anonymity, a property I think valued highly at least by creator Satoshi and the development team working closely with him.

But bitcoin is starting to become viral, with n00bs arriving daily. User experience and easy access to education to help them bootstrap and go competently forward is going to become more important for the system's long-term health, when it expands beyond the tech-heads (yes, and cue the idea of most folk using an online wallet provider a-la Mybitcoin. Fair enough. Some will desire personal control, yet still with a safe and easy use experience. I don't think that's asking a lot.).

Perhaps I care less about the obfuscation of my transactions within the sea of the block-chain, and would rather everything come to/from a single key I make. I can still get more creative and make special-purpose keys to denote bitcoin received from different sources, or to obfuscate my transactions (as is the default now). The point it this aspect of the way bitcoin works should be controllable and transparent, and as easy PGP.

Finally...I'm beginning to come around to the notion that perhaps bitcoin will not become a major value store for day-to-day transacting, as we're often envisioning its future. But through developing systems like OpenTransactions, becomes the missing mechanism allowing more customary media of exchange to be tied together in a way that confers the benefits of bitcoin to the convenience and instantaneous-finality of more familiar forms of money.

For more on that, I highly recommend these mind-blowing shows:

http://agoristradio.com/?p=234
http://agoristradio.com/?p=246

yeah it depends on ur financial situation.  i have a large savings wallet which goes into the safety deposit box and other areas for 10 yrs.  my spending wallet is different but yes i do have to back up before tx 100.

Surely at some point I will need to transfer coins from my savings to the spending address ? I will have to count how many times I'm doing it to check if I need to do a new backup. With persisting address, I need backup only once.

Anyway the remark was more about foolproofing the default behavior of the client, lower the surprise effect and improve data loss prevention.

I understand there is a tradeoff between built-in untraceability and having persisting addresses for the sake of simplicity and managing one's accounts.
Right now I don't care too much about traceability of my transactions, since it is still hard to match an address with a physical person. I know others will care and will use a different address for each tx. Should it be on them to opt-in ? Could I at least opt-out ?

With the keypool of 100 addresses created at initialization, I will have plenty of room to segment my wallet for different usages. I could have a spending address dedicated to food, another for books, etc. all within the same wallet.

To sum it up:
- keypools = great. If I backup on day one, I have a hundred of addresses backed up.
- using a new address to send the spare change of a transaction = should be opt-in or disabl-able. Otherwise my backup dies at tx 101 without me noticing it.

joan,

you can overcome this with a saving and a spending wallet.

I would like to second Charles P. Bianchi suggestion.

By default the change generated from an outgoing transaction should just back loop to the emitting address.
This is the most simple behavior and avoid surprises.

I would argue this is actually what new users (like me) wrongly assume to happen if they don't pay too much attention to the address number.
Would allow to backup my wallet right after installation and forget about it.

If my day-to-day client always use the same address, I can restore anytime from the initial backup. The only thing really valuable should be the private key. The rest is stored on the network.
Otherwise even the most contrieved scheme of splitting and backing up wallet files into physical drives inside hidden buckets as I read in another thread would only work reliably up to 100 outgoing tx.

Then there should be a more powerful mode where you can opt-in to have the change sent back to a newly generated address each time.

Greater control over your addresses list would be desirable if they are more persistent. A nice thing might be the possibility to create aliases locally. Since each address is like a little account, it would ease management.

in your case, nothing should have gone wrong whether u sent to one of ur own addresses or to someone else's.  your client would have taken your chosen amt from one of your stored addresses with coins just above the chosen amt and sent it to the chosen address.  the change would be sent to another of your pre stored addresses.  then the process would have been repeated with the only address reused being the one you sent to.

That isn't what happened to him at all. I question your copy/paste ability based on reading comprehension.

Dude, im so glad someone else experienced this too. I sent 5BTC to an address that i copy and pasted, then i copy and pasted again just to make sure that i did it right, long story short, idk were they went

I endorse this idea.

Add to the client an automated backup system.  You click a button to back up, it encrypts your wallet file with the password of your choice and uses a file system browser to ask you where you want to store it.  You select your thumbdrive, have a nice day.  It warns you to back up again when you are down to 25 and 10 and 0 remaining keys.

Also add a restore button that allows a person to easily and quickly import a wallet file provided they know the password.

Some highly technical thing that most regular people will probably answer wrong (meaning, in a way that won't get them the behavior they will later wish they got) and which must be answered every single send?  MEH.

People can't be expected to think through the consequence of every transaction. Sometimes you won't realize until long after the fact that you really wish some payment or another had been more anonymous. The system should try to avoid creating situations the user will regret if it can.

How about just keeping track of when backups were taken via the formal backup function and whining at the user once they get low on keys which have been backed up?

Once the encrypted wallet support is in— how about integrated online backup support?… if there was a feature integrated in the client I'm sure there would be no shortage of people selling backup services that accept a few bitcents in exchange for keeping a copy of your encrypted wallet in some web accessible place.

These things would solve more problems and be much more user friendly than some complicated decision with every TXN.

For clarity, at least when in dumbed-down mode, it should maybe not take dumbed-down to mean hide all the details like a knowledgeable user might prefer due to knowing all the details very clearly but rather, to be dumbed down like a "wizard" or even a "guru" routine.

That is, walk you through it step by step asking for confirmation of each step ("wizard") and possibly even (a la "guru") actually attempting to explain what it is all really about despite those who prefer the "wizard" approach wanting very very much not to ever ever have to understand anything about anything and most certainly not anything even remotely related to computers or computing.

So maybe something like:

You have asked to pay #.## bitcoins to [alias or address of intended recipient].

Due to the precise denominations of the coins currently in your wallet, this
will be done by sending them # coins, and in addition will require "breaking" #
coins to attain the precise amount to send, resulting in #.## change which
can be sent to any of the following addresses:

[ ] A new never before used one of your own that you have not yet backed up due to it has not been invented yet.

[ ] An old never before used one of your own created on ##/##/####, which might exist in a backup if you made a backup since then.

[ ] A (or the) default change-address you configured for this wallet in the wallet configuration section (see help etc etc)

[ ] The default charity you have configured for this wallet in the wallet configuration section (see help etc)

[ ] Some other address, in which case please input said address now: _________________________________________

-MarkM-

(I left out [ ] Send the change to the same address the payment is being sent from, because it seems unclear why, or even whether, it is apparently necessary at least in some circumstances to send it elsewhere. Does the protocol prohibit the sending address(es) appearing in the list of recipient addresses maybe for example or what?)

assuming for a moment that you will use the same Bitcoin client to access both your day to day and your vaulted wallets, what names can you safely give to them since they are both by default wallet.dat?  when u decide to dig up your vaulted wallet and put it back in the Bitcoin directory, do u have to make sure the day to day wallet is not in there too?  is there some stupid way u could corrupt one or both of the wallets by doing this simple access?

"If transactions coming in are to the same address, then you don't have to worry, because it will only create one new address."

i don't understand this statement.  with an incoming receive tx, where is the new address created other than the one you've given to the sender?

Well, your day-to-day account should be backed up regularly, but hopefully you aren't keeping a huge balance in there anyway, so if it is lost, it would only be a few thousand dollars worth of bitcoins at most.  You could keep a copy of your vault backup in a safe deposit box too, so if you suddenly have need for the equivalent of tens of thousands of dollars worth of bitcoins, then you could grab the backup from your safe deposit box and start it up.  I would imagine you could constrain yourself to less than 100 transactions from this account per lifetime to ensure the resiliency of your other backup.

I agree the current solution is not ideal.  Ideally, backups WOULD be 100% persistent.  I am just saying that there are easy ways to work around the issue if you know what you are doing.  For the general populace, we do need a better solution, as you KNOW there will be people out there who don't understand the process.

I would advocate that people choose whether to generate a new address for the "change" in a transaction, or whether to just keep them at the old address.

I still think a backup service, whose sole purpose is to back up wallet files and release them only when required by the individual or a lawyer on behalf of an individual's next of kin, would be great.

That is correct. And the backed up 'vault' idea is also correct. But we are not all hoarding money. I assume you have a bank account right? You deposit money and you EXTRACT money to pay bills. When you die, the account can be liquidated.

Most people are not so fortunate that they can dump money in one direction entirely (never dipping into those funds ever again) for the benefit of their estate. At some point, most people will need to use the money (that is not STORED, but) BACKED UP on the bottom of the ocean.

BitCoin presents no good analogy to the current system. I can backup money (vault) while still having access to money (copy). I am only advocating that a backup actually be a reliable backup. Anonymity through obfuscation is not the only use case. Control is another.

* sighs upon seeing stupid people sorry for a guy that lost BTC 1 year ago and probably will not read this *

dont worry, its only ~78 000 dollars.








...

It's only transactions going out that use new addresses though, right?  If transactions coming in are to the same address, then you don't have to worry, because it will only create one new address.

Therefore, you could create two wallets.  One is your day-to-day wallet that is always on your computer.  Another is created for the sole purpose of being a vault.  This second wallet is the one you give to your lawyer, and the one you put in a sealed container at the bottom of the ocean.  Any time you have an excess of funds, you can simply transfer them to this secondary wallet.

And your computer will still have your current wallet installed/running anyway, which hopefully, you've given password access to said computer to your lawyer to hand off to your next of kin in the event of your departure from this world.  So your next of kin should have access to both of your wallets, even while one is heavily protected and in storage for 100 years.

That is correct. It still sets a precedent of false expectations. First of all, aside from geeks who read the source code, the majority of users will assume that 'change' is returned to the same address. In fact, most users will not understand the distinction between addresses and wallets, particularly because the client does not present the pool of addresses or the balances per address. As soon as a user learns "phew! I can just restore from backup", they will similarly assume they can ALWAYS restore from the same backup.

In fact, that is precisely what should be DESIRED. I should be able to store a physical offline backup of my addresses in a sealed container at the bottom of the ocean. I should be able to give a sealed copy of the keys to my lawyer to be opened by my next of kin upon my death, which I hope will be a hundred years from now.

Let's not focus entirely on anonymity and encryption, while making it very difficult to hold on to our money. A cycle of 100 static RECYCLED keys is probably good enough for most casual users. Let the paranoid pre-generate a billion keys, but then keep it STATIC until the end of time. Or just use one address per wallet and let the paranoid launder their money using a predictable process.

Looks like this was already added to the list of known lost bitcoins. I did make a minor correction to the number, though.

Wow , tough break:'(.  I wish I could help you .

You are correct.

As I was perusing the source code, I got the impression that each time a key is used, the wallet is topped off to have 100 spares.  I could have understood it wrong, but if I understood it correctly, I don't think it's the 101st transaction that gets hosed... it's the 101st transaction since the backup you have.  (e.g. backup at 55 transactions, you'll get hosed if you need key #156).  Someone please correct me if I'm wrong.

That'd be the equivalent of not having a change at all, I believe.
And, well, honestly, I don't know why it was implemented like this. Unless there's something in the protocol that forces an address to be completely spent when it's used as input, I don't see much utility in this change feature.

> There was talk a while ago about building up a sizeable list of addresses
> in the wallet that would be hidden but used for change for future
> transactions.  The benefit there is that if a user backs up his wallet and
> something like this happens after future transactions, his old backup
> will still "contain" all of the bitcoins since it actually has the addresses
> that change coins were sent to.
>
> What ever happened to that idea?  I think it's a good one.

Since this post a year ago, this feature has been implemented as keypools=100. Personally, I think it is a bad idea and does not address the problem, only pushes it off to the 101st address. At that point, users will come to expect certain backup behavior and then one day (presumably when they have more 'real' rather than 'play' money) it doesn't work as expected.

Unless the pool is recycled (change is returned to a random or cycle of addresses) then this is far more dangerous.

I propose, instead, what is expected. The change should be returned to the same address that the BitCoins were sent from.

I understand this decreases deniability/anonymity. But if someone really is paranoid, they should really be laundering money through multiple addresses in random amounts at random intervals. Sending change to a new address is just an obvious 'paper trail' considering all transactions are public, it doesn't take genius investigator to follow the money.

You mean when you lose wallet.dat you should still have your coins? He would have been fine if he didn't erase his HD.

I would like to promote this idea as well (or something similar).  What happened to Stone Man simply should be nearly impossible from the start, regardless of backups.

There was talk a while ago about building up a sizeable list of addresses in the wallet that would be hidden but used for change for future transactions.  The benefit there is that if a user backs up his wallet and something like this happens after future transactions, his old backup will still "contain" all of the bitcoins since it actually has the addresses that change coins were sent to.

What ever happened to that idea?  I think it's a good one.

More than just Accounts, I'd really like total visibility into what bills are in my wallet, the specific transactions in/out, some total register view that has all the details available.  I feel like the UI is a bit dumbed down, and most of the early adopters are curious about the machinery, not just their total balance.

Just my BTC 0.02.

I will 2nd (3rd, 4th... whatever) this suggestion.  This would help the accounting tremendously.

and, stone man, wow, my condolences.

Ta,

I just want to add my voice to those recommending strongly that the client have an "Accounts" Tab showing an Address and amount stored in each Account.

This would be a natural place to add backup functionality and provisos and warnings.

You can get away with backing up without stopping the client if you don't do anything or receive a payment within a few seconds before the backup.  (like 5 seconds)

No, it doesn't usually empty your wallet with each transaction.  It uses the smallest set of coins it can find to add up to near the amount.  In this case, unfortunately, his wallet had a single 9000 BTC bill in it, and it had to break it to get 1 BTC and 8999 BTC change.

Your 'wallet' is more like your own personal bank
It contains many different accounts. (in Bitcoin these are called Addresses)

Each Address has a balance associated with it, how much money is in it.
Your Wallet balance is the sum of all the balances of the Addresses in your wallet.

When you perform a transaction, it empties enough Addresses to make up the required output amount, but since it has to completely empty each Address there is often money left over in this case the 'change' is returned to you at a new Address.

In this case Stone Man did a transaction that sent all the money from 700+ addresses into a single address and therefore had a wallet containing only one Address that had any money in it.

That Address/Account was emptied as part of the 1BTC transaction and the 8999 change was returned to a new Address to which he lost the private key.

If he had not consolidated all his coins into one Address, he would have lost far fewer coins.

I think the moral of the story might be: "Don't put all your eggs in one basket"


No you can't find out which addresses have which balances using the current GUI.

I think it is "very important" information. 
I also think it is important to be able to chose which addresses are the source of transactions.

This is an example of why the wallet metaphor is bad.

Ok, re-reading carefully and checking the referenced thread, I finally understand this in detail. It's not that bitcoin "empties your wallet" at each transaction - it fully empties an address used to send coins. In this particular case, the user had previously sent themselves the 9000 coins in a lump, resulting in all those coins being held at a single address.

So far as I can tell, there is no way from the GUI client to actually get at the information of what coins in your balance are held at what addresses? Given that the addresses are what is "really important" that information might be of value.

My understanding is that it finds an address or addresses that have at least the number of coins you want to send and sends the change to a new address.

So if he had addresses with 1000, 2000, 2500, and 3500. It would have selected one of them (the lowest one?) and sent 1 away and sent all but 1 back to a new address of his. In this case he would not lose them all, just the remainder of what was in that one address.

It appears he had only 1 address with coins in it. This is probably because he got them all from the market in one go. If that is not the case, then I don't know why he would lose them all.

Wait, I'm confused again. I thought the essence of the surprise was that Bitcoin is programmed to "empty your wallet" for EACH transaction. According to the description I read, when you send coins from address A in your wallet to address B externally, the transaction is actually done by sending ALL the coins out from address A, and the ones that aren't going to address B get sent to address C which is your own address - in other words, even if I'm only paying you a single bitcoin out of my 9000, I mail 1 bitcoin to you and 8999 to myself at a new address.

In other words (unless I'm confused), every transaction you make will result in your old, backed-up wallet addresses become emptied out.

You were right before. The reason all of his coins were lost is that he first transfered all ฿9000 to himself, merging them into a single TxIn. If he had skipped that step and gone straight to sending himself ฿1, he would have only lost the smallest payment that he had previously received that was over ฿1.

I think the client needs to communicate TxIns and TxOuts better to the user. I don't know how to do that without being confusing, but there are real privacy, safety, and security implications in which coins the client chooses to transfer.

A simple solution to this would be to store your wallet.dat on an encrypted partition that goes to an S3-based storage system, like Dropbox. They'll keep versions for you on update.

This would make a nice opt-in service that could be sold in the native client as a way to help fund research, by the way. The data storage needs are incredibly small.

News to me is that *all* your coins are at risk.  I thought it was just clumps of coins (previously received transactions) involved in the transaction, not my aggregate balance.  Yikes.

As a countermeasure, I've manually split my wallet up into five new wallets of distributed value.  Now only one is at risk at any given time.

Good to know this, I thougt I was save with my backups, but it could have happened to me to.
Stoneman didn't really do anything stupid. 

This is indeed a good solution. Solves more than one problem actually.

I think Satoshi proposed a protection against this in the following post:

+1

This shocked me actually. The poor guy didn't do anything stupid.

Automatic backups are really important after knowing that. And while that cannot be implemented, I think a sort of information message explained this in short words should be displayed every time someone sends coins. A popup saying something like, if I have T coins and I send S coins:
"S coins where sent to address X.
(T-S) coins where sent to address Y, which belongs to you, in order to improve privacy. Please, realize that old backups you might have of your wallet will now be useless. You should make a new one."

This simple message would have avoided this awful lost. (of course, you may also add a "Don't display this again" check box)

Regards

If you backup a spreadsheet, do you not expect to lose changes you made after that backup when you restore?

There are many types of transactions that will survive a wallet restore. This is one of the circumstances in which you will lose coins.
If he'd transferred all 9000 to the address he would still have them.

I'm not trying to blame the user. I agree this is a problem that needs to be fixed.

This problem has been previously identified and there are plans to implement a fix, which has been talked about in the Privacy versus Safety: handling change thread. But it has not been implemented yet.

Perhaps a bounty is in order?

According to the theory, generating a address collision is so remotely unlikely as to never happen.
(You're more likely to get struck by lightning, win all the worlds lotteries, and then get abducted by aliens on the same day.)

However if you manage to find a weakness in a) the encryption method or  b) Bitcoins implementation of it. it may make it easy enough that you don't have to get abducted by aliens as well.

So they practically are lost forever.

So, they aren't lost forever? I guess it is more likely to generate the key of the address of someone else then from this specific address.

I added to the FAQ the warning to back up after each transaction. Is it necessary btw to stop the client before making a backup? That's a bit inconvenient. Automatic backups would be useful indeed.

Until this thread I was totally unaware of this behavior and risk. The usual behavior of a "backup" of a file is to preserve the contents of that file, and the intuitive understand of the wallet.dat file is that it "contains" the coins. I understand this is a case of the system behaving as designed, but the information that you need to backup your wallet after every transaction and each transaction makes your old wallet backups useless needs to be communicated much more clearly to users, I think. It also seems to me that it would be useful to add features for 'automatic backup' of a wallet after each transaction and easy support for multiple wallets - a "savings account" and a "checking account" so to speak.

Sorry about your loss, Stone Man.

Ugh, that's awful.

Actually, the sad part was I wasn't even testing the wallet backup when this happened. I was trying to watch when the network should have confirmed a payment to a website that takes bitcoins by paying myself 1 coin at about the same time. I never dreamed it could compromise my whole balance, especially since I was sending to myself.

I used bitcointools: http://github.com/gavinandresen/bitcointools
To look for your transactions in the block chain.

(Here are the edited highlights)

So the transactions are definitly in there.
Your only hope is to have the private key for the 167ZWTT8n6s4ya8cGjqNNQjDwDGY31vmHg address.

If you can get bitcointools working you can dump your wallet.dat using:
which will show you all the keys you know about in your wallet.
But you'll probably find that 167ZWTT8n6s4ya8cGjqNNQjDwDGY31vmHg is not in there

On the upside, (although it is probably of no consolation to you) there is now a decent bounty for anyone who can find an address collision...

_{Edit: I'd pasted the wrong block number and date for the first block. Fixed.}

Address1: 157PiPgqphedUvrco3mKU3Xoof7yzhj9pW
Address2: 157PiPgqphedUvrco3mKU3Xoof7yzhj9pW
Address3: 167ZWTT8n6s4ya8cGjqNNQjDwDGY31vmHg

Yes, I do.

I saw on another thread that gavinanderson has made some tools to do more in depth research but did not know how to proceed.

If you could point me in the right direction I would greatly appreciate it.

Oh, man. Why did you send the 1? Were you just testing to see how the backup worked? Why with so many coins?

That sucks.

It sounds like they are lost for good.



Lesson to other people: Remember you need to back your wallet up after every transaction!
(The wallet metaphor is actually a bad one, because it makes problems like this more common than they should be.)

Edit: (more explaination)

When you do a transaction you always spend ALL of the coins in that address, part of the money goes to whoever you're paying, and there rest is returned to you an a new 'change' address. (This helps your anonymity as someone watching cannot tell which amount you spent and which amount you kept.)

Before transaction:
Address1: 9000
Address2: 0

After transaction:
Address1: 0
Address2: 1
Address3: 8999

It sounds like you have wallet.dat that contains the private keys for Address1 and Address2, but not Address3.
(Although the numbers in your story don't add up.)
What were the Bitcoin addresses? (If we know these we can do some more in-depth block chain research)

Do you still have the wallet.dat that once "contained" the 9000 BTC? I might be able to help you.

Here are the details.

1) Bought 9,000 BTC on one of the exchanges over time.
2) Transferred them to my client running on a linux live CD distro of Debian.
3) Backed up the wallet file to a flash drive.
4) Sent 1 BTC to myself
5) Closed client before any confirmations
6) Shut down system (wiped system disk loaded into memory and therefore the ./bitcoin folder
7) Loaded system back up
Copied old wallet.dat file into ./bitcoin folder
9) After some confirmations appeared the balance was 1 BTC and there was a transaction saying I spent 8,900 BTC to an address I did not recognize
10) I read on the forum threads that people have had problems like this but it seemed only when they were trying to double-spend by sending coins to another address and reloading an old wallet file


Is there anything I can do?

I do have the address that the 8,900 were supposedly sent to but the old wallet file is gone for good.

Thanks,
Stone Man