This email, or email excerpt, was quoted by Gavin Andresen in an interview in 2014.
I wish you wouldn’t keep talking about me as a mysterious shadowy figure, the press just turns that into a pirate currency angle. Maybe instead make it about the open source project and give more credit to your dev contributors; it helps motivate them.
Mike Hearn <[email protected]> Sun, Apr 12, 2009 at 12:46 PM
To: [email protected]Hi Satoshi,
I read your paper on BitCoin with great interest. I found it a bit
confusing though - I believe it may be easier to follow if you provide
some examples.
Specifically, it's not quite clear to me what blocks contain. If I understand correctly, there is only one (or maybe a few) global chain(s) into which all transactions are hashed. If there is only one chain recording "the story of the economy" so to speak, how does this scale? In an imaginary planet-wide deployment there would be millions of even billions of transactions per hour being hashed into the chain. I realize that each PoW can wrap many transactions in one block, nonetheless, that's a large amount of data to hash. If there are many chains, how are transactions assigned to each chain such that it is still difficult to overpower the network? Eg, if there are 10 global chains, the amount of cpu power you need to beat the system is only 10% of what it was previously.
I also wonder if the assumption of 1 core = 1 vote is sound. If the majority of nodes are on standard computers, it seems likely that an attacker could use FPGA or custom ASICs to get significantly better performance. What are your thoughts on using custom hardware to beat the chain?
I found the section on incentives hard to follow. In particular, I'm not clear on what triggers the transition from minting new coins as a reason to run a node, to charging transaction fees (isn't the point of BitCoin largely to zero transaction costs anyway?). Presumably there's some human in charge of the system - eg, you decided somehow that 24 million coins was a good number to have, and would distribute some kind of rules file saying "coins minted after this timestamp must have an N+1 zero bits prefix", which honest nodes enforce.
How did you decide on the inflation schedule for v1? Where did 24 million coins come from? What denominations are these coins? You
mention a way to combine and split value but I'm not clear on how this works. For instance are bitcoins always denominated by an integer or can you have fractional bitcoins?
So many questions
But it's rare that I encounter truly revolutionary ideas. The last time I was this excited about a new monetary scheme was when I discovered Ripple. If you have any thoughts on Ripple, I'd also love to hear them.
thanks -mike
Satoshi Nakamoto <[email protected]> Sun, Apr 12, 2009 at 10:44 PM
To: Mike Hearn <[email protected]>Hi Mike,
I'm glad to answer any questions you have. If I get time, I ought to write a FAQ to supplement the paper.
There is only one global chain.
The existing Visa credit card network processes about 15 million Internet purchases per day worldwide. Bitcoin can already scale much larger than that with existing hardware for a fraction of the cost. It never really hits a scale ceiling. If you're interested, I can go over the ways it would cope with extreme size.
By Moore's Law, we can expect hardware speed to be 10 times faster in 5 years and 100 times faster in 10. Even if Bitcoin grows at crazy adoption rates, I think computer speeds will stay ahead of the number of transactions.
I don't anticipate that fees will be needed anytime soon, but if it becomes too burdensome to run a node, it is possible to run a node that only processes transactions that include a transaction fee. The owner of the node would decide the minimum fee they'll accept. Right now, such a node would get nothing, because nobody includes a fee, but if enough nodes did that, then users would get faster acceptance if they include a fee, or slower if they don't. The fee the market would settle on should be minimal. If a node requires a higher fee, that node would be passing up all transactions with lower fees. It could do more volume and probably make more money by processing as many paying transactions as it can. The transition is not controlled by some human in charge of the system though, just individuals reacting on their own to market forces.
Eventually, most nodes may be run by specialists with multiple GPU cards. For now, it's nice that anyone with a PC can play without worrying about what video card they have, and hopefully it'll stay that way for a while. More computers are shipping with fairly decent GPUs these days, so maybe later we'll transition to that.
A key aspect of Bitcoin is that the security of the network grows as the size of the network and the amount of value that needs to be protected grows. The down side is that it's vulnerable at the beginning when it's small, although the value that could be stolen should always be smaller than the amount of effort required to steal it. If someone has other motives to prove a point, they'll just be proving a point I already concede.
My choice for the number of coins and distribution schedule was an educated guess. It was a difficult choice, because once the network is going it's locked in and we're stuck with it. I wanted to pick something that would make prices similar to existing currencies, but without knowing the future, that's very hard. I ended up picking something in the middle. If Bitcoin remains a small niche, it'll be worth less per unit than existing currencies. If you imagine it being used for some fraction of world commerce, then there's only going to be 21 million coins for the whole world, so it would be worth much more per unit. Values are 64-bit integers with 8 decimal places, so 1 coin is represented internally as 100000000. There's plenty of granularity if typical prices become small. For example, if 0.001 is worth 1 Euro, then it might be easier to change where the decimal point is displayed, so if you had 1 Bitcoin it's now displayed as 1000, and 0.001 is displayed as 1.
Ripple is interesting in that it's the only other system that does something with trust besides concentrate it into a central server.
Satoshi
[Quoted text hidden]
Mike Hearn <[email protected]> Mon, Apr 13, 2009 at 1:39 PM
To: Satoshi Nakamoto <[email protected]>Thanks Satoshi,
I tried the app yesterday. It seems to work pretty well running on Wine (I tried it on MacOS but it should run on Linux too, and will try that next week when I am back at work).
In the lower right hand corner it has a block count which increases rapidly and then stops. Is this the length of the global chain? It seems to advance far too fast for that. Or is this the number of genesis blocks that have been tried but did not result in a partial collision? I'm not sure if the way it stops and starts is expected, or some glitch caused by it running under emulation. My best guess - it is the length of the global chain, and the rapid advance at the start is as the software downloads and verifies the preceding blocks in the chain as being valid.
With regards to the buyer/seller experience, I understand that the global chain advances at about 6-7 blocks per hour under the current settings. If we assume that 0.1% is a good risk rate, then z=5 thus any transaction must wait a bit less than an hour before being solidified in the chain. As micropayments for things like web content or virtual goods are by definition something that requires low overhead, waiting an hour seems like quite a significant hurdle.
I understand that nodes attempt to find a POW to advance the global chain in an uncoordinated fashion. This sentence however:
"If a majority of CPU power is controlled by honest nodes, the honest chain will grow the fastest and outpace any competing chains."
is confusing for me, because it appears the only way the honest chain can grow faster than a chain worked on by 1 attacking cpu is if the keyspace to scan looking for a partial collision is sharded evenly amongst the participating honest nodes. That way the speed at which collisions are found would be proportional to the number of nodes. Yet I don't see any discussion of such work sharding, which obviously adds complexity. Likewise:
"To compensate for increasing hardware speed and varying interest in running nodes over time, the proof-of-work difficulty is determined by a moving average targeting an average number of blocks per hour. If they're generated too fast, the difficulty increases."
How is the required difficulty of each block communicated through the network and agreed upon?
Thanks once again. I have yet more questions but this is enough for one email
I will be happy to summarize these discussions into an FAQ-like document at some point. Apologies if the questions seem trivial.
-mike
[Quoted text hidden]
Mike Hearn <[email protected]> Mon, Apr 13, 2009 at 10:51 PM
To: Satoshi Nakamoto <[email protected]>Something else that isn't clear to me - does the global chain only get extended when there is actual work to do? Currently it seems to grow
all the time, although there are only a few people in the network. So presumably it gets extended with null blocks. Is this actually required? The timestamping doesn't have to be actually in parallel with real time does it ... it's merely establishing an ordering of events.
[Quoted text hidden]
Satoshi Nakamoto <[email protected]> Mon, Apr 13, 2009 at 11:00 PM
To: Mike Hearn <[email protected]>Mike Hearn wrote:
My best guess - it is the length of the global chain, and the rapid advance at the start is as the software downloads and verifies the preceding blocks in the chain as being valid.
Right. I'm trying to think of more clear wording for that, maybe "%d network blocks" or "%d block chain".
If we assume that 0.1% is a good risk rate, then z=5 thus any transaction must wait a bit less than an hour before being solidified in the chain. As micropayments for things like web content or virtual goods are by definition something that requires low overhead, waiting an hour seems like quite a significant hurdle.
For the actual risk, multiply the 0.1% by the probability that the buyer is an attacker with a huge network of computers.
For micropayments, you can safely accept the payment immediately. The size of the payment is too small for the effort to steal it. Micropayments are almost always for intellectual property, where there's no physical loss to the merchant. Anyone trying to steal a micropayment would probably not be a paying customer anyway, and if they want to steal intellectual property they can use the file sharing networks.
Currently, businesses accept a certain chargeoff rate. I believe the risk with 1 or even 0 confirming blocks will be much less than the rate of chargebacks on verified credit card transactions.
The usual scam against a merchant that doesn't wait for confirming blocks would be to send a payment to a merchant, then quickly try to propagate a double-spend to the network before the merchant's copy. What the merchant can do is broadcast his transaction and then monitor the network for any double-spend copies. The thief would not be able to broadcast during the monitoring period or else the merchant's node would receive a copy. The merchant would only have to monitor for a minute or two until most of the network nodes have his version and it's too late for the thief's version to catch up and reach many nodes. With just a minute or two delay, the chance of getting away without paying could be made much too low to scam. A thief usually needs a high probability of getting an item for free to make it worthwhile. Using a lot of CPU power to do the brute force attack discussed in the paper in addition to the above scam would not increase the thief's chances very much.
Anything that grants access to something, like something that takes a while to download, access to a website, web hosting, a subscription or service, can be cancelled a few minutes later if the transaction is rejected.
is confusing for me, because it appears the only way the honest chain can grow faster than a chain worked on by 1 attacking cpu is if the keyspace to scan looking for a partial collision is sharded evenly amongst the participating honest nodes. That way the speed at which collisions are found would be proportional to the number of nodes. Yet I don't see any discussion of such work sharding, which obviously adds complexity.
The keyspace is huge, 2^256. The thing being hashed includes the node's public key and a random nonce, so the chance of any two nodes duplicating work on the same space is negligible.
How is the required difficulty of each block communicated through the network and agreed upon?
It's not communicated. The formula is hardcoded in the program and every node does the same calculation to know what difficulty is required for the next block. If someone diverged from the formula, their block would not be accepted by the majority.
Thanks once again. I have yet more questions but this is enough for one email
I will be happy to summarize these discussions into an
FAQ-like document at some point. Apologies if the questions seem trivial.
No problem, thanks for testing it on Mac Wine.
Satoshi
[Quoted text hidden]
Satoshi Nakamoto <[email protected]> Mon, Apr 13, 2009 at 11:11 PM
To: Mike Hearn <[email protected]>It keeps getting extended all the time. If it stopped, an attacker would have time to catch up. Don't worry, empty blocks aren't very big.
As you say, it's the order of events that matters.
[Quoted text hidden]
Mike Hearn <[email protected]> Mon, Apr 13, 2009 at 11:18 PM
To: Satoshi Nakamoto <[email protected]>Oh yes, of course, that's fundamental. Silly me. Thanks for your answers. I'd recommend being over-explicit for early versions of the software, something like "Global chain is currently %d blocks long".
I guess the key problem right now is that once you generate coins, there's nobody to test it with, even for dummy transactions. Is there a plan for a mailing list or some kind of trivial marketplace to give people something to do with their newly minted bitcoins?
Satoshi Nakamoto <[email protected]> Tue, Apr 14, 2009 at 7:41 PM
To: Mike Hearn <[email protected]>I started implementing a marketplace feature earlier that facilitates offering things for sale and taking orders, it's only half done though. A bit like e-bay but without auctions, just "buy now". Among other things, it would make it easy for anyone to offer currency exchange.
If you send to 1PhUXucRd8FzQved2KGK3g1eKfTHPGjgFu and e-mail me your bitcoin address, or IP if you can accept incoming connections, I'll send back the same amount +50.
[Quoted text hidden]
Mike Hearn <[email protected]> Sat, Apr 18, 2009 at 3:08 PM
To: Satoshi Nakamoto <[email protected]>Hi Satoshi,
I sent you 32.51 coins, my bitcoin address is 1JuEjh9znXwqsy5RrnKqgzqY4Ldg7rnj5n
My IP is currently 84.73.233.199, however, it's a laptop so may or may not be online at the time you act on this mail. I suggest using the bitcoin address instead. It'd be convenient if the same comment functionality was available via indirect transfer. Can the comment be encrypted using the public key of the receiver and placed into a block?
[Quoted text hidden]
Satoshi Nakamoto <[email protected]> Sat, Apr 18, 2009 at 6:16 PM
To: Mike Hearn <[email protected]>I sent back 32.51 and 50.00.
I badly wanted to find some way to include a comment with indirect transfers, but there just wasn't a way to do it. Bitcoin uses EC-DSA, which was essential for making the block chain compact enough to be practical with today's technology because its signatures are an order of magnitude smaller than RSA. But EC-DSA can't encrypt messages like RSA, it can only be used to verify signatures.
[Quoted text hidden]
Mike Hearn <[email protected]> Sat, Apr 18, 2009 at 9:25 PM
To: Satoshi Nakamoto <[email protected]>Thanks. I sent you back 50, so now we're even.
For some reason your transfer to me shows up as "From: unknown" even though I added you to my address book.
I have a "Generated (not accepted)" line in my transaction list, it seems like an attempt to generate a coin went wrong somehow. Not sure
what happened here - presumably my node successfully solved a block but then I went offline before it was sent to the network?
I suppose for sending metadata with a transaction some other mechanism will be needed, for instance, broadcast of encrypted messages associated with a transaction that persist for (say) a month, with some kind of budget on how much storage a node can use for messages.
Alternatively, a payee could generate some reference number which is of some significance to themselves but otherwise opaque, and give it to the payer, thus it does not need to be encrypted and can be put into the block directly.
[Quoted text hidden]
Satoshi Nakamoto <[email protected]> Sat, Apr 18, 2009 at 10:52 PM
To: Mike Hearn <[email protected]>Got the 50.
Transactions sent to a bitcoin address will always say "from: unknown". The transaction only tells who it's to. Sending by bitcoin address has a number of problems, but it's so nice having the fallback option to be able to send to anyone whether they're online or not. There are a number of ideas to try to improve things later. For now, if things work out like the real world where the vast majority of transactions are with merchants, they'll pretty much always make sure to set up to receive by IP. The P2P file sharing networks seem fairly successful at getting a large percentage of their users to set up their firewalls to forward a port.
The "Generated (not accepted)" normally happens if two nodes find a block at close to the same time, one of them will not be accepted. It's normal and unavoidable. I plan in v0.1.6 to hide those, since they're just confusing and annoying and there's no reason for users to have to see them. While the network is still small like it is now, if you can't receive incoming connections you're at more of a disadvantage because you can't receive block announcements as directly.
[Quoted text hidden]
Mike Hearn <[email protected]> Sat, Apr 18, 2009 at 11:23 PM
To: Satoshi Nakamoto <[email protected]>Yes, I believe most P2P clients use the UPnP protocol to get routers to open up the port automatically. That would probably improve the listen rate significantly. I just discovered DMZ wasn't enabled on my router, though I thought it was. That's now fixed.
Is there a way to be told of new versions? Does the app auto update itself? Again, some kind of mailing list would be excellent.
I was thinking through how a practical micropayment implementation for the web might work in the last few days. One key issue is ensuring micropayments are fully automatic, yet can't be easily abused to drain the users account. I think the right approach would be to allow any website that presents an EV SSL cert to automatically request a micropayment, by default the browser always accepts as long as the charge is "low" and displays a small notification of what has occurred. Sites can then show that content requires payment in any way that suits their site design. Abusive sites that don't meet some simple guidelines (eg, showing unambiguously that clicking a link will trigger payment, or taking payment from direct search engine links) would simply have their SSL cert blacklisted, much like anti-phishing filters work today.
The protocol could be very straightforward and implemented by a Firefox extension or an IE BHO. Some static file (eg, a protocol buffer) is hosted on the site. It specifies the charge, a transaction description, the target IP and a URL for the browser to load after the transaction was accepted by the target node, to which the user
identifier is sent in a URL parameter. The site can then give back a cookie and the paywalled content. The entire process is automatic and simply results in, say, a little coin animation in the URL bar. Thus it's as convenient as regular web browsing. The users software would have some limit on what payments are automatically accepted.
The main problem with this approach is that somebody has to decide what the user interface guidelines are, then enforce them via blacklisting, as well as decide what payment requirements are low enough to be automatic vs requiring a user prompt. This introduces a trusted authority back into the system. However, it's one that the user can choose in an open market.
By the way, if you're not already using protocol buffers for the node-to-node traffic, I recommend them. We use them here at Google for everything, they solve a lot of versioning problems simply and efficiently.
Satoshi Nakamoto <[email protected]> Sun, Apr 19, 2009 at 2:14 AM
To: Mike Hearn <[email protected]>The list is:
[email protected]Subscribe/unsubscribe page:
http://lists.sourceforge.net/mailman/listinfo/bitcoin-listArchives:
http://sourceforge.net/mailarchive/forum.php?forum_name=bitcoin-listI'll always announce new versions there. Automatic update, or at least notification of new versions, is definitely on the list. There could potentially be necessary changes in the future where nobody will want to talk to you until you upgrade, and there needs to be code in the older version to convey that to the user. This is all the harder in the context of not trusting anyone.
Your approach to micropayments sounds right. At first, it might be a good idea to default to asking permission until the user gets comfortable and is ready to set it to automatic. The end goal though should get to something like you describe, where it's similar to using your cell phone without really having to think about the per minute charges.
I looked at Google protocol buffers when they were released last year, but I had already written everything by then. What I did was something similar to Boost Serialisation. For this application, where I was parsing messages from strangers who might have extreme incentive to hack the protocol, it was necessary to make it as basic as possible so I could crawl over every line of code to convince myself it was airtight. It became clear that any unnecessary degrees of freedom in the binary format multiplied the potential angles of attack. You guys are so right though to standardize across the company on protocol buffers. I think you've got the optimal solution in the general case.
The following are a series of emails from Satoshi Nakamoto to Hal Finney, written in January 2009 as the two were working on early versions of the bitcoin software. Mr. Finney supplied these emails to The Wall Street Journal in the spring of 2014.
Since these emails were all coming from Nakamoto to Mr. Finney, they are Nakamoto’s responses to Mr. Finney’s emails, the body of which is marked by the > symbol. The exchange begins on Jan. 10, 2009, and ends on Jan. 24, 2009, and comprises the time they were working on versions 0.1.0 through 0.1.3 of the
bitcoin software.
---------- Forwarded message ----------
From: Satoshi Nakamoto <[email protected]>
Date: Sat, Jan 10, 2009 at 11:52 AM
Subject: RE:Crash in bitcoin 0.1.0
To: [email protected]Normally I would keep the symbols in, but they increased the size of the EXE from 6.5MB to 50MB so I just couldn't justify not stripping them. I guess I made the wrong decision, at least for this early version. I'm kind of surprised there was a crash, I've tested heavily and haven't had an outright exception for a while. Come to think of it, there isn't even an exception print at the end of debug.log. I've been testing on XP SP2, maybe SP3 is something. I've attached bitcoin.exe with symbols. (gcc symbols for gdb, if you're using MSVC I can send you an MSVC build with symbols)
Thanks for your help!
>Hi Satoshi - I tried running bitcoin.exe from the 0.1.0 package, and
>it crashed. I am running on an up to date version of XP, SP3. The
>debug.log output is attached. There was also a file db.log but it was
>empty.
>
>The crash allowed me to start up a debugger, but there were no
>symbols. The exception was at address 00930AF7. The displayed call
>stack was 942316 called by 508936.
>
>When I have a chance, I'll try building it, although it looks like it
>would take me a while to acquire all the dependencies.
>
>Hal
From: Satoshi Nakamoto <[email protected]>
Date: Sat, Jan 10, 2009 at 2:59 PM
Subject: Re: Crash in bitcoin 0.1.0
To: [email protected]I was temporarily able to reproduce the bug and narrowed it down to the "mapAddresses.count" in the following code. It was absolutely the last piece of code to go in and mainly only got tested with the MSVC build. It's not essential and I'm inclined to turn off optimization and delete the section of code
until I figure out what's going on. I'm attaching a dbg exe you can try that deletes the line of code and turns off optimization. I'm not able to reproduce it anymore at the moment.
irc.cpp:
if (pszName[0] == 'u')
{
CAddress addr;
if (DecodeAddress(pszName, addr))
{
CAddrDB addrdb;
if (AddAddress(addrdb, addr))
printf("new ");
else
{
// make it try connecting sooner
CRITICAL_BLOCK(cs_mapAddresses)
if (mapAddresses.count(addr.GetKey()))
mapAddresses[addr.GetKey()].nLastFailed = 0;
}
addr.print();
}
else
{
printf("decode failed\n");
}
}
>Yes, actually the version with MSVC symbols would be better, that is
>the one I am using.
>
>I found that if I launched this one from a cygwin shell, it does not
>crash. But if I launch it from Windows, double-clicking on the file,
>it does crash similarly to the previous version. However, I am pretty
>sure that the previous version did crash even when I launched it from
>cygwin.
>
>I have to go out but I'll leave this version running for a while.
>
>Hal
---------- Forwarded message ----------
From: Satoshi Nakamoto <[email protected]>
Date: Sat, Jan 10, 2009 at 6:55 PM
Subject: Re: Crash in bitcoin 0.1.0
To: [email protected]I isolated the problem. If I spawn a thread and do mapAddresses.count, even as the very first thing in the program,
it segfaults. The workaround is to needlessly call mapAddresses.count in the main thread once and it's fine from then
on. I hate to blame the compiler, and I've never had a GCC compiler bug before, but this feels like one. Maybe some bit of init code it tries to optimize out if it's not called at least once in the same thread, or some STL optimization that's not thread
friendly. I'm really dismayed to have this botch up the release after all that stress testing.
The attached file: bitcoin-0.1.1.rar (filesize 2,132,686) is the version where I deleted the mapAddresses.count line, and that should be the safest version. (that was the only use of mapAddresses.count) If you could try this version and confirm that the crash is fixed, I'd appreciate it.
Thanks,
Satoshi
---------- Forwarded message ----------
From: Satoshi Nakamoto <[email protected]>
Date: Sat, Jan 10, 2009 at 7:11 PM
Subject: Re: Crash in bitcoin 0.1.0
To: [email protected]OK, thanks. The one in bitcoin-0.1.1-exe-dbg.rar is the same build as in bitcoin-0.1.1.rar. I forgot, when you build debug on MSVC, it uses the debug versions of the runtime DLLs, which aren't included with Windows distributions. Actually, MSVC 6.0's runtime (MSVC60.DLL) is the last version that shipped preinstalled on Windows, which is why the continued interest in that ancient version of the compiler. Later Visual C versions can't create a standalone EXE that doesn't require additional runtime packages installed.
I can't use MSVC 6.0 for the release because its optimization of the SHA-256 routines is too slow.
I've attached a copy of the debug runtime DLLs. (They're redistributable)
>Hi Satoshi - The version with the .pdb file did not run for me, I got
>an error about MSVCP60D.DLL not being found. I imagine this is due to
>the version incompatibility you were worried about.
>
>The next version, that deleted the questionable line of code and
>turned off optimization, seems to run fine for me. So the problem may
>be related to that bit.
>
>Hal
---------- Forwarded message ----------
From: Satoshi Nakamoto <[email protected]>
Date: Sun, Jan 11, 2009 at 4:36 PM
Subject: How's v0.1.2 going?
To: [email protected]Well this doesn't look good. After you upgraded to 0.1.2, your node responded to one or two messages and then stopped replying to messages. It's still accepting connections and seems to be alive on IRC. That could happen if ThreadSocketHandler or ThreadMessageHandler is hung or crashed or blocked. Usually when there's an exception or other problem, it only stops the affected thread and everything else keeps running. I'm attaching the msvc debug version in case you need it.
Satoshi
---------- Forwarded message ----------
From: Satoshi Nakamoto <[email protected]>
Date: Sun, Jan 11, 2009 at 4:49 PM
Subject: v0.1.2 gcc debug build attached
To: [email protected]Could you send me your debug.log?
The gcc debug version is attached.
gdb is easier to use than you'd think. gdb.exe is the only file. You run
gdb bitcoin.exe
then type "run"
then if it crashes, type "backtrace" for a stack dump, or it may do it automatically. (The stack trace
doesn't always go far enough back unfortunately)
---------- Forwarded message ----------
From: Satoshi Nakamoto <[email protected]>
Date: Sun, Jan 11, 2009 at 5:25 PM
Subject: Re: v0.1.2 debug.log
To: [email protected]OK, so no crash or exception window or anything. debug.log is all I need then. It looks like there's a "select failed: 10038" error (the sockets select function failed) and then network communication goes quiet after that (except for IRC which is still working). I've never had select fail before. It looks like sockets is somehow partially hosed. At least now I know what's wrong now. You should restart it. It's not doing anything right now. I don't know if it'll just get the "select failed" error again, or be fine for a while.
If I can't think of anything else, I can always shut down and restart sockets if it gets hosed like that. I'm sure everyone who's written an internet app like a browser or p2p app had to slog through all the ways the Internet can trash you. The Internet is a brutal, rough and tumble place.
The issue of bitcoin.exe still running after you close it is a known issue. It does a careful shutdown of everything to be extra safe, in case some important transaction is in progress, but it's completely fine and totally safe to just kill it if it doesn't exit on its own. I'll have to work on figuring out what's getting hung up. I may just have it kill itself after a timeout.
Thanks!
>Hi Satoshi - debug.log attached. When I started 0.1.2 this afternoon,
>I first quit the previous version which was running. However, 0.1.2
>would not start up. Looking at the debug log, it said "Existing
>instance found". I ran task manager, and found two processes called
>bitcoin.exe running. I killed them both and started up the new one,
>and it seemed to run OK. It says at the bottom "3 connections". I
>haven't tried the debug version, I'm not sure what I would look for.
>
>Hal
---------- Forwarded message ----------
From: Satoshi Nakamoto <[email protected]>
Date: Sun, Jan 11, 2009 at 9:31 PM
Subject: select failed 10038 fix
To: [email protected]I believe I've fixed the bug related to "select failed: 10038" (error WSAENOTSOCK). The select error is not a big deal, but it led the communications thread to get blocked on a socket that should have been in non-blocking mode but wasn't. It never came up until now because as long as select never failed, receive would never be called unless there was data.
Without this fix, your node's communication sometimes goes dead. Connections are still made, but no data is passed. Any generated blocks would probably not be accepted since you can't broadcast them and other nodes will leave your branch behind. That's why
Generate doesn't run when you're not connected. This could also have caused bitcoin.exe to fail to exit. There's no reason for shutdown to wait for the com thread, so I made it only wait for the message processing thread. I'll do a more thorough forced shutdown later. Looks like your node's com thread just now got blocked on this bug again. It went for a few hours this time before it did.
Version 0.1.3 exe attached.
---------- Forwarded message ----------
From: Satoshi Nakamoto <[email protected]>
Date: Mon, Jan 12, 2009 at 8:41 AM
Subject: Re: select failed 10038 fix
To: [email protected]It definitely looks like 0.1.3 solved it. It was getting so there were so many zombie nodes, I was having a hard time getting a reply to any of my messages. Now, four inventory messages go out, four getdata messages come back.
Did you get any "not accepted" blocks? The connectivity bug could have caused a generated block not to be accepted if the node wasn't able to broadcast at the time. Once the status is above 5 or so it's safely accepted.
Unfortunately, I can't receive incoming connections from where I am, which has made things more difficult. Your node receiving incoming connections was the main thing keeping the network going the first day or two. You can send to my Bitcoin address if you want to, but you won't get to see the full transfer sequence:
1NSwywA5Dvuyw89sfs3oLPvLiDNGf48cPD
You could always findstr /c:"version message" debug.log and send a test to some random person you're connected to near the end of the
list. The ones ending in port 8333 can receive connections. I just thought of something. Eventually there'll be some interest in brute force scanning bitcoin addresses to find one with the first few characters customized to your name, kind of like getting a phone number that spells out something. Just by chance I have my initials.
Satoshi
>Thanks, Satoshi, this new version seems to be running much better.
>I've got 8 connections, and watching debug.log there seems to be quite
>a bit of activity. I see you sent me a payment, thanks! Let me know
>your address and I will try sending one to you. I managed to generate
>a block yesterday and the coins are about to mature, if I understand
>it correctly.
>
>Hal
---------- Forwarded message ----------
From: Satoshi Nakamoto <[email protected]>
Date: Mon, Jan 12, 2009 at 10:50 AM
Subject: Re: select failed 10038 fix
To: [email protected]Could you send me the debug.log from the 0.1.3 crash?
I can usually get a lot just from that.
I'll send you the debug builds shortly.
>Looks like 0.1.3 crashed during the night, unfortunately. Next time I
>will try running the debug version. Today I am working and will need
>to take this computer up and down quite a bit, so I won't be able to
>run it for most of the day. Tonight I will try to look at it a little
>bit.
>
>Hal
---------- Forwarded message ----------
From: Satoshi Nakamoto <[email protected]>
Date: Mon, Jan 12, 2009 at 11:26 AM
Subject: Re: v0.1.3 msvc debug build
To: [email protected]Here's the 0.1.3 MSVC debug build
>Looks like 0.1.3 crashed during the night, unfortunately. Next time I
>will try running the debug version. Today I am working and will need
>to take this computer up and down quite a bit, so I won't be able to
>run it for most of the day. Tonight I will try to look at it a little
>bit.
>
>Hal
>
>On Mon, Jan 12, 2009 at 8:41 AM, Satoshi Nakamoto <
[email protected]> wrote:
>> It definitely looks like 0.1.3 solved it. It was getting so there
>> were so many zombie nodes, I was having a hard time getting a
>> reply to any of my messages. Now, four inventory messages go out,
>> four getdata messages come back.
>>
>> Did you get any "not accepted" blocks? The connectivity bug could
>> have caused a generated block not to be accepted if the node
>> wasn't able to broadcast at the time. Once the status is above 5
>> or so it's safely accepted.
>>
>> Unfortunately, I can't receive incoming connections from where I
>> am, which has made things more difficult. Your node receiving
>> incoming connections was the main thing keeping the network going
>> the first day or two.
>>
>> You can send to my Bitcoin address if you want to, but you won't
>> get to see the full transfer sequence:
>> 1NSwywA5Dvuyw89sfs3oLPvLiDNGf48cPD
>>
>> You could always findstr /c:"version message" debug.log and send a
>> test to some random person you're connected to near the end of the
>> list. The ones ending in port 8333 can receive connections.
>>
>> I just thought of something. Eventually there'll be some interest
>> in brute force scanning bitcoin addresses to find one with the
>> first few characters customized to your name, kind of like getting
>> a phone number that spells out something. Just by chance I have
>> my initials.
>>
>> Satoshi
>>
>>>Thanks, Satoshi, this new version seems to be running much better.
>>>I've got 8 connections, and watching debug.log there seems to be quite
>>>a bit of activity. I see you sent me a payment, thanks! Let me know
>>>your address and I will try sending one to you. I managed to generate
>>>a block yesterday and the coins are about to mature, if I understand
>>>it correctly.
>>>
>>>Hal
>>>
>>>On Sun, Jan 11, 2009 at 9:31 PM, Satoshi Nakamoto <
[email protected]> wrote:
>>>> I believe I've fixed the bug related to "select failed: 10038"
>>>> (error WSAENOTSOCK). The select error is not a big deal, but it
>>>> led the communications thread to get blocked on a socket that
>>>> should have been in non-blocking mode but wasn't. It never came
>>>> up until now because as long as select never failed, receive would
>>>> never be called unless there was data.
>>>>
>>>> Without this fix, your node's communication sometimes goes dead.
>>>> Connections are still made, but no data is passed. Any generated
>>>> blocks would probably not be accepted since you can't broadcast
>>>> them and other nodes will leave your branch behind. That's why
>>>> Generate doesn't run when you're not connected.
>>>>
>>>> This could also have caused bitcoin.exe to fail to exit. There's
>>>> no reason for shutdown to wait for the com thread, so I made it
>>>> only wait for the message processing thread. I'll do a more
>>>> thorough forced shutdown later.
>>>>
>>>> Looks like your node's com thread just now got blocked on this
>>>> bug again. It went for a few hours this time before it did.
>>>>
>>>> Version 0.1.3 exe attached.
---------- Forwarded message ----------
From: Satoshi Nakamoto <[email protected]>
Date: Mon, Jan 12, 2009 at 11:39 AM
Subject: Re: v0.1.3 gcc debug build
To: [email protected]and the gcc debug build w/gdb.exe
>Looks like 0.1.3 crashed during the night, unfortunately. Next time I
>will try running the debug version. Today I am working and will need
>to take this computer up and down quite a bit, so I won't be able to
>run it for most of the day. Tonight I will try to look at it a little
>bit.
>
>Hal
---------- Forwarded message ----------
From: Satoshi Nakamoto <[email protected]>
Date: Mon, Jan 12, 2009 at 11:59 PM
Subject: Re: select failed 10038 fix
To: [email protected]Definitely the disk full. I completely put off disk full handling until a later version. Probably about time I did it now.
Well, that's a relief.
Satoshi
>Hi Satoshi - Sorry I have not been able to do more today, this looks
>like a busy week for me. I started 0.1.3 again under the MSVC debugger
>this time so if it crashes tonight I may be able to get some more
>information.
>
>I remember now that last night, my disk filled up. I had downloaded a
>bunch of the dependencies (boost, etc) with an eye towards trying to
>build it myself, and my disk was already pretty full. I'm pretty sure
>this is what caused 0.1.3 to crash. I've attached the debug.log, which
>also includes some other runs. The error is about 1/3 of the way down
>and says,
>
>EXCEPTION: NSt8ios_base7failureE
>CAutoFile::read : end of file
>
>Normally this should be a rare occurrence with the large disk sizes
>people have today.
>
>Hal
>
>On 1/12/09, Satoshi Nakamoto <
[email protected]> wrote:
>> Could you send me the debug.log from the 0.1.3 crash?
>> I can usually get a lot just from that.
>>
>> I'll send you the debug builds shortly.
>>
>>
>>>Looks like 0.1.3 crashed during the night, unfortunately. Next time I
>>>will try running the debug version. Today I am working and will need
>>>to take this computer up and down quite a bit, so I won't be able to
>>>run it for most of the day. Tonight I will try to look at it a little
>>>bit.
>>>
>>>Hal
---------- Forwarded message ----------
From: Satoshi Nakamoto <[email protected]>
Date: Tue, Jan 13, 2009 at 2:42 PM
Subject: Re: disk full
To: [email protected]
If you build the dependencies, let me know how that goes.
Everything is always harder to build on Windows than Linux. I've
always hated projects with a lot of big dependencies, but there's
no avoiding it, each one is essential.
I still haven't figured out how you managed to get a read
exception rather than a write exception when your disk filled up.
It's unlikely but maybe possible that the incident could have
messed up your block data file. In that case, it might manifest
as a similar exception again, or if your block count in the status
bar stopped going up, that would also indicate a problem. As of
this moment it's at 375 blocks.
If there is a problem, it could easily be solved by deleting your
block files, as follows:
(exit Bitcoin and make sure it's stopped)
cd /d "%appdata%\bitcoin"
(backup this directory first)
del blk0001.dat
del blkindex.dat
It'll then re-download the block chain. Your transactions and
generated blocks show as 0/unconfirmed until it's done downloading.
The crucial file to backup is wallet.dat. If bitcoin is running
then you have to backup the whole %appdata%\bitcoin directory
including the database subdirectory, but even if it's not running
it certainly feels safer to always backup the whole directory.
The database unfortunately names its files "log.0000000001". To
the rest of the world, "log" means delete-at-will, but to database
people it means delete-and-lose-everything-in-your-other-files. I
tried to put them out of harm's way by putting them in the
database subdirectory. Later I'll write code to flush the logs
after every wallet change so wallet.dat will be standalone safe
almost all the time.
Satoshi
>Hi Satoshi - Sorry I have not been able to do more today, this looks
>like a busy week for me. I started 0.1.3 again under the MSVC debugger
>this time so if it crashes tonight I may be able to get some more
>information.
>
>I remember now that last night, my disk filled up. I had downloaded a
>bunch of the dependencies (boost, etc) with an eye towards trying to
>build it myself, and my disk was already pretty full. I'm pretty sure
>this is what caused 0.1.3 to crash. I've attached the debug.log, which
>also includes some other runs. The error is about 1/3 of the way down
>and says,
>
>EXCEPTION: NSt8ios_base7failureE
>CAutoFile::read : end of file
>
>Normally this should be a rare occurrence with the large disk sizes
>people have today.
>
>Hal
---------- Forwarded message ----------
From: Satoshi Nakamoto <[email protected]>
Date: Sat, Jan 24, 2009 at 4:47 PM
Subject: Re: disk full
To: [email protected]I hate duplicating code, but the compiler forces us. Copy the body
of the function above it, like this:
void insert(iterator it, const_iterator first, const_iterator last)
{
if (it == vch.begin() + nReadPos && last - first <= nReadPos)
{
// special case for inserting at the front when there's room
nReadPos -= (last - first);
memcpy(&vch[nReadPos], &first[0], last - first);
}
else
vch.insert(it, first, last);
}
#if !defined(_MSC_VER) || _MSC_VER >= 1300
void insert(iterator it, const char* first, const char* last)
{
if (it == vch.begin() + nReadPos && last - first <= nReadPos)
{
// special case for inserting at the front when there's room
nReadPos -= (last - first);
memcpy(&vch[nReadPos], &first[0], last - first);
}
else
vch.insert(it, first, last);
}
#endif
The modified version of serialize.h is attached.
BTW, in my tests, VC8 produced an EXE that would only run on
systems that had VC8 installed on them. The error it gives
is extremely vague. I think they expect you to install a
package during setup, but bitcoin doesn't have a setup.
My testing has been with MSVC 6.0 SP6 and GCC 3.4.5.
GCC is the release build. There's nothing wrong with the
MSVC 6.0 build other than its optimization of the SHA routines
for generating blocks is slow.
Satoshi
