Topic: Python client - page 2. (Read 36708 times)

Nice work! I made some small changes.

nBestHeight is the number of blocks minus one, which means it's probably the index of the last block the host knows about. That's so the nodes can choose who should ask for blocks after the initial version message exchange. I don't know what the rest of those are, except that nLocalHostNonce is used (at least partially) to make sure the client isn't connecting to itself and nLocalServices appears to always be 1.
[/quote]

Don't forget "tx", used to insert a transaction.

I started the Wiki page here http://code.google.com/p/pybitcoin/wiki/BitcoinProtocol

So far we have the version message pretty worked out, except the nLocalServices, nLocalHostNonce and the nBestHeight, which I can't fathom what they are used for.

As for the other messages I have identified these so far:

• version
• verack
• getdata
• addr
• inv
• getaddr
• block
• ping
• reply
• subscribe
• sub-cancel
• publish
• pub-cancel

Thanks mate

Sure. I sent you a PM. Anyone who wants to be part of the project just drop me a PM with your Google Account / Gmail email address.

The on google code or github wiki looks very nice to document our efforts. Could I get access to it?

--EDIT--
Here's my (sloppy) Python script to read and parse the initial version message. Run it with no arguments to connect to localhost or one argument to connect to the computer at that IP / hostname.
http://www.alloscomp.com/bitcoin/version.pys
I also decided to host the code on GitHub and create a Google Code page. I'll be happy to give commit access / read-write privs to the Google Code page to anyone who's interested.
Sample output:
--/EDIT--

Pretty close.
Absent for version messages. I think it will be included for every message _after_ version though.

This is in host byte order afaict:

I'm getting some weird stuff here: my string is 3 bytes long and starts with 0x02:
"0x02 0x2E 0x30" ("\02.0"). It's also not 0x00 terminated. On an older client with no subversion string, it's just one 0x00.

Oh nevermind: it's one byte of length followed by the string.
So 0x02: ".0". On the older client, the string's length is 0. That's much better than I anticipated.

Everything else in your summary looks good!

Finally, it looks like checksum is only calculated when message size > 0. That's wise. Checksum also doesn't appear to be sent for "version" messages even when one client knows the other is new enough to accept the checksum.

Upon connecting to one another, both clients send a "version" message. With that complete, they both send "verack" upon receiving the other's version message. Then one sends an "inv" message that appears to contain a list of transactions IDs that node knows about. The other node then queues the transactions it doesn't know about to "mapAskFor" and executes a getdata request (or several - it cuts them when their size > 1000) for those transactions.

Great, I was able to reconstruct the IP address from my dump and it turns out correctly. So just to sum it up:

version

• {0xf9,0xbe,0xb4,0xd9}
• "version" (0x00 padded)
• 4 byte message size
• 4 byte checksum
• 8 byte nLocalServices (always 1 if !fClient, no idea either what that means)
• 8 byte timestamp (remember to use network byte order)
• Remote address (the address this Node thinks he is):
  
  • nServices - uint64 (8b), still cryptic, don't know the meaning yet
  • pchReserved - (12b): some reserved space, apparently for later IPv6
  • ip - uint (4b)
  • port - unsigned short (2b)
• Local address (the address this Node sees you under):
  
  • nServices - uint64 (8b), still cryptic, don't know the meaning yet
  • pchReserved - (12b): some reserved space, apparently for later IPv6
  • ip - uint (4b)
  • port - unsigned short (2b)
• 8 byte nLocalHostNonce (needed for a handshake, if I'm not mistaken)
  
• A subversion string ".0" in my case
• nBestHeight - int (4b): appears to be the last block number

Well I think clients in other languages would be very valuable. I would particularly enjoy looking through a Java client. There may be some unknown danger to connecting a different client to the production Bitcoin network, but you can aways just code it to only connect to the test network and include a note in the source code requesting that people not connect to the production network. I'm sure if it was very thoroughly tested and received plenty of attention, eventually no one would have a problem with it connecting to the production server.

From my research, the header is _not_ considered in the size. It cannot be considered in the checksum, since the checksum would have to be known to calculate the checksum.

Here's the relevant code. Some comments added. I don't know what exactly the Hash() function does yet.
It's from net.h:703-716.

The problem I'm having is parsing the version command's data. Version _should_ be made of these things serialized, as per net.h:580:

However, VERSION = 304 according to my client, and here's the hex dump of the data I'm getting:
You know, I just realized: the checksum is 304... So my header parsing code must be doing something wrong. The header _is_ supposed to be 24 bytes, right?

--EDIT--
Alright, I've been playing with the "version" message the client sends automatically when you connect to it. First of all, as much as I respect Satoshi, I really don't like this format. Something like protocol buffers would have been much neater and more efficient.

Secondly, I suspect that this message doesn't have a checksum because it can't be sure if the client it is talking to has a version > 209 until it gets a verack or version message. I'll play with that next, but I think the checksum is omitted initially to deal with older clients.

Here's my analysis of the components of the data section of this message:

Now for the address struct, here's the IMPLEMENT_SERIALIZE block:
from net.h:222-233.

From what I can tell, this indicates that the over the wire format is:
Interestingly, the IP and Port are in Big-Endian / Network Byte Order, while everything in the main struct is in Little-Endian / Host Byte Order

That sounds reasonable :-)
Any idea what Checksum algorithm this is? And is the header (4bytes and message type) also considered in the checksum and message size?

Edit: Just checked: the size is the number of bytes following the size byte itself, I guess that the checksum is for that range also.

According to main.cpp:1761 (svn r113):

I'm starting to reverse engineer the protocol, but my C++ is incredibly rusty 
As far as I understand it the protocol is based on simple messages each 16 bytes long with variable argument lists. Each message starts with the sequence {0xf9,0xbe,0xb4,0xd9}, and the command itself is 12 bytes (0x00 padded). Depending on the type of the message we have to parse some arguments. Thus far I've analyzed the "version" command, whose goal is to exchange some initial information (both parties send and receive a version message):

version

• {0xf9,0xbe,0xb4,0xd9}
• "version" (0x00 padded)
• 4 byte message size
• 4 byte checksum
• 8 byte nLocalServices (always 1 if !fClient, no idea either what that means)
• 8 byte timestamp (remember to use network byte order)
• Two adresses which I was unable to recognize (~44 bytes)
  
• 8 byte nLocalHostNonce (needed for a handshake, if I'm not mistaken)
• A subversion string ".0" in my case
  

The adresses are exchanged only if we are not proxied, this way the nodes can learn their external address.

Anyone else interested in byte-guessing 

I strongly believe that a system like BitCoin is only destined to survive for a long time if the protocol is standardized and split from the Mainline client development.

A minimalistic, but clean, Python implementation of the client could be used as a reference for future clients that could then be custom tailored to the needs of its users.

https://bitcointalksearch.org/topic/bitcoins-implementations-221 etc

Where did you hear that?

Hi!

I had almost working pure perl client, but since I've heard alt clients are not welcomed, i've abandoned that project.

--
mtve

I've been tempted... but no, I'm more interested in doing web-based Bitcoin apps, and extending the existing C++ implementation's JSON-RPC API is a whole lot less work than re-implementing the whole shebang.

Are you going to be working on a full client yourself? 

I've started reverse-engineering and documenting the wallet and block databases, and have written some Python code that deserializes many of the Bitcoin data structures.  I was going to let it ferment a little more before announcing, but Mr. Google will surely find and index it soon, and it should be a good head start for anybody who wants to start a Python bitcoin client.

Open source, MIT license, at:
  https://code.google.com/p/bitcointools/
  MOVED TO git:  http://github.com/gavinandresen/bitcointools

I was wondering whether someone here might be interested in creating an alternative client to the Project, which would not only make it easier for some people to get going with BitCoin but also would make it easier to write customized clients and take the load off the main development team.

For this to work we'd have to know exactly how the protocol is designed. I could reverse engineer it but I probably won't be able to explain everything

BTW: Python is not my first choice when it comes to programming languages but for this I think it'd be the most flexible.

Edit: just thought of it: Google App Engine supports Python so there's another application area