Topic: Satoshi Client Operation: Overview - page 3. (Read 64633 times)

Subscribe.

Instresting, abandon downloading blockchains from .323 client speed the downloading up.

Thanks bitrick! I have not seen this information elsewhere.

Great job!

these are quiet educational, but it will take me days to digest as a newbie programmer.

Thanks Bitrick.

Thanks Gavin!
The first patch is the most promising. I actually tested turning off VerifySignature last week and calculated around double the block processing rate, when CPU limited. I was hoping for more, but every bit helps.

The last two patches I have less hope for. I know 0.3.23 can disconnect inappropriately, but in my testing any such disconnects should only result in a "brief" stall (brief == a few minutes of a multi-hour process). Note that the client should not disconnect a connection until all incoming and outgoing buffers are drained, so even if there is a disconnect, all the blocks queued prior to that can still get processed, so there should not be totally unproductive ongoing disconnections. (Or did I misunderstand the nature of the 0.3.23 problem?) I did not see a lot of 0.3.23 disconnect delays in my testing but maybe I just got lucky.
 
Thanks everyone for the nice comments. To reiterate: I do plan on putting this information into the wiki for future reference (and thanks for the suggestion Eto).

p.s. I am not Satoshi.

Very nice bitrick!

By the way, this patch speeds up initial download quite a lot:
  https://github.com/gavinandresen/bitcoin-git/commit/042a619709fab1329e8286c6aedbb2cdc8eb3497

...as do these, which have already been pulled into git head:
  https://github.com/bitcoin/bitcoin/commit/fb45259967032d409bca4d542b55414a7c522fba
  https://github.com/bitcoin/bitcoin/commit/ec74e8a44338202bfb82faa2cef4611cc37e7fa5

Excellent stuff!

Are you satoshi? 

I don't have the time to read or use these info but they look really awesome -> wiki!

Bitrick, this is very good information.  It is exactly the kind of information I wanted, as someone who wants to get involved in BTC development but doesn't have the advanced C++ software skills needed to comprehend the Satoshi client code.  I look forward to using this as a secondary reference in my work, and helping iron out the details.

I would recommend you put links to each of the other posts at the bottom of this "Overview" post, and then one of the admins make this thread sticky.  I think people would expect to see this kind of information when they come to the "Development & Technical Discussion" forum,  and right now this is the most concise yet seemingly-complete version of this information I've seen so far.

Nice work!
-Eto

I haven't read all of this yet either, but it looks like an amazing piece of badly needed documentation. Thank you so much.

I didn't yet read all of this, but it seems like a terrific job! Thanks for doing this!

I suggest putting it somewhere, like a wiki, and adding some images and formatting for better presentation.

Thank you very much for this. I haven't had the time to go over everything in detail, but from what I've seen, it is correct and well researched. Please make sure this ends up on the wiki - people wanting to contribute need this kind of information.

I'm only part way through 2 of yr articles so far but I just wanna say thanks for an awesome job.  This sort of documentation is exactly what new devs need to get across the client quickly.

Are you planning to publish this series on a site somewhere?  It would be great to have it all in one place where people can bookmark it and find it easily for reference.  It's going to get buried eventually in the forum.

Satoshi's Original Bitcoin Client - An Operational View

Preface
---------

I thought my client was taking too long to download the block chain and it
did not appear to operate smoothly. I thought I could do something to decrease
the block download time. So I downloaded the code and dug in.  Ultimately,
I failed to find the silver bullet to eliminate the long download delays
(big suprise!). But I did manage to penetrate the C++ code and figure
out how things worked for the most part.

So, I decided to write down my understanding of the code from an operational
perspective, to spare those who are not fluent in C++ from having to wade
through the code, which is quite dense and bit of a chore to pick apart,
when they really just want to know "how it works".

My focus was initially on the block download process, but I decided to
go ahead and cover all the major operational aspects I could (before losing
interest . I do think I found some areas for improvement, but that is not
the point of these articles. I will try to make it clear when I am stating
the facts versus when I am writing commentary.

I intend these articles to go into the Wiki at some point but I also
thought it would be useful to open topics in the forum in order to
allow for review in case I made a mistake or missed something big,
and for reference.


Overview
------------

This series of articles will focus on how the Satoshi bitcoin client
program operates, and less so on the protocol details and the rules
for processing blocks and transactions.

Satoshi's bitcoin client is a C++ program, so be sure to look for code in
both the .cpp and the .h header files. Also, the program is multithreaded.
This leads to some complexity and the use of certain code patterns to deal
with concurrency that may be unfamiliar to many programmers. Also, the
code is aggresive in the use of C++ constructs, so it will help to be
fluent with map, multmap, set, string, vector, iostream, and templates.


For information on how the bitcoin protocol works, see:
    The original Satoshi whitepaper:
        http://bitcoin.org/bitcoin.pdf
    The articles on the bitcoin.it Wiki:
        https://en.bitcoin.it/wiki/Category:Technical
    With special mention of the protocol specification:
       https://en.bitcoin.it/wiki/Protocol_specification
    And the protocol rules:
       https://en.bitcoin.it/wiki/Protocol_rules


-- Operations --

The client is oriented around several major operations, including:

    Initialization and Startup
        Upon startup, the client performs various initilization routines
        including starting multiple threads to handle concurrent operations.

    Node Discovery
        The client uses various techniques find out about other bitcoin
        nodes that may exist.

    Node Connectivity
        The client initiates and maintains connections to other nodes.

    Sockets and Messages
        The client processes messages from other nodes and sends
        messages to other nodes using socket connections.
    
    Block Exchange
        Nodes advertise their inventory of blocks to each other and
        exchange blocks to build block chains.

    Transaction Exchange
        Nodes exchange and relay transactions with each other.
        The client associates transactions with bitcoin addresses in the
        local wallet.

    Wallet Services
        The client can create transactions using the local wallet.
        The client associates transactions with bitcoin addresses in the
        local wallet. The client provides a service for managing
        the local wallet.

    RPC Interface
        The client offers an JSON-RPC interface over HTTP over sockets
        to perform various operational functions and to manage the local
        wallet.

    User Interface
        The user interface code is scheduled to be superseded by bitcoin-qt.
        Therefore, it is not covered in further detail.

See their individual articles for more detail on each of these operations.


-- fClient Mode --

It is worth noting that there is code in the client to allow it to
operate in a mode where it only downloads block headers.
The implementation is intended to be used as a lightweight client mode which
can operate without verifying and storing all blocks and transactions.

This is controlled by the fClient variable in the code which is currently
hard coded to false.  This is currently not considered to be finished code.

This mode is known as fClient mode and the phrase Simplified Payment
Verification (or SPV) mode has also been used to describe a lightweight
client approach.



-- Main Thread Level Functions --

init.cpp:
    main()
    ExitTimeout
    Shutdown
net.cpp:
    StartNode
    ThreadGetMyExternalIP
    ThreadMapPort
    ThreadSocketHandler
    ThreadOpenConnections
    ThreadMessageHandler
rpc.cpp:
    ThreadRPCServer
irc.cpp:
    ThreadIRCSeed
db.cpp:
    ThreadFlushWalletDB
ui.cpp:
    ThreadDelayedRepaint
    SendingDialogStartTransfer


-- Significant Classes By File --

net.cpp/.h:
    CNode:  handes one socket connection
    CInv
    CAddress
    CMessageHeader
    CRequestTracker

main.cpp/.h:
    CDiskTxPos
    CInPoint
    COutPoint
    CTxIn
    CTxOut

    CTransaction
    CMerkleTx
    CTxIndex

    CBlock
    CBlockIndex
    CDiskBlockIndex
    CBlockLocator

    CAlert : CUnsignedAlert

wallet.cpp/.h
    CWallet : CKeyStore
    CReserveKey
    CWalletTx : CMerkleTx
    CWalletKey
    CAccount
    CAccountingEntry

db.cpp/.h:
    CTxDB
    CKeyPool
    CWalletDB

bignum.h
    CBigNum

util.h
    CCriticalSection: used for thread contention


--
Search on "Satoshi Client Operation" for more articles in this series.

Transaction Exchange : https://bitcointalksearch.org/topic/satoshi-client-operation-transaction-exchange-41730
Block Exchange : https://bitcointalksearch.org/topic/satoshi-client-operation-block-exchange-41729
Sockets and Messages : https://bitcointalksearch.org/topic/satoshi-client-operation-sockets-and-messages-41727
Node Connectivity : https://bitcointalksearch.org/topic/satoshi-client-operation-node-connectivity-41726
Node Discovery : https://bitcointalksearch.org/topic/satoshi-client-operation-node-discovery-41722
Initialization and Thread Startup : https://bitcointalksearch.org/topic/satoshi-client-operation-initialization-and-thread-startup-41719