Topic: A bitcoin blockchain parser in a few (thousand) lines of C++ (Read 16979 times)

Very simple. Sum of all unspent yet outputs.
The second question: how many bitcoins were lost?
The answer: sum of all unspent outputs which are provable unspendable

can this parser be used to do an audit of bitcoin total coins?

How do we know that there are exactly the number of coins in the blockchain as have been mined since the beginning?

https://code.google.com/p/blockchain/source/checkout

recently was refactored and now works grerat! thank you!

good news - duatiugame  refactored code and it`s working again:
https://code.google.com/p/blockchain/source/checkout

#define MAX_BITCOIN_ADDRESSES 60000000 // 60 million unique addresses.
#define MAX_TOTAL_TRANSACTIONS 70000000 // 70 million transactions.
#define MAX_TOTAL_INPUTS 250000000 // 250 million inputs.
#define MAX_TOTAL_OUTPUTS 250000000 // 250 million outputs
#define MAX_TOTAL_BLOCKS 600000      // 600,000 blocks.

I could not find a docs on how to do it - can you plz help? for example i want to get every address from blockchain and sort it in order of higher current balance.
Thank you!

Yes.

Hi! your tool is definitely usefull! however since blockchain become huge (~30 GB) and more then 52 mln addresses and 200+ inputs outputs - tool is no longer working ( moreover it`s not possible to recompile it changeing just:

#define MAX_BITCOIN_ADDRESSES 48000000 // 40 million unique addresses.
#define MAX_TOTAL_TRANSACTIONS 90000000 // 40 million transactions.
#define MAX_TOTAL_INPUTS 268000000 // 200 million inputs.
#define MAX_TOTAL_OUTPUTS 268000000 // 200 million outputs
#define MAX_TOTAL_BLOCKS 600000      // 1/2 million blocks.

because of it leads to Warning   16   warning C4307: '*' : integral constant overflow

these are the highest numbers tool is still compile-able however
it won`t let you poarse blockchain after.

It would be much appreciated if you can recompile it to be compatible with current blockchain size! or tips on how to do it )))
Thank you.

Can this be used to compute the bitcoin rich list (list of all addresses with balance greater than x)?

If you don't mind C (versus C++), picocoin's "blkstats" utility parses the blockchain in under 3 minutes.

https://github.com/jgarzik/picocoin/blob/master/src/blkstats.c
https://bitcointalksearch.org/topic/announce-picocoin-and-libccoin-c-based-bitcoin-library-and-client-128055
https://github.com/jgarzik/picocoin/

Have you tried clang++?
See http://stackoverflow.com/questions/16660854/compiling-with-clang-with-c11-enabled-fails
If this solves some errors but not all of them, then try
http://stackoverflow.com/questions/16352833/linking-with-clang-on-os-x-generates-lots-of-symbol-not-found-errors

is this topic alive? i'm trying to compile your software with gcc 4.5 on OS X Mavericks, but I get these errors:

Yes, I use the STL a lot, and mostly for that reason, because it's 'standard'.  However, I also try to avoid using it if I don't have to.  I won't go and write my own container class when the same container class already exists in STL but, on the other hand, if I can get away without using a container class at all, that's fine too.  I really try to keep it out of public API's if at all possible.  It just makes the code harder to follow I believe; so I typically might use it for internal implementation code but keep the public facing API as simple as possible.

The problems with the STL abound though, and they have been discussed a great deal online.  At my professional work, we have an internal standard not to use the STL.  Instead we have our own customized templates which we have highly optimized.

Our code here has to run on the following platforms:

Win32/Win64
Linux (many flavors)
Wii
Wii-U
Xbox-360
Xbox-720
PS3
PS4
Android
IOS

It's pretty crazy having to target high performance physics code on all of those platforms at the same time.

We also have a requirement that we capture every single memory allocation ever performed and never use any exception handling anywhere in the code base.

Game engines have some pretty high performance requirements, and the STL and exception handling usually don't fit this pattern.

That said, for my open source work, I always use the STL if I need a container.

John

Me communicate not so good.

I haven't reimplemented *any* cryptographic algorithms.  All I did was copy the two cryptographic routines I needed, and maybe slightly refactored the interface to them.

I just want the code I need and not the code I don't need and in a form which will compile without dragging in dependencies on hundreds of other source files and libraries (i.e. STL, BOOST, networking code, etc.)

To repeat, I haven't reimplemented any cryptographic algorithm, I've just tried to organize the source so I can easily build and understand it.

As I said before, if SHA256 were part of the C99 standard and built into every C/C++ compiler, that would be fine. But, it's not.

And I get that you guys have no problem building things like 'opensll' just because you need access to one or two routines; but that's not really my coding style.  By pulling out just the two routines I need into a standalone code snippet I am not 'reimplementing' anything, I'm just cleaning up an existing implementation to make it more usable and accessible and easier to understand.

After running CLOC on openSSL it reports a total of 1,589 files; 946 C files, 169 PERL scripts, 262 C/C++ header files, 76 makes files, 13 assembly files, 61 shell scripts, and a total of 234,537 lines of C code and 69,175 lines of code in header files.

In contrast, the code for *just* SHA256 and RIPMD160 the only two hash routines needed to parse the blockchain (which was my project goal) are 750 lines of code.

If you want to add a quarter million line source code project to your little utility program just so you can access 750 lines of code, go for it.  But, as I said, that's not my style.

John

P.S. For what it's worth, cryptopp doesn't seem as bad.  It's 270 source files and 'only' 56,000 lines of source.

There is little engineering upside to reimplementing highly complex cryptographic algorithms on your own, given the levels of engineering review and crypt-analysis of your own codebase versus an existing crypto lib.

It might be personally satisfying, but it makes little sense unless you are truly an expert crypto mathematician.

I do agree with the "minimal" approach although the STL and exceptions are *standard* C++ (if your compiler doesn't support these then it is not a C++ compiler at all but probably one of those horrible non-standard EC++ compilers from the 90's).

BTW if you want a fairly minimal SHA256 (one source and one header) then there is one in the CIYAM project here: https://github.com/ciyam/ciyam/blob/master/src/sha256.cpp and here: https://github.com/ciyam/ciyam/blob/master/src/sha256.h.

+1

Make it as simple as possible, but not simpler.

Let me reform the question using your analogy.

Let's say I'm working on a project and I need a log function.  Now, let's suppose that the log function (like SHA256) is not part of the C99 standard and built into the set of default libraries available in all compatible C++ compilers.

So, what if the only way to get access to a 'log' function is that I have to download some math library comprising 100,000 lines of source code with dependencies on streams, network code, a specific compiler, a complex build and configuration requirement, BOOST, the STL, exception handling, and god knows what else.

Because I needed access to 'log'.

Given the choice between dragging that into my code base or just copying only the 'log' function source code I would choose the much simpler solution.

People don't want to learn about algorithms by downloading 100,000 lines of source code libraries.

That's my position, and when I check with many of my colleagues in the computer game industry (where I come from) that tends to be our preference.

I'm a senior software engineer and I have been working on computer games and multi-platform development since 1979.

I just really prefer small code bases with minimal dependencies that are compiler and operating system agnostic.

John

Yeah, I know all about hashes. My issue isn't about any particular hash algorithm, my issue is having to download and build a massive library just to get access to the *one* I need.

Don't get caught up going down tangent paths.  Your issues with hashing sound much like someone who's never seen a logarithm before.  Reimplementing the log() function in C++ is not very useful nor educational, unless your goal is compiler optimizations or embedded software.  It's more important to understand the purpose and properties of the logarithm (hashing), than to distract those you are trying to educate with unnecessary details.  If I told you I didn't know what a logarithm but I needed it for my financial software, and I was working on implementing my own version of logarithm, you'd probably have the same reaction:  "go look up what a logarithm does and why it's important, and then just #include and focus on which properties of it are needed by the financial software and why."   

Hashing is the one of the most fundamental and important atoms of cryptographic systems.  Especially Bitcoin.  And implementing it from scratch is not likely to give you any insight into how it works or how it fits into Bitcoin.   What you need to understand is that it's a function that takes any input, completely mangles it, and produces a seemingly-random X-byte output.  It must always produce the same answer for identical input, but change any bit should look equivalent to producing a random value.   It is intentionally non-invertible (i.e. SHA256 can take in any size string and always spits out 32 bytes), and each output bit should have no correlation with any input bit. 

In the context of Bitcoin transactions, it's goal is to produce a unique 32-byte identifier for a transaction, regardless of how many kB it is.  If any byte of the transaction is changed, the hash will look like a different 32-byte random number.  In the context of headers, this "randomness" to create a sort of lottery for miners.  Since we know the hash function should produce essentially-uniform randomness, we know that for a given difficulty and hashrate on the network, it should take approx 10 min on average for someone to find a lucky hash.

Similarly, if you go any further with this, you're going to need an ECDSA library in addition to hashing.  While elliptic curves are mathematically neat, you would be best to not reimplement it yourself unless you are working on speed-optimizing it.  If you think publickey->address was hard, try privatekey->publickey! (no, don't try it, just get a library and understand the properties of it).

I did one more cleanup pass/revision before the end of today.

There is now a code snippet which does just base58 encoding.  It's based on the source code in 'cbitcoin' but I removed all memory allocation.

I also revised the BitcoinAddress snippet to go from public-key to bitcoin binary address, and from a bitcoin binary address to ASCII and from ASCII back to binary (checked).

Finally, my original snippet folds in all of this code to parse the bulk of the blockchain.  I still haven't yet assembled all of the individual transactions, I would do that the next time I find some time to spend on this effort.

All that said, I think these individual snippets, which are pretty well documented, will hopefully prove useful educational tools for future developers who won't have to go through the same trouble I went through figuring out how all of these pieces fit together.

Like a previous poster said, I prefer looking at source code as documentation than a Wiki page; so hopefully others will find this useful too.

http://codesuppository.blogspot.com/2013/07/bitcoin-code-snippets.html