FWIW the set of unspent tx-out's should fit nicely in RAM, for some time to come.
I know that's not helpful to applications doing historical queries, but it may be helpful for other queries.
Topic: . - page 7. (Read 39124 times)
Finally, it's a good idea to do this: in the end the blockchain will anyway grow
so big that even the hashmaps won't fit in RAM, and my approach will start
swapping like crazy. Pre-indexing in a disk-level DB hash table is the right thing
to do.
Well that's why I started looking into this. My original blockchain utilities design in Armory were more of an educational adventure to see how fast I could get the scanning from scratch (no external libraries), with lots of pointers, etc. It was educational, and I was happy with the results. But I had the same realization that even the index will eventually not fit in RAM, and thus there is no way to make the library scalable without using some kind of disk-based engine. To that end, I had no interest re-inventing disk-based DB operations, etc. When I heard the devs talking about LevelDB, I looked into and quickly realized why they chose it. It is very simple, portable... and apparently fast!
However, gmaxwell just told me how to clear my disk cache and then retest. It was dramatically slower for sure. Probably by a factor of 5-10x. However, I never ran the same test using my other blockchain utilities, so I don't know the actual comparison. Given that modern OS'es are good at doing that caching, the LevelDB version is clearly a very good solution.
I'm currently investigating how I can transition Armory to maintain it's own blockchain data now with LevelDB
I don't intend to discourage or insult your efforts (in fact, I think you said you were getting similar speeds). Plus, I think that writing such a parser/scanner is both useful and educational. I just bring it up because I would guess that this is about as good as anyone can do with a non-SSD drive. And it's especially impressive considering I did no optimizations. I'm especially curious how leveldb works under the hood that it could be so efficient. Maybe there's something to learn from it.
LevelDB provides the backend to Google's services, so it is not surprising that it is so heavily optimized. One of the key optimizations is that writes go to a log, and then another thread comes along in the background and "optimizes" the log data back into the main database, using hueristics to hopefully place the data in a useful, locality-friendly location on disk.
BitcoinJ author Mike Hearn has proposed replacing BDB with LevelDB in the Satoshi client, and we are looking seriously at that. Here is his pull request: https://github.com/bitcoin/bitcoin/pull/1619
I'm sure you know I've spent a lot of time doing blockchain scanning, and attempting to optimize it in various ways in Armory. In the end, the fastest I could achieve was about 20-25s on a full blk0001.dat (2GB) on Linux-64bit using memory-mapping. I couldn't achieve the same thing in Windows (mmap'ing seems to be garbage there). I just wanted to share my experience from playing with leveldb, how awesomely optimized it is.
First, I iterated through blk0001.dat and stored all transactions in a leveldb database using Key:TxHash-->Value:RawTx. I plugged 4 addresses into a set object. I then iterated (out of order) through the entire database, pulling each transaction into memory from the DB, and then searching for TxOuts matching any of those 4 addresses. This took 8.0 seconds! (but was only the TxOuts). This included putting the TxOuts into a map (OutPoint, nBTC), so that I could then scan the entire DB again and search for TxIns.
At the end of both scans, I had the balance and unspent TxOut list from those four addresses. It took approximately 12-13 seconds! And this is with a non-SSD drive: it seems like it shouldn't even be possible to go that quickly... I had to quadruple-check everything to verify that I wasn't missing something or accidentally supplying the answer. Perhaps there was some OS-level caching of the DB files, though I would've expected such caching to improve my custom blockchain utilities performance.
I don't intend to discourage or insult your efforts (in fact, I think you said you were getting similar speeds). Plus, I think that writing such a parser/scanner is both useful and educational. I just bring it up because I would guess that this is about as good as anyone can do with a non-SSD drive. And it's especially impressive considering I did no optimizations. I'm especially curious how leveldb works under the hood that it could be so efficient. Maybe there's something to learn from it.
First, I iterated through blk0001.dat and stored all transactions in a leveldb database using Key:TxHash-->Value:RawTx. I plugged 4 addresses into a set
At the end of both scans, I had the balance and unspent TxOut list from those four addresses. It took approximately 12-13 seconds! And this is with a non-SSD drive: it seems like it shouldn't even be possible to go that quickly... I had to quadruple-check everything to verify that I wasn't missing something or accidentally supplying the answer. Perhaps there was some OS-level caching of the DB files, though I would've expected such caching to improve my custom blockchain utilities performance.
I don't intend to discourage or insult your efforts (in fact, I think you said you were getting similar speeds). Plus, I think that writing such a parser/scanner is both useful and educational. I just bring it up because I would guess that this is about as good as anyone can do with a non-SSD drive. And it's especially impressive considering I did no optimizations. I'm especially curious how leveldb works under the hood that it could be so efficient. Maybe there's something to learn from it.
What's your definition of "taint"?
If there is 6% taint from pirate in that huge address, does that mean 6% of these BTC were at one point in time owned by a pirate address or that pirate deposited 6% of these coins to that address?
If it is the first definition, isn't there the possibility to have >100% total taint? E.g. pirate sends BTC to MtGox and then MtGox sends these to the huge address - now they are "tainted" by pirate AND "tainted" by MtGox...
What I would still like to do is to enter an address, get the closure of that and then get the closures of every address where money was sent to or received from that cluster (ideally I could enter how many levels deep, with "0" being the current closure setting).
If there is 6% taint from pirate in that huge address, does that mean 6% of these BTC were at one point in time owned by a pirate address or that pirate deposited 6% of these coins to that address?
If it is the first definition, isn't there the possibility to have >100% total taint? E.g. pirate sends BTC to MtGox and then MtGox sends these to the huge address - now they are "tainted" by pirate AND "tainted" by MtGox...
What I would still like to do is to enter an address, get the closure of that and then get the closures of every address where money was sent to or received from that cluster (ideally I could enter how many levels deep, with "0" being the current closure setting).
Why does this address: 12BdKh5HHUyGFduGm6y8nY4KS1LxzsduXo
Link to this address: 12cbQLTFMXRnSzktFkuoG3eHoMeFtpTu3S
Closure reports it has no outputs then returns that one.
Link to this address: 12cbQLTFMXRnSzktFkuoG3eHoMeFtpTu3S
Closure reports it has no outputs then returns that one.
I see the same.
It tells me:
Code:
error: specified key was never used in a TX output
but this transaction has 12BdKh5HHUyGFduGm6y8nY4KS1LxzsduXo as an output, so the message is a little confusing.
And then the address isn't in its own reported cluster - that's got to be a bug.
OK, got a question now:
Why does this address: 12BdKh5HHUyGFduGm6y8nY4KS1LxzsduXo
Link to this address: 12cbQLTFMXRnSzktFkuoG3eHoMeFtpTu3S
Closure reports it has no outputs then returns that one.
On a side note, script written and code modified to take a wallet address, and then just output the balance of the final wallet
Why does this address: 12BdKh5HHUyGFduGm6y8nY4KS1LxzsduXo
Link to this address: 12cbQLTFMXRnSzktFkuoG3eHoMeFtpTu3S
Closure reports it has no outputs then returns that one.
On a side note, script written and code modified to take a wallet address, and then just output the balance of the final wallet
Code:
runlinux@ubuntu:~/blockparser$ ./wallet.sh 1MrPonziaM4QT2S7SdPEKQH88BGa4LRHJU
transactions = 211
received = 207.30710185
spent = 207.30680287
balance = 0.00029898
transactions = 211
received = 207.30710185
spent = 207.30680287
balance = 0.00029898
- at least I can read the code! 
Temporarily set HOME to wherever you want it to read the blockchain from:
Already did that. That's where I symlinked the file to.Quote
I doubt the parser made any blockchain files. It opens the files readonly, which won't create any files.
I didn't say it did?Quote
About a week ago the blockchain file grew bigger than 2GB and so a 2nd file was created by bitcoin-qt:
Quote
$ ls -l blk*
-rw------- 1 chris chris 2097307549 2012-07-12 20:07 blk0001.dat
-rw------- 1 chris chris 104659251 2012-07-19 02:52 blk0002.dat
-rw------- 1 chris chris 751022080 2012-07-19 02:52 blkindex.dat
-rw------- 1 chris chris 2097307549 2012-07-12 20:07 blk0001.dat
-rw------- 1 chris chris 104659251 2012-07-19 02:52 blk0002.dat
-rw------- 1 chris chris 751022080 2012-07-19 02:52 blkindex.dat
I don't know where the 0003 file came from though!
Because I already had a blk0001.dat, my client started with blk0002.dat.
Tried it some other way.
I have a specific problem. I had a blk0001.dat already when I first started the bitcoin-client. Instead of using that file it made a blk0002.dat and recently it made a blk0003.dat. The parser won't open the blockchain file: "Invalid argument: failed to open block chain file /home/jouke/.bitcoin/blk0001.dat"
I guessed that 0002 was actually 0001 and tried to symlink 0002 to 0001 and 0003 to 0002 in a separate directory, but your parser still won't open the file.
Any sugestions?
Temporarily set HOME to wherever you want it to read the blockchain from:
Quote
$ HOME=/path/to/folder ./parser closure
[...]
fatal: No such file or directory: failed to open block chain file /path/to/folder/.bitcoin/blk0001.dat
[...]
fatal: No such file or directory: failed to open block chain file /path/to/folder/.bitcoin/blk0001.dat
I doubt the parser made any blockchain files. It opens the files readonly, which won't create any files.
About a week ago the blockchain file grew bigger than 2GB and so a 2nd file was created by bitcoin-qt:
Quote
$ ls -l blk*
-rw------- 1 chris chris 2097307549 2012-07-12 20:07 blk0001.dat
-rw------- 1 chris chris 104659251 2012-07-19 02:52 blk0002.dat
-rw------- 1 chris chris 751022080 2012-07-19 02:52 blkindex.dat
-rw------- 1 chris chris 2097307549 2012-07-12 20:07 blk0001.dat
-rw------- 1 chris chris 104659251 2012-07-19 02:52 blk0002.dat
-rw------- 1 chris chris 751022080 2012-07-19 02:52 blkindex.dat
I don't know where the 0003 file came from though!
Tried it some other way.
I have a specific problem. I had a blk0001.dat already when I first started the bitcoin-client. Instead of using that file it made a blk0002.dat and recently it made a blk0003.dat. The parser won't open the blockchain file: "Invalid argument: failed to open block chain file /home/jouke/.bitcoin/blk0001.dat"
I guessed that 0002 was actually 0001 and tried to symlink 0002 to 0001 and 0003 to 0002 in a separate directory, but your parser still won't open the file.
Any sugestions?
Ok, it does not compile on g++ 4.7, it does on g++ 4.5
OK.
Let me install 4.7 and see what's what.
Fixed and fix pushed to github
Compiles nicely
xubuntu 12.10 or something like that
Code:
In file included from cb/allBalances.cpp:4:0:
./util.h: In instantiation of ‘void GoogMap::Map::setEmptyKey(const Key&) [with Key = const unsigned char*; Value = Addr*; Hasher = Hash160Hasher; Equal = Hash160Equal]’:
cb/allBalances.cpp:67:37: required from here
./util.h:122:21: error: ‘set_empty_key’ was not declared in this scope, and no declarations were found by argument-dependent lookup at the point of instantiation [-fpermissive]
./util.h:122:21: note: declarations in dependent base ‘google::dense_hash_map > >’ are not found by unqualified lookup
./util.h:122:21: note: use ‘this->set_empty_key’ instead
In file included from cb/taint.cpp:25:0:
./util.h: In instantiation of ‘void GoogMap::Map::setEmptyKey(const Key&) [with Key = const unsigned char*; Value = long double; Hasher = Hash256Hasher; Equal = Hash256Equal]’:
cb/taint.cpp:70:35: required from here
./util.h:122:21: error: ‘set_empty_key’ was not declared in this scope, and no declarations were found by argument-dependent lookup at the point of instantiation [-fpermissive]
./util.h:122:21: note: declarations in dependent base ‘google::dense_hash_map > >’ are not found by unqualified lookup
./util.h:122:21: note: use ‘this->set_empty_key’ instead
In file included from cb/transactions.cpp:5:0:
./util.h: In instantiation of ‘void GoogMap::Map::setEmptyKey(const Key&) [with Key = const unsigned char*; Value = int; Hasher = Hash160Hasher; Equal = Hash160Equal]’:
cb/transactions.cpp:64:37: required from here
./util.h:122:21: error: ‘set_empty_key’ was not declared in this scope, and no declarations were found by argument-dependent lookup at the point of instantiation [-fpermissive]
./util.h:122:21: note: declarations in dependent base ‘google::dense_hash_map > >’ are not found by unqualified lookup
./util.h:122:21: note: use ‘this->set_empty_key’ instead
make: *** [.objs/taint.o] Error 1
make: *** Waiting for unfinished jobs....
make: *** [.objs/allBalances.o] Error 1
make: *** [.objs/transactions.o] Error 1
In file included from parser.cpp:2:0:
./util.h: In instantiation of ‘void GoogMap::Map::setEmptyKey(const Key&) [with Key = const unsigned char*; Value = const unsigned char*; Hasher = Hash256Hasher; Equal = Hash256Equal]’:
parser.cpp:449:29: required from here
./util.h:122:21: error: ‘set_empty_key’ was not declared in this scope, and no declarations were found by argument-dependent lookup at the point of instantiation [-fpermissive]
./util.h:122:21: note: declarations in dependent base ‘google::dense_hash_map > >’ are not found by unqualified lookup
./util.h:122:21: note: use ‘this->set_empty_key’ instead
./util.h: In instantiation of ‘void GoogMap::Map::setEmptyKey(const Key&) [with Key = const unsigned char*; Value = Block*; Hasher = Hash256Hasher; Equal = Hash256Equal]’:
parser.cpp:450:32: required from here
./util.h:122:21: error: ‘set_empty_key’ was not declared in this scope, and no declarations were found by argument-dependent lookup at the point of instantiation [-fpermissive]
./util.h:122:21: note: declarations in dependent base ‘google::dense_hash_map > >’ are not found by unqualified lookup
./util.h:122:21: note: use ‘this->set_empty_key’ instead
make: *** [.objs/parser.o] Error 1
In file included from cb/closure.cpp:4:0:
./util.h: In instantiation of ‘void GoogMap::Map::setEmptyKey(const Key&) [with Key = const unsigned char*; Value = long unsigned int; Hasher = Hash160Hasher; Equal = Hash160Equal]’:
cb/closure.cpp:43:38: required from here
./util.h:122:21: error: ‘set_empty_key’ was not declared in this scope, and no declarations were found by argument-dependent lookup at the point of instantiation [-fpermissive]
./util.h:122:21: note: declarations in dependent base ‘google::dense_hash_map > >’ are not found by unqualified lookup
./util.h:122:21: note: use ‘this->set_empty_key’ instead
make: *** [.objs/closure.o] Error 1
./util.h: In instantiation of ‘void GoogMap
cb/allBalances.cpp:67:37: required from here
./util.h:122:21: error: ‘set_empty_key’ was not declared in this scope, and no declarations were found by argument-dependent lookup at the point of instantiation [-fpermissive]
./util.h:122:21: note: declarations in dependent base ‘google::dense_hash_map
./util.h:122:21: note: use ‘this->set_empty_key’ instead
In file included from cb/taint.cpp:25:0:
./util.h: In instantiation of ‘void GoogMap
cb/taint.cpp:70:35: required from here
./util.h:122:21: error: ‘set_empty_key’ was not declared in this scope, and no declarations were found by argument-dependent lookup at the point of instantiation [-fpermissive]
./util.h:122:21: note: declarations in dependent base ‘google::dense_hash_map
./util.h:122:21: note: use ‘this->set_empty_key’ instead
In file included from cb/transactions.cpp:5:0:
./util.h: In instantiation of ‘void GoogMap
cb/transactions.cpp:64:37: required from here
./util.h:122:21: error: ‘set_empty_key’ was not declared in this scope, and no declarations were found by argument-dependent lookup at the point of instantiation [-fpermissive]
./util.h:122:21: note: declarations in dependent base ‘google::dense_hash_map
./util.h:122:21: note: use ‘this->set_empty_key’ instead
make: *** [.objs/taint.o] Error 1
make: *** Waiting for unfinished jobs....
make: *** [.objs/allBalances.o] Error 1
make: *** [.objs/transactions.o] Error 1
In file included from parser.cpp:2:0:
./util.h: In instantiation of ‘void GoogMap
parser.cpp:449:29: required from here
./util.h:122:21: error: ‘set_empty_key’ was not declared in this scope, and no declarations were found by argument-dependent lookup at the point of instantiation [-fpermissive]
./util.h:122:21: note: declarations in dependent base ‘google::dense_hash_map
./util.h:122:21: note: use ‘this->set_empty_key’ instead
./util.h: In instantiation of ‘void GoogMap
parser.cpp:450:32: required from here
./util.h:122:21: error: ‘set_empty_key’ was not declared in this scope, and no declarations were found by argument-dependent lookup at the point of instantiation [-fpermissive]
./util.h:122:21: note: declarations in dependent base ‘google::dense_hash_map
./util.h:122:21: note: use ‘this->set_empty_key’ instead
make: *** [.objs/parser.o] Error 1
In file included from cb/closure.cpp:4:0:
./util.h: In instantiation of ‘void GoogMap
cb/closure.cpp:43:38: required from here
./util.h:122:21: error: ‘set_empty_key’ was not declared in this scope, and no declarations were found by argument-dependent lookup at the point of instantiation [-fpermissive]
./util.h:122:21: note: declarations in dependent base ‘google::dense_hash_map
./util.h:122:21: note: use ‘this->set_empty_key’ instead
make: *** [.objs/closure.o] Error 1
From the README:
Code:
sudo apt-get install libssl-dev build-essential g++-4.4 libboost-all-dev libsparsehash-dev git-core perl
Did you do that ?
yes,
http://pastebin.com/tQUq28Zk
Jump to: