Topic: Experimental pre-0.8 builds for testing - page 3. (Read 19841 times)

Sorry, but I think I should disagree strongly! This might to be an (or better the!) important bottleneck for each newbie who want to start to use bitcoin-qt.
You are true, it is no bottleneck for you and probably for > 95% of all forums posters here who has done already the initial download of the blockchain data. "crappy block download and signature validation" becomes the next bottle neck AFTER you have your personal block chain in a bitcoin-client-conform format!

I also dislike to see it to be replaced  (the currently undocumented "blkindex.dat"-format) by new formats in rev*dat, blk*dat, coins-directory (and  blktree-diretory?), if these stay undocumented or to be over-complex.  Again making it impossible to use directly this raw block chain data. I think I have shown with my block-parser, that conversion of this data into a bitcoin-qt usable structure can be done only in a few minutes on a typical hardware and not in more than 20 times real time for bitcoin-turbo and checking everything inside the main block chain except the script-verification! Yes, before, 0.7.?, it were more than 200-times of real time, so a great achievement of your new experimental 0.7.99 version.

Please remember: I only started activity in this Bitcoin Forum, because the VERY SLOW initialization with the main block chain by the (most spread) bitcoin-qt client might put off increasingly many want-to-be starters into the bitcoin world.


smtp

Sorry, but you remind me only dark fuzzy remembers decades ago that such a command exist(ed).

1) my ulimit has no -c option (only a -f option) on Linux (not only on my gentoo, but also on Ubuntu 10.2 I just checked)
2) called "ulimit" gives the default of unlimited
but most important
3) There occurs no crash in gdb, thus it is not meaningfull (in my context) to talk about "use the debugger after the crash".

BTW: I just discovered the ulimit with many options on SunOS. "ulimit -c" sets the "core" limit of the process there. I wonder what core precisely is meant.

Anyway, thanks for your thoughts
smtp

I just wanted to remind you about "ulimit -c unlimited" to be able to meaningfully use the debugger after the crash. You're probably running with the default of "ulimit -c 0".

I just notice this reindex-bug is a really nasty bug. If I restart bitcoin-qt next (only with -maxconnection=0) it starts immediately with reindexing. :-(
I quite it and start it again, it starts reindexing ...

Lots of lines
"ProcessBlock: ORPHAN BLOCK, prev=0000000000000640dd32c57bbda456631968a2d57ccc3d40190c767ab2ce7e99"
now writen to debug.log.

Ups! debug.log was truncated by bitcoin-qt -- only 77030 lines of these "ProcessBlock: ORPHAN BLOCK, prev=...."  inside now.

Well, I better should throw all away now .... and wait for Pieter to debug it.

smtp

2013-01-14 22:38:44 Reindexing finished
 98:20 min CPU-time until block-height 216823  and 99:16 min until 21650 height (the maxium no of blocks in the blk*dat files).

I lookup up in the debug.log : Start at 2013-01-14 20:42:55 which means 116:11 min total real-time or 85.4 % CPU-time.

Compare this with the bootstrap.dat of 216823 blocks using 150 min CPU-time. It should be about equal - but I did a simple trick. *gg*
I recompiled the turbo-bitcoin/sources with the g++ additional options -march=athlon64 -O2
:-)

Nearly a boost of 50% in speed up! I wonder why there is also in the official sources not at least a -O2 option for the g++-compiler.
Everything will profit from -O2.

BTW: This "./bitcoin-qt -maxconnections=0 -reindex -dbcche=500 -logtimestamps -benchmark"  (also with -dbcache inplace of -dbcche) crashes
the client reproducable after start. Sadly not, if invoked with gdb. :-(

smtp

I don't know, but I believe
I used the same options as
except this -par=4

smtp

You mean -dbcache, right? I also observed a crash while reindexing with current master, but I didn't yet take a closer look at it.

Dia

Hi

There is highly probable a bug in the "-reindex" option called code.

After updating with the bitcoin-qt-0.7.99 to the current main blockchain , this works fine, I restarted it again:

./bitcoin-qt -maxconnections=0 -reindex -dbcche=500 -logtimestamps -benchmark

and it crashed after a few seconds without any error message.
Here is the tail of the debug.log

2013-01-14 20:37:15 Opened LevelDB successfully
2013-01-14 20:37:15  block index              51ms
2013-01-14 20:37:15 Loading wallet...
2013-01-14 20:37:15 nFileVersion = 79900
2013-01-14 20:37:16  wallet                  976ms
2013-01-14 20:37:16 Loading addresses...
2013-01-14 20:37:16 Reindexing block file blk00000.dat...
2013-01-14 20:37:16 Loaded 10094 addresses from peers.dat  40ms
2013-01-14 20:37:16 mapBlockIndex.size() = 0
2013-01-14 20:37:16 nBestHeight = -1
2013-01-14 20:37:16 setKeyPool.size() = 100
2013-01-14 20:37:16 mapWallet.size() = 0
2013-01-14 20:37:16 mapAddressBook.size() = 1
2013-01-14 20:37:16 Done loading
2013-01-14 20:37:16 refreshWallet

Maybe you get an idea.
BTW: Could you explain the new format of these blk*.dat files (and if easy, the rev*.dat) files or is this anywhere already documented?

Appendum: I just want to give you more debugging info and therefore started "gdb ./bitcoin-qt" and then inside
run "-maxconnections=0 -reindex -dbcche=500 -logtimestamps -benchmark"

and it runs ...
:-/

smtp

I have never claimed otherwise. That is not the bottleneck now, however (crappy block download and signature validation are).


It should report block connect/validation speeds in debug.log.

Finished:

/usr/bin/time ./bitcoin-qt -detachdb -benchmark
Application asked to unregister timer 0x3a000004 which is not registered in this thread. Fix application.
8680.81user 338.18system 4:18:23elapsed 58%CPU

Elapsed time is much too high, because it waits 1.5 hours until I finished it manually - 2 h 45 min real-time for loading the bootstrap.dat,
93.8 % CPU time looks right and about 12 minutes I/O-time until finish loading.
It only needs 1-2 seconds to finish bitcoin-qt after I have clicked the Quit-Button (-detachdb no longer needed?).
Total size of directory "blocks" is 5417516 kB containing 2*37 files (about 11% overhead, old blkindex.dat had 32% overhead compared to the blk00*.dat).
Directory "coins" has size 164324 kB.

I think there is still a lot of room for speed-optimization of the load of the initial bootstrap.dat. Moreoever I see no effect of this "-benchmark" option (maybe this ".....Fix application." message?)

BTW: I run it with "maxconnections=0" set in the bitcoin.conf file such that at the end of the load no further action was done.

smtp

block height 21000 was reached after 82 min cpu-time, currently it is at block height 214045 needed 128 min Cpu-time and got 93 % CPU-time in the real-time. And I discovered that the new blocks are stored in the sub-directory blocks together with rev*.dat files. I still have the feeling no signature-verification is done . It procesess currently with 439 tx/sec (no txin-count availible in debug.log). I think it will finish (reaching block height 216283) in the next 20 minutes when I'm at lunch.

smtp

Hi

Last night I only started this bitcoin-qt with an empty $HOME/.bitcoin to see whether it principle worked.
Now the bitcoin-qt 0.7.99 is running first time seriously on my hardware (2 GHz Athlon-64 3200+ stepping 00 with a total of 4 GB ECC-Ram).

Firstly (an experiment) I had set two softlinks to the existing blk0001.dat and blk0002.dat and started with "./bitcoin-qt  -reindex -detachdb", but it immediately crashed without error-message (no bootstrap.dat exists).

Next I built the new bootstrap.dat  of size 4883237032 (block height 216283), deleting theses links to blk00*.dat and started
"/usr/bin/time ./bitcoin-qt -detachdb -benchmark" Now it runs
and seems to process all these blocks. After 15 min CPU-time consumed, it still had ~ 59000 left . I wonder why it is so slow, for signature-checkng too fast
but still uses mostly 98% CPU-time, some drops from time to time, but I estimate still > 90% CPU-time on average. Periodcially disk-writes occurs, but because it has taken 3.3 GB system-cache this doesn't really matter to real-time behaviour. Currently, I just look, at remaining ~ 31600 blocks, it still
processes as about 10 - 20 blocks /sec (difficult to estimate). I wonder what it is doing internally with such a high cpu-usage. Disk-reads are fairly sparse in real-time.


... to be continued (now 39 min CPU-time used and 23400 blocks left to be processed).

BTW: System is a flavor of Linux 2.6.32-gentoo-r7 x86_64.

smtp

Sorry, I deduced (wrongly) from the fact that if it are your files (URL https://github.com/sipa/bitcoin/tree/turbo) then at least you were responseable for this "-static" patch, sipa.


Please remove this total misleading message:

fatal: Not a git repository (or any parent up to mount parent )
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set)

from the build-process, especially tragic if a different error occurs later which could have been founded (for an outsider) on this reason.

And one more proposal for the next official bitcoin-release:

Mention in the doc/build-unix.txt the possibility to customize theses 6 variables
# Dependency library locations can be customized with:
#    BOOST_INCLUDE_PATH, BOOST_LIB_PATH, BDB_INCLUDE_PATH,
#    BDB_LIB_PATH, OPENSSL_INCLUDE_PATH and OPENSSL_LIB_PATH respectively
in the bitcoin-qt.pro file.

BTW: If noone made errors, nobody could learn.

smtp

Sorry, that is fixed in the last revision of the stack-protector pull now .

Dia

So it was related to that single " I mentioned a few days ago on Github, HA .

Dia

I have no clue at all about GUI stuff. Wladimir wrote that.

Because the bitcoin-turbo.tar.gz file was not changed which I downloaded after you just had announced the remove of the syntax error in bitcoin-qt.pro  today in this thread, I looked into your git-repository and copied the new changed bitcoin-qt.pro and made a diff to my downloaded in the tar-ball.

A last question for today (it got very late and I must get up tomorrow early): What is the x11-resource name of the color which backgrounds the "This is a pre-release ...." warning in the wallet-window and which controls the red of "(not synchron)" , such that I can change these specific colors?

smtp

In such a setup I'd advise you to disable listening, as that will reduce the amount of nodes trying to fetch block data from you significantly.

Oh, that's not a change I made, though I should have caught it.

The -static was intended to only be added for Windows.

Dear Pieter

I'm a very little bit upset due to your quick and dirty patching of your current bitcoin-turbo/bitcoin-qt.pro file.


Why?

After disabling UPNP-support via the line
UPNP=-
in the bitcoin-qt.pro file everything works fine until the final linkage. There I got the error message:

g++ -static -fstack-protector-all -Wl,-O1 -Wl,-rpath,/usr/lib64/qt4 -o bitcoin-qt build/bitcoin.o build/bitcoingui.o ......
......  -L/usr/X11R6/lib -lQtCore -lgthread-2.0 -lrt -lglib-2.0 -lpthread

/usr/lib/gcc/x86_64-pc-linux-gnu/4.4.5/../../../../x86_64-pc-linux-gnu/bin/ld: cannot find -lboost_system

But these library seems to be in the standard /usr/lib directory. Thus I thought either the line was too long (2486 characters) or some kind of order
problem with the -L$LIBPATH -llibrary was the reason. After about a quarter hour -- I couldn't believe why g++ suddenly doesn't see these libraries --  my eyes
catched at the beginning of the line: "-static " -- and remembered the diff of the new made bitcoin-qt.pro file from today and its predecessor in my downloaded *.tar.gz file. You have inserted

# use static linking
QMAKE_LFLAGS *= -static

in the bitcoin-qt.pro  newly!

I removed the "-static" from the above g++ line and no problem to do the default dynamic library linkage, getting a 13 MB huge dynamical linked 64bit-executable bitcoin-qt! You should tell everybody, that you now expect all libraries to be statically linked! A new feature of the upcoming 0.8.0-release?

BTW: Of course I like to encourage (dispite of all my critic) to continue your valuable work.

Best wishes,
achim