@sipa Can you think of a cause, that leads to a massive spam of ORPHAN BLOCKS with current master during an initial chain sync in testnet?
Could that be caused by a BitCoinJ:0.6-SNAPSHOT/BC vend server:1.0 node somehow?
Dia
Topic: Ultraprune merged in mainline (Read 25403 times)
Apparently you had an unclean shutdown, and lost a data file. The debug.log is full of errors from LevelDB missing files.
This is interesting. First, it should complain much more loudly (like trying to recover, or exit with a fatal error). Secondly, this shouldn't happen... as far as I can see, there wasn't even an attempt to write to the blockdb. Did you copy the datadir from one system to another, or did your OS crash?
I think you are right, i had an unclean shutdown (Window 7, power off).This is interesting. First, it should complain much more loudly (like trying to recover, or exit with a fatal error). Secondly, this shouldn't happen... as far as I can see, there wasn't even an attempt to write to the blockdb. Did you copy the datadir from one system to another, or did your OS crash?
Never mind, i'll reinstall. Thanks for your time.
Apparently you had an unclean shutdown, and lost a data file. The debug.log is full of errors from LevelDB missing files.
This is interesting. First, it should complain much more loudly (like trying to recover, or exit with a fatal error). Secondly, this shouldn't happen... as far as I can see, there wasn't even an attempt to write to the blockdb. Did you copy the datadir from one system to another, or did your OS crash?
This is interesting. First, it should complain much more loudly (like trying to recover, or exit with a fatal error). Secondly, this shouldn't happen... as far as I can see, there wasn't even an attempt to write to the blockdb. Did you copy the datadir from one system to another, or did your OS crash?
Can you put your debug.log online somewhere, or send it to me?
sent to [email protected]btw: other 2 pre0.8 nodes are ok.
Warning: Displayed transactions may not be correct! You may need to upgrade, or othernodes may need to upgrade.
Can you put your debug.log online somewhere, or send it to me?
@sipa:
running pre0.8 edtion,
shutdown for 3 days,
now it's stuck at 209586, shows:
Warning: Displayed transactions may not be correct! You may need to upgrade, or othernodes may need to upgrade.
I guess it's stuck because of reorg ?
running pre0.8 edtion,
shutdown for 3 days,
now it's stuck at 209586, shows:
Warning: Displayed transactions may not be correct! You may need to upgrade, or othernodes may need to upgrade.
I guess it's stuck because of reorg ?
While I have your attention ... what's in the revXXXX.dat files ?
They contain undo data for blocks. If you see blocks are authenticated patches that modify the state of unspent transaction outputs forward in time, the rev* files contain data to go backward in time. These are needed when doing reorganisations.
Quote
Is the dataset in blkXXXX.dat a complete enough representation of the ledger
or must the info in revXXXX.dat somehow be taken into account ?
or must the info in revXXXX.dat somehow be taken into account ?
Yes, the rev* files are constructed from the normal block files when connecting blocks, so the information in them is redundant in practice.
Robustness is not what my block parser is about, speed is.
As a matter of fact, robustness is the very least of my concerns.
As a matter of fact, robustness is the very least of my concerns.
Read the reference loadblocks function anyway. Robust vs. Fast isn't always a trade-off. Whoever did this code did it right and got both. From my read of the function, it can find valid blocks buried under anything, even if an attacker managed to feed you a bogus file (for some strange reason), the worst it would do is waste a little of your time, and the parser is optimized to waste as little as possible.
At the end of the file you should see some zeroes, or other cruft, but there shouldn't be any in between the actual blocks. In fact, contrary to the old code, even when there's a crash in the middle of writing a block, there shouldn't ever be cruft except at the end of file (it remembers hoz much useful data is in each file, and starts appending/overwriting at that point).
Agreed. I couldn't find even a single byte out of place when I was parsing my block files, which were started with version 0.3.something and living on a computer that seems to crash every month or two. But writing the parser well didn't seem any harder than writing it poorly, and no slower in operation.
You are probably seeing the pre-allocation. The files are now allocated in blocks of 16 MiB, and only afterward filled in with data, to prevent fragmentation.
At the end of the file you should see some zeroes, or other cruft, but there shouldn't be any in between the actual blocks. In fact, contrary to the old code, even when there's a crash in the middle of writing a block, there shouldn't ever be cruft except at the end of file (it remembers hoz much useful data is in each file, and starts appending/overwriting at that point).
At the end of the file you should see some zeroes, or other cruft, but there shouldn't be any in between the actual blocks. In fact, contrary to the old code, even when there's a crash in the middle of writing a block, there shouldn't ever be cruft except at the end of file (it remembers hoz much useful data is in each file, and starts appending/overwriting at that point).
Pieter,
I am trying to make my blockparser utility work with your changes ...
In particular, it looks like blkXXXX.dat files do not obey the same structure as the
original blkXXX.dat files of the previous version (e..g sometimes the block magic
is not what I expect it to be).
I am trying to make my blockparser utility work with your changes ...
In particular, it looks like blkXXXX.dat files do not obey the same structure as the
original blkXXX.dat files of the previous version (e..g sometimes the block magic
is not what I expect it to be).
They should be exactly the same as <= 0.7.1: 4-byte pchMessageStart, 4-byte size, data.
If that is not the case, there is a bug that should be fixed.
They block magic is not always pchMessageStart.
It's very often zero, but looking at things closer, it looks like it's always close
to the end of the blkXXXX.dat, and therefore, I suspect it's an EOF mark.
The older blkXXX.dat did end exactly on the last byte of the last block.
Any chance you can do a hexdump, or otherwise narrow down what is being seen?
Just re-reviewed the code. Each record is written as described (4/4/block), with no EOF marks or anything else.
Perhaps your bitcoind crashed during a write, or a similar cause? Try downloading again, and see if you still have a corrupted blk*.dat.
Pieter,
I am trying to make my blockparser utility work with your changes ...
In particular, it looks like blkXXXX.dat files do not obey the same structure as the
original blkXXX.dat files of the previous version (e..g sometimes the block magic
is not what I expect it to be).
I am trying to make my blockparser utility work with your changes ...
In particular, it looks like blkXXXX.dat files do not obey the same structure as the
original blkXXX.dat files of the previous version (e..g sometimes the block magic
is not what I expect it to be).
They should be exactly the same as <= 0.7.1: 4-byte pchMessageStart, 4-byte size, data.
If that is not the case, there is a bug that should be fixed.
They block magic is not always pchMessageStart.
It's very often zero, but looking at things closer, it looks like it's always close
to the end of the blkXXXX.dat, and therefore, I suspect it's an EOF mark.
The older blkXXX.dat did end exactly on the last byte of the last block.
The parser in the reference client is extremely tolerant of extra cruft in the block files. Of course, it is only used to load extra block files and bootstrap files, but still. Since the real block files are never parsed sequentially in normal operation, they are allowed to be a little odd.
Take a look at the reference implementation for a solid, robust block file parser.
Sounds resonable that there are some unused bytes at the end of these files. Afaik they contain reserved space and if the conditions to start a new blockfile are met that space is left in the files, could that be the answer?
Dia
Dia
Pieter,
I am trying to make my blockparser utility work with your changes ...
In particular, it looks like blkXXXX.dat files do not obey the same structure as the
original blkXXX.dat files of the previous version (e..g sometimes the block magic
is not what I expect it to be).
I am trying to make my blockparser utility work with your changes ...
In particular, it looks like blkXXXX.dat files do not obey the same structure as the
original blkXXX.dat files of the previous version (e..g sometimes the block magic
is not what I expect it to be).
They should be exactly the same as <= 0.7.1: 4-byte pchMessageStart, 4-byte size, data.
If that is not the case, there is a bug that should be fixed.
Just tried to compile HEAD on FreeBSD:
Code:
localhost|/tmp/bitcoin-bitcoin-bb790aa/src> gmake -f makefile.unix
/bin/sh ../share/genbuild.sh obj/build.h
Building LevelDB ...
"Makefile", line 18: Need an operator
"Makefile", line 63: Missing dependency operator
"Makefile", line 65: Missing dependency operator
"Makefile", line 70: Need an operator
"Makefile", line 82: Need an operator
"Makefile", line 87: Need an operator
"Makefile", line 172: Missing dependency operator
"Makefile", line 194: Need an operator
"Makefile", line 200: Need an operator
make: fatal errors encountered -- cannot continue
g++ -c -O2 -pthread -Wall -Wextra -Wformat -Wformat-security -Wno-unused-parameter -g -DBOOST_SPIRIT_THREADSAFE -I/tmp/bitcoin-bitcoin-bb790aa/src -I/tmp/bitcoin-bitcoin-bb790aa/src/obj -I/usr/local/include -I/usr/local/include/db48 -DUSE_IPV6=1 -I/tmp/bitcoin-bitcoin-bb790aa/src/leveldb/include -I/tmp/bitcoin-bitcoin-bb790aa/src/leveldb/helpers -DHAVE_BUILD_INFO -fno-stack-protector -fstack-protector-all -Wstack-protector -D_FORTIFY_SOURCE=2 -MMD -MF obj/leveldb.d -o obj/leveldb.o leveldb.cpp
In file included from leveldb.cpp:5:
leveldb.h:144:2: warning: no newline at end of file
g++ -O2 -pthread -Wall -Wextra -Wformat -Wformat-security -Wno-unused-parameter -g -DBOOST_SPIRIT_THREADSAFE -I/tmp/bitcoin-bitcoin-bb790aa/src -I/tmp/bitcoin-bitcoin-bb790aa/src/obj -I/usr/local/include -I/usr/local/include/db48 -DUSE_IPV6=1 -I/tmp/bitcoin-bitcoin-bb790aa/src/leveldb/include -I/tmp/bitcoin-bitcoin-bb790aa/src/leveldb/helpers -DHAVE_BUILD_INFO -fno-stack-protector -fstack-protector-all -Wstack-protector -D_FORTIFY_SOURCE=2 -o bitcoind obj/alert.o obj/version.o obj/checkpoints.o obj/netbase.o obj/addrman.o obj/crypter.o obj/key.o obj/db.o obj/init.o obj/irc.o obj/keystore.o obj/main.o obj/net.o obj/protocol.o obj/bitcoinrpc.o obj/rpcdump.o obj/rpcnet.o obj/rpcmining.o obj/rpcwallet.o obj/rpcblockchain.o obj/rpcrawtransaction.o obj/script.o obj/sync.o obj/util.o obj/wallet.o obj/walletdb.o obj/noui.o obj/leveldb.o obj/txdb.o -Wl,-z,relro -Wl,-z,now -L/usr/local/lib -L/usr/local/lib/db48 -Wl,-Bdynamic -l boost_system -l boost_filesystem -l boost_program_options -l boost_thread -l db_cxx -l ssl -l crypto -Wl,-Bdynamic -l z -l pthread /tmp/bitcoin-bitcoin-bb790aa/src/leveldb/libleveldb.a /tmp/bitcoin-bitcoin-bb790aa/src/leveldb/libmemenv.a
g++: /tmp/bitcoin-bitcoin-bb790aa/src/leveldb/libleveldb.a: No such file or directory
g++: /tmp/bitcoin-bitcoin-bb790aa/src/leveldb/libmemenv.a: No such file or directory
gmake: *** [bitcoind] Error 1
Could you provide your debug.log file please?
debug.log: http://pastebin.com/mDYEG2d1
db.log: http://pastebin.com/fiXu6e2L
This contains the original "crash" and 2 tries of starting it (with the popups as described above).
The timer warning can apparently be a bug in Qt itself:
https://bugreports.qt-project.org/browse/QTBUG-18910
What version of Qt do you have installed?
https://bugreports.qt-project.org/browse/QTBUG-18910
What version of Qt do you have installed?
#> qmake --version
QMake version 2.01a
Using Qt version 4.7.2 in /usr/lib/qt4
#> pkg-config --modversion QtCore
4.7.2
I'm rather sure more recent Qt version on the Linux boxes would save a few headaches...
Dia
no headaches here. That "timer bug" only occurs at exit time.
Could you provide your debug.log file please?
debug.log: http://pastebin.com/mDYEG2d1
db.log: http://pastebin.com/fiXu6e2L
This contains the original "crash" and 2 tries of starting it (with the popups as described above).
The timer warning can apparently be a bug in Qt itself:
https://bugreports.qt-project.org/browse/QTBUG-18910
What version of Qt do you have installed?
https://bugreports.qt-project.org/browse/QTBUG-18910
What version of Qt do you have installed?
#> qmake --version
QMake version 2.01a
Using Qt version 4.7.2 in /usr/lib/qt4
#> pkg-config --modversion QtCore
4.7.2
I'm rather sure more recent Qt version on the Linux boxes would save a few headaches...
Dia
For those who are interested, I've created test binaries of the current development code.
Use with caution - several bugs were already fixed, and there are probably some left - so don't use for mining or merchant purposes.
The auto-import system is not yet implemented, so please run on a clean data directory. There are still issues with the inital block downloading, which require another major refactor to fix entirely, and with the recent improvements, those may now be the limiting factor. Use -connect to a trusted/local node, or use -loadblock or bootstrap.dat for initial sync, if you want to see the full potential speed.
The binaries are here. Use at your own risk.
Use with caution - several bugs were already fixed, and there are probably some left - so don't use for mining or merchant purposes.
The auto-import system is not yet implemented, so please run on a clean data directory. There are still issues with the inital block downloading, which require another major refactor to fix entirely, and with the recent improvements, those may now be the limiting factor. Use -connect to a trusted/local node, or use -loadblock or bootstrap.dat for initial sync, if you want to see the full potential speed.
The binaries are here. Use at your own risk.
nvm
Well the problem is probably that your old node had got stuck on block 178441 and wasn't making any further progress. I don't know why that might have happened though, was that block a special one (p2sh or something?).
It may be there's some kind of migration bug that only occurs when your node is in a particular stuck state. I'm not sure it's worth debugging all possible permutations of this given the ease of re-initialization.
It may be there's some kind of migration bug that only occurs when your node is in a particular stuck state. I'm not sure it's worth debugging all possible permutations of this given the ease of re-initialization.
This is possible. My old node wouldn't start any more ("out of memory").
So it's probably some special case, as you suspect, and I agree it's probably not worth the effort to dig into it.
The bad thing, however: it seems (at least to me as a user) to have screwed the wallet.dat (although it says the keys can still be loaded). Some users might have forgotten to back up and run into this. I wouldn't know how to progress from this state without a backup. I'm not sure a blockchain reload with the "screwed" wallet.dat would work... maybe I could try this (fresh .bitcoin dir + "screwed" wallet.dat)
Well the problem is probably that your old node had got stuck on block 178441 and wasn't making any further progress. I don't know why that might have happened though, was that block a special one (p2sh or something?).
It may be there's some kind of migration bug that only occurs when your node is in a particular stuck state. I'm not sure it's worth debugging all possible permutations of this given the ease of re-initialization.
It may be there's some kind of migration bug that only occurs when your node is in a particular stuck state. I'm not sure it's worth debugging all possible permutations of this given the ease of re-initialization.
Jump to: