Ultraprune merged in mainline - page 2.

molecular

donator

Activity: 2772

Merit: 1019

original problem description: https://bitcointalksearch.org/topic/m.1290009

Quote from: molecular on October 22, 2012, 02:08:31 PM

Quote from: Mike Hearn on October 22, 2012, 11:19:42 AM

Could you provide your debug.log file please?

debug.log: http://pastebin.com/mDYEG2d1
db.log: http://pastebin.com/fiXu6e2L

This contains the original "crash" and 2 tries of starting it (with the popups as described above).

Quote from: Mike Hearn on October 22, 2012, 11:19:42 AM

The timer warning can apparently be a bug in Qt itself:

https://bugreports.qt-project.org/browse/QTBUG-18910

What version of Qt do you have installed?


#> qmake --version
QMake version 2.01a
Using Qt version 4.7.2 in /usr/lib/qt4
#> pkg-config --modversion QtCore
4.7.2

sipa suggested on #bitcoin-dev to try again with empty .bitcoin directory. I did (with loadblock= from old moved-away file)

it synced allright (30 hours), of course I had fresh empty wallet.

I then swapped in the pre-ultraprune (0.7.0 I think) wallet backup and voila: everything works.

This means I can't reproduce the error, at least not without re-loading the whole chain and the old wallet in place while doing that. Is this something I should do? Mabybe with newest git? Or just ignore the problem?

ChrisKoss

full member

Activity: 169

Merit: 100

Mods, could you fork this discussion to keep it on topic?

Atlas, please make a new thread if you wish to discuss something that is not on-topic for this thread.

molecular

donator

Activity: 2772

Merit: 1019

Quote from: Mike Hearn on October 22, 2012, 11:19:42 AM

Could you provide your debug.log file please?

debug.log: http://pastebin.com/mDYEG2d1
db.log: http://pastebin.com/fiXu6e2L

This contains the original "crash" and 2 tries of starting it (with the popups as described above).

Quote from: Mike Hearn on October 22, 2012, 11:19:42 AM

The timer warning can apparently be a bug in Qt itself:

https://bugreports.qt-project.org/browse/QTBUG-18910

What version of Qt do you have installed?


#> qmake --version
QMake version 2.01a
Using Qt version 4.7.2 in /usr/lib/qt4
#> pkg-config --modversion QtCore
4.7.2

gmaxwell

staff

Activity: 4284

Merit: 8808

Quote from: Transisto on October 22, 2012, 11:35:27 AM

I'd like to have this implemented before moving anyone to a pruned chain.

It's a good thing this has nothing to do with moving anyone to a pruned chain.

Transisto

donator

Activity: 1731

Merit: 1008

I'd like to have this implemented before moving anyone to a pruned chain.

https://bitcointalk.org/index.php?topic=100779.20
(bitcoind saturating upload and affecting whole connection reliability)

Mike Hearn

legendary

Activity: 1526

Merit: 1134

Could you provide your debug.log file please?

The timer warning can apparently be a bug in Qt itself:

https://bugreports.qt-project.org/browse/QTBUG-18910

What version of Qt do you have installed?

molecular

donator

Activity: 2772

Merit: 1019

hey guys, decided to check out ultraprune (git git://github.com/bitcoin/bitcoin from about 20 hours ago) and ran into problems:

First, I wanted to load the old block files. bitcoin-qt started up and showed me that it's downloading (or loading?) blocks. Pretty slowly (slow atom cpu here). Next morning I see this:

Quote

#> ./bitcoin-qt -loadblock=~/.bitcoin/blk0001.dat -loadblock=~/.bitcoin/blk0002.dat
Application asked to unregister timer 0x2b000010 which is not registered in this thread. Fix application.

Trying to start again gives the following popup:

And then:

And exit:

Quote

#> ./bitcoin-qt
bitcoin-qt: /usr/include/boost/thread/pthread/recursive_mutex.hpp:62: boost::recursive_mutex::~recursive_mutex(): Assertion `!pthread_mutex_destroy(&m)' failed.
Aborted

So I thought maybe I should report this here. Need more info?

Mike Hearn

legendary

Activity: 1526

Merit: 1134

The original LevelDB branch I made did an auto-migration, actually. It was a pretty bad hack and it seems to have got lost when sipa rebased it onto ultraprune, but it's not like we don't think about these things.

Maged

legendary

Activity: 1204

Merit: 1015

Quote from: gmaxwell on October 21, 2012, 11:31:09 PM

Quote from: Maged on October 21, 2012, 11:19:39 PM

I'm not worried that it won't be added now. I was just concerned that such a simple upgrade path didn't appear to have been considered, because statements like that would have usually added a mention that those commandline options will not be needed by release in order to avoid a full re-download.

It's been considered and discussed, including trade-offs between upgrading in a way that allows switching back to old versions (but requiring a lot of extra disk space) or cleaning up... or offering a choice at first start up.

You're reading too much into a technology and development announcement. This isn't release notes. Tongue

Glad to hear it.

gmaxwell

staff

Activity: 4284

Merit: 8808

Quote from: Maged on October 21, 2012, 11:19:39 PM

I'm not worried that it won't be added now. I was just concerned that such a simple upgrade path didn't appear to have been considered, because statements like that would have usually added a mention that those commandline options will not be needed by release in order to avoid a full re-download.

It's been considered and discussed, including trade-offs between upgrading in a way that allows switching back to old versions (but requiring a lot of extra disk space) or cleaning up... or offering a choice at first start up.

You're reading too much into a technology and development announcement. This isn't release notes. Tongue

Maged

legendary

Activity: 1204

Merit: 1015

Quote from: Luke-Jr on October 21, 2012, 10:29:31 PM

Quote from: Maged on October 21, 2012, 10:02:28 PM

Quote from: jgarzik on October 20, 2012, 09:35:13 PM

Quote from: Syke on October 20, 2012, 09:25:44 PM

Quote from: Pieter Wuille on October 20, 2012, 05:37:51 PM

I've just merged my "ultraprune" branch into mainline (including Mike's LevelDB work).

Does this require downloading and re-processing the blockchain from the beginning?

Yes. However, to save downloading, you may provide

Code:

-loadblock=DATA_DIR/blk0001.dat -loadblock=DATA_DIR/blk0002.dat

to import the old data files into the new bitcoin database backend (ultraprune/leveldb).

* "DATA_DIR" should be replaced with the directory where your blockchain was stored in <= 0.7.1.

It doesn't do this automatically? It really should before this is officially released. You could even just add them as implied bootstrap.dat files.

I'm sure 0.8 will do this (and more!). Seriously, we just released 0.7.1 (from master)... 0.8 is at LEAST a month off (probably longer). Git master isn't supposed to be end-user friendly and go-for-production right now. If you need something stable, use a release!

(I'm not necessarily addressing you specifically, just the general "omg it's crazy broken" sentiment I've seen lately)

I'm not worried that it won't be added now. I was just concerned that such a simple upgrade path didn't appear to have been considered, because statements like that would have usually added a mention that those commandline options will not be needed by release in order to avoid a full re-download.

Luke-Jr

legendary

Activity: 2576

Merit: 1186

Quote from: Maged on October 21, 2012, 10:02:28 PM

Quote from: jgarzik on October 20, 2012, 09:35:13 PM

Quote from: Syke on October 20, 2012, 09:25:44 PM

Quote from: Pieter Wuille on October 20, 2012, 05:37:51 PM

I've just merged my "ultraprune" branch into mainline (including Mike's LevelDB work).

Does this require downloading and re-processing the blockchain from the beginning?

Yes. However, to save downloading, you may provide

Code:

-loadblock=DATA_DIR/blk0001.dat -loadblock=DATA_DIR/blk0002.dat

to import the old data files into the new bitcoin database backend (ultraprune/leveldb).

* "DATA_DIR" should be replaced with the directory where your blockchain was stored in <= 0.7.1.

It doesn't do this automatically? It really should before this is officially released. You could even just add them as implied bootstrap.dat files.

I'm sure 0.8 will do this (and more!). Seriously, we just released 0.7.1 (from master)... 0.8 is at LEAST a month off (probably longer). Git master isn't supposed to be end-user friendly and go-for-production right now. If you need something stable, use a release!

(I'm not necessarily addressing you specifically, just the general "omg it's crazy broken" sentiment I've seen lately)

Maged

legendary

Activity: 1204

Merit: 1015

Quote from: jgarzik on October 20, 2012, 09:35:13 PM

Quote from: Syke on October 20, 2012, 09:25:44 PM

Quote from: Pieter Wuille on October 20, 2012, 05:37:51 PM

I've just merged my "ultraprune" branch into mainline (including Mike's LevelDB work).

Does this require downloading and re-processing the blockchain from the beginning?

Yes. However, to save downloading, you may provide

Code:

-loadblock=DATA_DIR/blk0001.dat -loadblock=DATA_DIR/blk0002.dat

to import the old data files into the new bitcoin database backend (ultraprune/leveldb).

* "DATA_DIR" should be replaced with the directory where your blockchain was stored in <= 0.7.1.

It doesn't do this automatically? It really should before this is officially released. You could even just add them as implied bootstrap.dat files.

gmaxwell

staff

Activity: 4284

Merit: 8808

Quote from: marcus_of_augustus on October 21, 2012, 09:05:49 PM

What will be the new dependency(s) for a typical Linux box?
Will BDB still remain a dependency also?

No new dependencies right now. BDB is still used for the wallets and will remain a dependency for the foreseeable future even after it moves off it for the wallets just for import if nothing else.

marcus_of_augustus

legendary

Activity: 3920

Merit: 2349

Eadem mutata resurgo

Quote from: Pieter Wuille on October 20, 2012, 05:37:51 PM

(copy of mailinglist post)

I've just merged my "ultraprune" branch into mainline (including Mike's LevelDB work). This is a very significant change, and all testing is certainly welcome. As a result of this, many pull requests probably don't apply cleanly anymore. If you need help rebasing them on the new structure, ask me.

The idea behind ultraprune is to use an ultra-pruned copy (only unspent transaction outputs in a custom compact format) of the block chain for validation (as opposed to a transaction index into the block chain). It still keeps all blocks around for serving them to other nodes, for rescanning, and for reorganisations. As such, it is still a full node. So, despite the name, it does not implement any actual pruning yet, though pruning would be trivial to implement now. This would have profound effects on the network though, so may still need some discussion first.

A small summary of the changes:

Instead of blk000?.dat, we have blocks/blk000??.dat files of max 128 MiB, pre-allocated per 16 MiB
Instead of a Berklely DB blkindex.dat, we have a LevelDB directory blktree/. This only contains a block index, no transaction index.
A new LevelDB directory coins/, which contains data about the current unspent transaction output set.
New files blocks/rev000??.dat contain undo data for blocks (necessary for reorganisation).
More information is kept about blocks and block files, to facilitate pruning in the future, and to prepare for a headers-first mode.
Two new RPC calls are added: gettxout and gettxoutsetinfo.

The most noticeable change should be performance: LevelDB deals much better with slow I/O than BDB does, and the working set size for validation is an order of magnitude smaller. In the longer run, I think it is an evolution towards separation between validation nodes
and archive nodes, which is needed in my opinion.

What will be the new dependency(s) for a typical Linux box?

Will BDB still remain a dependency also?

Atlas

jr. member

Activity: 56

Merit: 1

gmaxwell: jgarzik: Atlas works at odds with his stated goals, he makes me want to centeralized bitcoin just so we can write him out of it. Tongue

This is what he said in response to this thread.

Atlas

jr. member

Activity: 56

Merit: 1

Quote from: ?? on ??

Quote from: Atlas on October 21, 2012, 07:20:44 PM

The frequent releases of the Bitcoin devs play a very small part especially when most users use custom GUIs or a web client to conduct Bitcoin business.

Right, sort of how companies like Cisco, Intel, Apple, Microsoft, and Google, and guys like Linus Torvalds, have little to do with people's ability to read their e-mail, and they owe 98% of the credit to the guys who invented the binary code and the transistor.

Why should we owe anyone anything? The debt has been paid inherently when they released their work. Why should they keep endless authority?

All I am calling for is no authority over the Bitcoin development culture. I am not questioning people doing things. However, I do question the "I am special, worship me, need me, depend on me." mentality.

Atlas

jr. member

Activity: 56

Merit: 1

Quote from: ?? on ??

Quote from: Atlas on October 21, 2012, 07:01:53 PM

The momentum I am referring to is the userbase that continues to grow which is not fully due to the post-2010 work of Bitcoin-Qt/Bitcoind.

I think it would be very hard to argue that it was because of the efforts of Mr. "Bitcoin is just a experiment, don't bet the house." that Bitcoin is trading over $12 a coin.

Satoshi Nakamoto made 98% of what the core Bitcoin protocol is today. That was the Big Bang. Every other software development is a secondary footnote. The real success is in user support of Satoshi's core product.

I recognize people want to be important, they want people to depend on them for Bitcoin to work but that's not how free societies work. We don't indebt ourselves to a supposed leader. We shouldn't have to owe Bitcoin's success to a certain class of people. We all stand on equal ground and stand independently. Bitcoin as a decentralized currency should represent that.

Satoshi also made 98% of SolidCoin, NameCoin, I0coin, IXCoin, and every other coin out there that is presently deep in the toilet.

Bitcoin trades at over $12 a coin because of the collective actions and frequent releases of the Bitcoin devs, entities like MtGox/BitInstant/BitPay, utilities like BlockChain.info, the many businesses who accept Bitcoin as payment, promoters like Bitcoin Magazine and WeUseCoins.com (not to mention my own coins), and like it or not, things like the Silk Road. When you say "Hey Gregory, you didn't build that!", that's being a douche and has nothing to do with how much money you have in Bitcoin - it's just plain and simple being a douche.

The frequent releases of the Bitcoin devs play a very small part especially when most users use custom GUIs or a web client to conduct Bitcoin business.

I am not saying other people haven't built anything. I am saying the Bitcoin.org dev team hasn't built everything and that we owe them nothing. They are doing this out of their own interest and their own time, especially when they want to force certain things down our throat.

All I am calling for is no recognition of significant power in this community: Bitcoin.org implying they have great say and clout in this offends that ideal. They imply it when they want to push radical changes. This change may not be too radical but I am waiting for the day a radical push comes.

Also, I have no shame in being a douche. I am just out for the long-time survival of Bitcoin. I have no agenda besides keeping things as they are: Working.

Anyways, I agree, we all built Bitcoin. As for the Bitcoin.org developers, no they don't have exclusive ownership over it.

TLDR: Nobody should be dependent on the work of anybody in the Bitcoin community.

Atlas

jr. member

Activity: 56

Merit: 1

Quote from: ?? on ??

Quote from: Atlas on October 21, 2012, 06:37:34 PM

Bitcoin has momentum that is very scarce. If destroyed, cryptocurrency might be doomed for another few decades.

Quote from: Atlas on October 21, 2012, 06:50:14 PM

The fact is Bitcoin would live on without his support. If a feature is truly needed, it will be done anyway by another party or two.

Which is it? How many fingers can you chop off your hand before it ceases to be a hand?

The momentum I am referring to is the userbase that continues to grow which is not fully due to the post-2010 work of Bitcoin-Qt/Bitcoind.

I think it would be very hard to argue that it was because of the efforts of Mr. "Bitcoin is just a experiment, don't bet the house." that Bitcoin is trading over $12 a coin.

Satoshi Nakamoto made 98% of what the core Bitcoin protocol is today. That was the Big Bang. Every other software development is a secondary footnote. The real success is in user support of Satoshi's core product.

I recognize people want to be important, they want people to depend on them for Bitcoin to work but that's not how free societies work. We don't indebt ourselves to a supposed leader. We shouldn't have to owe Bitcoin's success to a certain class of people. We all stand on equal ground and stand independently. Bitcoin as a decentralized currency should represent that.

Atlas

jr. member

Activity: 56

Merit: 1

Quote from: casascius on October 21, 2012, 06:42:38 PM

Quote from: gmaxwell on October 21, 2012, 06:14:07 PM

it's not generally considered polite to demand what people do with their time if you're not paying them.

The way I see it, by downloading the software, they are paying, just not with money, the same way they "pay" Google by seeing ads. These people are offering Bitcoin (the idea) a share of their mind and their time. They are incrementally adding to the entire Bitcoin-based economy by considering accepting Bitcoins in exchange for their goods and services. They are paying with their willingness to change how they see a BTC as something worth 0.00 to them, to something worth over $10. Of course, I am also respectfully keeping in mind that you have contributed substantially to bitcoind, and I have not.

The fact is Bitcoin would live on without his support. If a feature is truly needed, it will be done anyway by another party or two. There is enough capital and value here to sustain Bitcoin on a as-it-goes basis. Bitcoin as a protocol is sustaining just fine after all. There are no major vulnerabilities. Client innovation is being advanced by various people. The Satoshi client will likely become a relic.

Bitcoin could not live without the users putting their money and time into this. It's the user that is God in the Bitcoin ecosystem.

This isn't Ubuntu Linux or another hippy open-source software project. This is money. Real money. It's beyond the leagues of anything else in the development world.

Yes, I will continue to be a douchebag because I have too much money in this thing.

Topic: Ultraprune merged in mainline - page 2. (Read 25403 times)