Pages:
Author

Topic: [FIXED] MacOS X LevelDB Corruption Bounty (10.00 BTC + 200.2 LTC) - page 4. (Read 83881 times)

legendary
Activity: 1022
Merit: 1007
Sooner or later, a man who wears two faces forgets
Well this is my unofficial explanation.

Due of the mac updates, it changed standard applications.
This happened to me when updating to Maverick

Due of the changes, when having installed VLC, it uses your VLC to open your wallet.dat
On the other side, you would have to unlock your basic documents when installing Bitcoin-QT to have a new transaction being sent.
As this changes your file, MAC OS doesn't allow you to change the files without you unlocking them.

You also need to change a 'hidden' file on your Mac, to allow you to connect from files to the internet as Mac brought some "security" updates.
legendary
Activity: 3766
Merit: 1364
Armory Developer
Are the instances of DB corruptions on Windows caused by the same problems occuring on the OSX builds?
member
Activity: 98
Merit: 10
nearly dead
If the Bitcoin code can be refactored, or is already factored, to support easily changing the database "Backend" then you can try HyperLevelDB or RocksDB instead, they are both based off LevelDB and with identical APIs. This can be a good solution to try if problems re-surface with LevelDB.

I'm not seeing the reasoning here. Are you saying that those other options fixed this bug and other ones but didn't contribute it back to leveldb ? I doubt that is the case.
newbie
Activity: 7
Merit: 0
Thanks, that's clear!  Wink
hero member
Activity: 547
Merit: 500
Decor in numeris
You know it is corrupted by the client complaining that it is so.

And no, you do not lose coins.  The wallet is not affected (although you should always have a backup), the worst that could potentially happen is that your client "fortgets" you have the coins, or think you have too many (forgets you have spent some).  But rebuilding the database will fix that, since your bitcoins are not stored on your computer but in the blockchain.
newbie
Activity: 7
Merit: 0
Noob here, how do I know if my "leveldb" is corrupted?

I have both Bitcoin client and Litecoin client running for approx. 10 days now (with some short breaks / macbook sleep time).

Both my wallets are showing the correct value.

2nd question: are mac os x users risking losing their coins by running the current clients?

Thanks :-)
legendary
Activity: 1064
Merit: 1001
If the Bitcoin code can be refactored, or is already factored, to support easily changing the database "Backend" then you can try HyperLevelDB or RocksDB instead, they are both based off LevelDB and with identical APIs. This can be a good solution to try if problems re-surface with LevelDB.
newbie
Activity: 9
Merit: 0
Thanks Mike. Though I must point out that my summation above is a theory, and certainly shouldn't be taken as fact. But the premise is basically this: We can't really know if the GCC idiom is supported on Apple's compiler-of-the-week, but it seems we can be sure that OSMemoryBarrier will work as intended, so there should be no harm in switching to it.
legendary
Activity: 1526
Merit: 1134
Wow. That's bad and would indeed explain the issues (or at least some of them). Apple really should not be claiming in the headers to be GCC compatible and then ignoring the GCC memory barrier idiom. That's not compatible at all. So technically this is an Apple bug, I'd say, but it's one we can work around pretty easily.

Very exciting that this has been discovered, great work cfields, worthy of a bounty award for sure. It's hard to imagine this not causing corruption. And it would also explain the otherwise unexplainable "self-fixing corruption" problem.
newbie
Activity: 9
Merit: 0
Upon further investigation, this does indeed seem to be unintended in leveldb.

Their atomics are borrowed from here (see the note in port/atomic_pointer.h): https://code.google.com/p/gperftools/source/browse/src/base/atomicops.h#102

It's clear there that mach/apple takes precedence over x86. The patch above fixes leveldb to match that behavior. Regardless of whether or not this fixes the db corruption for bitcoin, I'll take it upstream.
sr. member
Activity: 263
Merit: 250
Updated: December 11h, 2013
The following versions contain cfields' memory barrier patch and phantomcircuit's no-mmap patch.  Extensive testing indicates no new corruption happens on MacOS X.

Official Bitcoin 0.8.6
http://bitcoin.org/en/download

Bitcoin 0.8.5-OMG10
http://download1.rpmfusion.org/~warren/bitcoin-0.8.5-OMG10/macosx/
https://github.com/litecoin-project/bitcoinomg/commits/0.8.5-OMG10
This build contains all the same bug fixes as 0.8.6 above.  Includes additional features like Coin Control.

Litecoin 0.8.6.1
https://download.litecoin.org/litecoin-0.8.6.1/
https://litecoin.info/Upgrading_Litecoin#Upgrading_to_0.8.6.1
Please follow these directions to use Litecoin 0.8.6.1.

https://github.com/bitcoin/bitcoin/issues/2770
Read more here.
member
Activity: 98
Merit: 10
nearly dead
In the current code, OSX falls into this this case:
Code:
#elif defined(ARCH_CPU_X86_FAMILY) && defined(__GNUC__)
(Note that clang also defines __GNUC__)

I believe that's unintentional. Moving the "#elif defined(OS_MACOSX)" up causes the barrier to be
Code:
OSMemoryBarrier();
rather than
Code:
asm volatile("" : : : "memory");

Oh, alright, that makes sense. You should bring it up to the leveldb people.
newbie
Activity: 9
Merit: 0
In the current code, OSX falls into this this case:
Code:
#elif defined(ARCH_CPU_X86_FAMILY) && defined(__GNUC__)
(Note that clang also defines __GNUC__)

I believe that's unintentional. Moving the "#elif defined(OS_MACOSX)" up causes the barrier to be
Code:
OSMemoryBarrier();
rather than
Code:
asm volatile("" : : : "memory");
member
Activity: 98
Merit: 10
nearly dead
I have a theory, and a binary to try out for someone who can reproduce. I don't see the issue on my macbook.

Patch is here: https://github.com/theuni/bitcoin/commit/ac0d918bb49e0f8393b1e27662bc4c25113980b2


Care to explain that your patch isn't just about moving a piece of code 20 lines above where it was previously ? I might be too tired, but that patch does nothing that wasn't being done already.
newbie
Activity: 9
Merit: 0
I have a theory, and a binary to try out for someone who can reproduce. I don't see the issue on my macbook.

Patch is here: https://github.com/theuni/bitcoin/commit/ac0d918bb49e0f8393b1e27662bc4c25113980b2
Binary is here for anyone who trusts me (you can, but of course you shouldn't) https://www.dropbox.com/s/r959wns6rm4xjsu/Bitcoin-Qt.dmg

Theory goes like this:

Currently leveldb's atomics use a gcc-style compiler barrier like:
asm volatile("" : : : "memory");

OSX has a native memory barrier that can be used instead. It's my understanding that on x86/x86_64, a real memory barrier is not needed, but only a flag to tell the compiler not to optimize+reorder store/loads. Looks to me like it's a mistake, and that the OSX function should be chosen instead.

In my tests, the asm hack was often completely optimized away, and loads/stores were reordered anyway. So I believe the barrier in current code is effectively a nop.

Since this is a build-time optim and not run-time, it does not explain why 10.8 is affected while 10.6 is. However, if atomics are busted, timings in other underlying functions (write/fsync/mmap/etc) would have a huge effect, so it seems reasonable that this could be the indirect cause.
legendary
Activity: 1221
Merit: 1025
e-ducat.fr
BitcoinQt 0.8.5 works like a charm on OSX 10.7.5 but crashes (with database corrupted message) on OSX 10.8.5 (brand new machine, no filevault involved nor any funky settings).
full member
Activity: 151
Merit: 100
Corrupted wallet.dat is not the issue of this particular bounty.

ok nm. than sorry I misread and though my issue was related.
staff
Activity: 4284
Merit: 8808
Wow! Theymos, thats awesome!
administrator
Activity: 5222
Merit: 13032
The Bitcoin Forum pledges 4 BTC to this bounty, and we'll probably also put up an ad about this in forum ad round 103. This is more than I really want to spend on non-forum development issues from the forum's treasury, especially when the forum software situation is non-optimal, but this issue seems to be becoming serious despite the efforts of many experts over a long period of time, so I think that increasing the bounty amount is important.
sr. member
Activity: 408
Merit: 261
Thank you to Gavin and Warren and everyone for your efforts to help resolve this issue.  I think this bounty is a great idea and I'm very happy to see the Mac client receive some 1st class attention.

2013 has probably been bitcoin's most exciting year, but it's been a rough one for some Mac bitcoin-qt users.  It was way back on December 14, 2012 when the last version of bitcoin-qt (v0.7.2 with the later May 15 workaround) to work reliably on my MacBook Pro was released.  Every one of the many v0.8.x releases since has corrupted for me.  I've grown weary of reindexing the blockchain and testing out new builds.

Some of my experiences and setup are detailed in the original github link:
https://github.com/bitcoin/bitcoin/issues/2770

I'm sure that many new Mac users who have gotten into bitcoin this year have not had the same bullet-proof experience that I enjoyed when I first got started with the earlier versions.  It can be very frustrating, after all the complexity inherent to bitcoin, to have something not function the way it should when you're first getting started.  All financial software needs to be rock solid, especially with the sort of values that are being tossed around these days with our experimental little currency.

So with that in mind, I pleased to support this very worthwhile effort with a contribution of 0.48552571 BTC, so you can now update your bounty fund to an even 6.0 BTC + 200.2 LTC:

https://blockchain.info/tx/dc17a4e4b9ae40fdec9699936f965b57f0fe3c6fce8722033c56c30a11b01900

May bitcoin-qt have a long and illustrious future running on the world's fruitiest operating system, and may this bug meet a swift and permanent demise!!! 
Pages:
Jump to: