Bitcoin-Qt blockchain getting indexed by spotlight on the Mac

Carlton Banks

legendary

Activity: 3430

Merit: 3071

Quote from: grue on March 18, 2013, 11:13:53 PM

Quote from: smooth on March 18, 2013, 10:52:48 PM

Oh, the big blk....dat files at the top level of Bitcoin are now unused?

i'm not quite sure about that. If you upgraded to 0.8, then the .blk files may be still used. Personally, I did a fresh blockchain download using 0.8, only keeping wallet.dat, and that leaves no blk files.

Not sure about on Mac, but the Linux version appears to create a levelDB index file, but keeps the old data structure (blk00x.dat). It then records all new blocks from there on using the levelDB scheme (138Mb data files).

I myself deleted the blk00x.dat files from the BDB days and re-synchonised the whole blockchain into the levelDB/0.8.x data layout, it saved me nearly 500 Mb disk space.

grue

legendary

Activity: 2058

Merit: 1431

Quote from: smooth on March 18, 2013, 10:52:48 PM

Oh, the big blk....dat files at the top level of Bitcoin are now unused?

i'm not quite sure about that. If you upgraded to 0.8, then the .blk files may be still used. Personally, I did a fresh blockchain download using 0.8, only keeping wallet.dat, and that leaves no blk files.

smooth

legendary

Activity: 2968

Merit: 1198

Oh, the big blk....dat files at the top level of Bitcoin are now unused?

Still, expecting users to exclude three separate folders from Time Machine is a bit much. Most won't even know about it and will just have their backups blow up, or will possibly exclude the top level when this happens, no longer backing up their wallet.

How about putting the BCD in ~/Caches instead of Library? I think Time Machine excludes that by default, and this seems like logically the right approach to me anyway. (Unfortunately I think Spotlight still indexes Caches).

grue

legendary

Activity: 2058

Merit: 1431

Quote from: smooth on March 18, 2013, 09:58:42 PM

There is also an issue with Time Machine. The (huge) blockchain database (BCD) files are constantly being updated, which means they are copied and recopied to the backup disk every hour.

Excluding the Bitcoin foilder from Time Machine means the wallet won't be backed up either, which is bad.

There should be some kind of storage layout where the old blocks are in static files that don't change. Or at least the wallet should be moved out of the same folder as the BCD so the BCD can be excluded from backup without excluding the wallet.

This affects every system with an incremental backup mechanism, not just Mac, but it is a bigger issue on Mac because Time Machine is built into the OS and is widely used.

if you use 0.8, you will see that the non-important data are stored in /Bitcoin/blocks, /Bitcoin/blocks/chainstate, and /Bitcoin/database

smooth

legendary

Activity: 2968

Merit: 1198

There is also an issue with Time Machine. The (huge) blockchain database (BCD) files are constantly being updated, which means they are copied and recopied to the backup disk every hour.

Excluding the Bitcoin foilder from Time Machine means the wallet won't be backed up either, which is bad.

There should be some kind of storage layout where the old blocks are in static files that don't change. Or at least the wallet should be moved out of the same folder as the BCD so the BCD can be excluded from backup without excluding the wallet.

This affects every system with an incremental backup mechanism, not just Mac, but it is a bigger issue on Mac because Time Machine is built into the OS and is widely used.

smooth

legendary

Activity: 2968

Merit: 1198

Quote

It seems that there's basically no way for an app to mark a directory as excludable from Spotlight

Is there a way to mark files as being of a non-indexable type?

Quote

Did you try to add the directory to spotlight privacy tab in system preferences ?

Yes, as stated in the original post. That works fine.

BenTuras

hero member

Activity: 826

Merit: 1001

Did you try to add the directory to spotlight privacy tab in system preferences ?

Mike Hearn

legendary

Activity: 1526

Merit: 1129

Hm, well, that's dumb. It seems that there's basically no way for an app to mark a directory as excludable from Spotlight.

According to this blog, some directories are excluded by default:

http://mactechnotes.blogspot.ch/2005/10/spotlight-control.html

Unfortunately I don't think a non-root user could actually use any of them.

One other thing I saw mentioned is that Spotlight ignores any hidden directories (that start with a dot). So whilst it'd be annoying and weird, we could potentially move the LevelDBs into hidden directories on MacOS and that might make Spotlight ignore them. I think it'll still be woken up by the kernel, see the files are uninteresting and go back to sleep though.

smooth

legendary

Activity: 2968

Merit: 1198

I've noticed that with 0.8.0-beta on Snow Leopard, when updating the block chain, mds (the spotlight indexer) wakes up frequently and burns a lot of CPU (presumably indexing and reindexing the block chain database). Adding Library/Application Support/Bitcoin to Spotlight privacy fixes the problem. I don't know if there is some way for the client to indicate to spotlight that the files in there shouldn't be indexed.

I don't know about newer versions of Mac OS do the same thing, but I wouldn't be surprised if they do, since spotlight isn't changed much.

Topic: Bitcoin-Qt blockchain getting indexed by spotlight on the Mac (Read 2413 times)