Armory - Discussion Thread - page 104.

maaku

legendary

Activity: 905

Merit: 1011

Code:

# undo the mess you got yourself into
git checkout master
git branch -d ramreduceleveldb

Code:

# checkout the branch
git checkout -b ramreduceleveldb origin/ramreduceleveldb

Ente

legendary

Activity: 2126

Merit: 1001

Quote from: etotheipi on September 09, 2013, 12:16:54 AM

Anyone in Linux want to help? It's the ramreduceleveldb branch.

WTF.
I'm late to the party, and now I'm spilling drinks over everybody too:
I tried to switch to the ramreduceleveldb branch, but am too stupid to do so.

Quote

git checkout ramreduceleveldb
git pull origin ramreduceleveldb

This seems to create a local branch, with the data of some other branch, probably "master" or "testing".

Quote

git branch -a
master
* ramreduceleveldb
remotes/origin/HEAD -> origin/master
remotes/origin/armoryd
remotes/origin/backupcenter
remotes/origin/bitcoind0.8andP2Pool
remotes/origin/bitsafetest
remotes/origin/blkchainagain_optimized
remotes/origin/cmake
remotes/origin/cryptoppstatic
remotes/origin/dev
remotes/origin/ecdsacalc
remotes/origin/leveldb
remotes/origin/managesatoshi
remotes/origin/master
remotes/origin/migratewallet
remotes/origin/newwallet
remotes/origin/osx_managesat
remotes/origin/pointcompress
remotes/origin/python_test
remotes/origin/qtdev
remotes/origin/ramreduceleveldb
remotes/origin/rrldbtest
remotes/origin/sddblspend
remotes/origin/testing
remotes/origin/threading

Debian Wheezy here. Probably of no big help anyway, now?

Ente

etotheipi

legendary

Activity: 1428

Merit: 1093

Core Armory Developer

BACKUP CENTER MERGED:

I have merged the new backup center into the ramreduceleveldb branch. If you went to the Bitcoin conference and you saw the fragmented-backup and SecurePrint demos, then you already know what's there. Please test it! (p.s. new backups now only have two lines of data to copy/print!). There's still a few quirks that need to be fixed, but it's just about fully functional.

At the moment, it looks like Armory is "usable" in Linux. The biggest problems remaining are:

(1) Lack of interface feedback on initial blockchain build. Just wait 4-5 hours, don't worry if it looks like it's not doing anything.
(2) Slowness and a screen full of harmless errors and warnings on load -- if you shut down Armory and start it later, it can take a couple minutes to sync with the latest blocks. Will have to investigate, since it clearly builds the database much faster than that on an initial sync.
(3) Restored wallets appear to have incorrect balances until you restart. Luckily, restarting only takes about about 5 sec now

I will track down some of the slowness issues later, since it technically works right now. At the moment, have a long list of tweaks that have been waiting a few months to go in, and a few quirks to fix in the backup center. Please keep testing! (if you can)

etotheipi

legendary

Activity: 1428

Merit: 1093

Core Armory Developer

Quote from: goatpig on September 10, 2013, 12:33:32 PM

leveldbwin is a google project for msvc 9 and 10 that allows you to complile leveldb as a dll on windows

https://code.google.com/p/leveldbwin/downloads/detail?name=leveldb_1_20_win32_src.zip&can=2&q=

To compile, it requires ATL librairies, which are available in WDK 7.1 or pay versions of visual studio 9 and above.

Once you get the DLL to compile the integration is straight forward.

I personally use vc11 express so I have to get the WDK before I can compile the project and test it. I have however successfully compiled this project a few months ago with vs9.

*edit*

After installing WDK, the project successfully compiled, yielding a static library for snappy and both static and dynamic libraries compile options for leveldb.

Awesome! Finally some encouraging news on this front. I don't know how I missed the leveldbwin project...

goatpig

legendary

Activity: 3738

Merit: 1360

Armory Developer

leveldbwin is a google project for msvc 9 and 10 that allows you to complile leveldb as a dll on windows

https://code.google.com/p/leveldbwin/downloads/detail?name=leveldb_1_20_win32_src.zip&can=2&q=

To compile, it requires ATL librairies, which are available in WDK 7.1 or pay versions of visual studio 9 and above.

Once you get the DLL to compile the integration is straight forward.

I personally use vc11 express so I have to get the WDK before I can compile the project and test it. I have however successfully compiled this project a few months ago with vs9.

*edit*

After installing WDK, the project successfully compiled, yielding a static library for snappy and both static and dynamic libraries compile options for leveldb.

etotheipi

legendary

Activity: 1428

Merit: 1093

Core Armory Developer

Quote from: goatpig on September 10, 2013, 12:21:41 PM

Quote from: etotheipi on September 10, 2013, 01:27:12 AM

Getting closer! At least I think I have all the issues contained. Except for that battle with leveldb+windows. Btw, I'm still offering 2 BTC to anyone who helps me get leveldb built in Windows and integrated into my project. It is remarkably unpleasant.

Which compiler are you using?

https://bitcointalksearch.org/topic/m.3095869

goatpig

legendary

Activity: 3738

Merit: 1360

Armory Developer

Quote from: etotheipi on September 10, 2013, 01:27:12 AM

Getting closer! At least I think I have all the issues contained. Except for that battle with leveldb+windows. Btw, I'm still offering 2 BTC to anyone who helps me get leveldb built in Windows and integrated into my project. It is remarkably unpleasant.

Which compiler are you using?

payb.tc

hero member

Activity: 812

Merit: 1000

Quote from: etotheipi on September 07, 2013, 06:02:43 PM

Quote from: chrisrico on September 07, 2013, 05:41:42 PM

Quote from: etotheipi on September 07, 2013, 03:52:43 PM

I'm not 100% convinced of that. The issue is not slowness (well, it is an issue), but that these systems may be running out of RAM. For instance, it just can't be run on 32-bit because of the lack of address space. But if you virtualize a system that runs 64-bit OS with 16 GB of RAM (using 14 GB of disk), it may be possible to actually run it even though it takes 3 hours to sync.

I'd be interested to see someone try it, though I would bet 2:1 that it still doesn't work at all. But I can see why it might work.

Can you run a 64 bit VM on 32 bit hardware? That doesn't seem right.

Anyway, the original question was whether setting up a VM would help alleviate the problem of only have 2GB of memory, as the VM could "have" 64GB. The answer is of course no.

From stackoverflow:

Quote

You can't run a 64-bit VM session on a 32-bit processor. However, you can run a 64-bit VM session if you have a 64-bit processor but have installed a 32-bit host OS and your processor supports the right extensions.

It's my understanding that every desktop/laptop CPU under the sun is 64-bit by now, even though many people still run 32-bit OS. So it is most likely possible to do this.

Yes, slow as dirt. Painfully slow. But it might actually, eventually work, instead of seg-faulting when it hits 94%.

agreed, this is my assumption

i'll post here once i try it out on an unused laptop with lots of hdd space and barely any ram

and i'll pull out a calendar to time how long it takes to launch

halfawake

hero member

Activity: 490

Merit: 500

Quote from: etotheipi on September 09, 2013, 03:49:29 PM

Quote from: superbit on September 09, 2013, 03:42:50 PM

When I loaded armory this AM, it said it detected a corrupt block in the blockchain.

I closed armory, opened bitcoin 0.8.4 and it offered to repair the chain. I let this run through and it completed giving me the green checkmark in the bottom right corner. Then as soon as I went back into armory I got the same error, now I go back into the bitcoin client and it is starting the repair again??

I really need to send coins in my armory wallet out today, what can I do to end this loop?

Unfortunately that is actually a bitcoin-qt problem that cropped up today. From the mailing list: prepare for some turbulence.

Crap, so reindexing it doesn't fix it? Guess I've got to change the checklevel so it doesn't keep doing this tomorrow. Definitely seems like a bug, though obviously not in Armory.

etotheipi

legendary

Activity: 1428

Merit: 1093

Core Armory Developer

Fixed the wallet balance issue. What a strange bug! If you saw strange balances, it should be resolved when you pull, recompile and run. I finally get all the right balances on all of my wallets, and I have some pretty active wallets! (stretching back 2 years and hundreds of tx).

It appears to work, strictly speaking. A couple things to note:

(1) There's clearly a very large inefficiency in startup. I already noticed it rewrites the entire headers DB on each load -- which is actually not that slow, but it's also unnecessary. Similarly, it reapplies a bunch of block on each load even though it isn't supposed to. Neither of these cause inaccuracies, but they do slow it down dramatically and represent issues to be to be fixed.
(2) It appears that the new backup stuff was merged before the latest version. I might merge the latest backup stuff ASAP so that people can test it, then I'll work on fixing #1

Getting closer! At least I think I have all the issues contained. Except for that battle with leveldb+windows. Btw, I'm still offering 2 BTC to anyone who helps me get leveldb built in Windows and integrated into my project. It is remarkably unpleasant.

etotheipi

legendary

Activity: 1428

Merit: 1093

Core Armory Developer

Quick update:

With snappy, my last rebuild took 4.5 hours after deleting the .armory/databases directory. There's a lot of room for improvement on that one, but a lot of the options are kind of heavy changes, and it's not critical for stability. I'll try now it without snappy and see if it improves.
Displayed balances are really incorrect when used on mainnet. Perhaps it has to do with length of history. But luckily, it's an "aesthetic" error -- I just confirmed that the raw DB entries are 100% correct, the interface is just reading the DB incorrectly.
You can expect a whole bunch of errors and warnings when you restart, because my code accidentally rescans the last few blocks and then triggers those log entries because things that it's expecting to be unspent are already marked spent, etc.

And most importantly:

Armory uses 288 MB of RAM after the DB build is complete (on Linux). That's means that Armory online mode should be usable on even a lot of older hardware!
There's a compile-time configurable parameter to adjust how much RAM is used when it's building the DB. Right now, it's set so that it won't use more than 1 GB of RAM while building. I will make that run-time configurable and do some testing on it RAM-vs-performance.

etotheipi

legendary

Activity: 1428

Merit: 1093

Core Armory Developer

Quote from: laurentb on September 09, 2013, 06:18:28 PM

Quote from: gmaxwell on September 09, 2013, 06:11:10 PM

Why are you using snappy? It hurt performance and increased size when it was tested with Bitcoin.

Oh, good to know!
Does that mean it has to be disabled at compile time though? Can't bitcoin just ask not to use it at runtime?

The documentation suggests that most applications will benefit from it, and that it's very lightweight and much faster than most file I/O operations so it shouldn't hurt you in terms of performance. I guess that's why it's enabled by default. But Bitcoin is special in that the blockchain data is very dense and hardly compressible. I had suspected it would be of no benefit here and thus no reason to use it (and meant the baseline to disabled it). I didn't realize it would actually hurt you! Yeah, good to know...

laurentb

newbie

Activity: 5

Merit: 0

Quote from: gmaxwell on September 09, 2013, 06:11:10 PM

Why are you using snappy? It hurt performance and increased size when it was tested with Bitcoin.

Oh, good to know!
Does that mean it has to be disabled at compile time though? Can't bitcoin just ask not to use it at runtime?

etotheipi

legendary

Activity: 1428

Merit: 1093

Core Armory Developer

Quote from: gmaxwell on September 09, 2013, 06:11:10 PM

Quote from: etotheipi on September 09, 2013, 04:59:43 PM

Did you recompile? Also, did you install the libleveldb package?

You are really going to use a system database library for Bitcoin software? I wish you luck!

Why are you using snappy? It hurt performance and increased size when it was tested with Bitcoin.

Oh, duh. I'm not! (or rather, I didn't mean to) I committed leveldb as part of the project for that reason, and it should be part of the (poorly made) Makefile. Perhaps I am building leveldb but actually linking to the system libraries by accident. I'll check.

Also, I thought I had disabled snappy. But perhaps I didn't. I assumed the linker error could happen even when you weren't using it.

EDIT: Okay I misread the documentation, I didn't realize it uses snappy by default. I thought I had to tell it to use it. I guess I never got around to testing with and without. I'll disable it and see if I get a performance improvement.

gmaxwell

staff

Activity: 4242

Merit: 8672

Quote from: etotheipi on September 09, 2013, 04:59:43 PM

Did you recompile? Also, did you install the libleveldb package?

You are really going to use a system database library for Bitcoin software? I wish you luck!

Why are you using snappy? It hurt performance and increased size when it was tested with Bitcoin.

justusranvier

legendary

Activity: 1400

Merit: 1013

Leveldb is compiled without snappy support.

Here's where it gets complicated:

Luke-jr maintains the Bitcoin packages on Gentoo. In accordance with their policies, he's not using the bundled leveldb libraries included with the bitcoin-qt source, so instead forces Portage to build leveldb independently, but identically to the bundled version.

This enforces USE="-snappy" for leveldb.

I can tell the Armory ebuild to require leveldb build with USE="+snappy", but that creates a package conflict since Bitcoin-Qt and Armory now have incompatible dependency requirements.

The upshot is that it will be impossible to install Bitcoin-Qt and Armory at the same time on a Gentoo system. That doesn't affect me personally because I use bitcoind anyway - I just uninstalled bitcoin-qt to resolve the block, but that's not a general solution.

laurentb

newbie

Activity: 5

Merit: 0

Quote from: etotheipi on September 09, 2013, 05:26:04 PM

However, it's also possible that when Gentoo installs leveldb it does not install the snappy compression library which is needed for leveldb.

The Gentoo Bitcoin ebuilds force you to install leveldb with snappy disabled.
I haven't been able to gather a real reason why.

Is it possible to compile leveldb locally and embed it? Maybe it could be used as a temporary workaround.

justusranvier

legendary

Activity: 1400

Merit: 1013

Quote from: etotheipi on September 09, 2013, 05:26:04 PM

Quote from: justusranvier on September 09, 2013, 05:06:33 PM

Quote from: etotheipi on September 09, 2013, 04:59:43 PM

Did you recompile?

This is Gentoo, so there is no installation method that's not recompiling.

That's not strictly true, here, since i didn't giveyou an installer. If you have previously compiled the project and then checkout the new branch and/or pull, it will update the code, but you might have to do a "make clean; make" from the base directory to get it to discard the previous binaries and recompile all of them. You may know this, but I've frequently forgotten to recompile after pulling an update and been confused by the errors that look like what you just showed.

I'm using an ebuild, so what happens with Gentoo is that when you install a package it extracts the source code to a working directory in /var/tmp/portage, which then gets deleted after installation. There are never any intermediate files left lying around - each compilation takes place in a clean environment.

Quote from: etotheipi on September 09, 2013, 05:26:04 PM

However, it's also possible that when Gentoo installs leveldb it does not install the snappy compression library which is needed for leveldb. While trying to get leveldb working I've seen that topic come up a few times even though I didn't have the problem myself on Ubuntu. But it might be an issue on other system.

I'll check into that.

etotheipi

legendary

Activity: 1428

Merit: 1093

Core Armory Developer

Quote from: justusranvier on September 09, 2013, 05:06:33 PM

Quote from: etotheipi on September 09, 2013, 04:59:43 PM

Did you recompile?

This is Gentoo, so there is no installation method that's not recompiling.

That's not strictly true, here, since i didn't giveyou an installer. If you have previously compiled the project and then checkout the new branch and/or pull, it will update the code, but you might have to do a "make clean; make" from the base directory to get it to discard the previous binaries and recompile all of them. You may know this, but I've frequently forgotten to recompile after pulling an update and been confused by the errors that look like what you just showed.

However, it's also possible that when Gentoo installs leveldb it does not install the snappy compression library which is needed for leveldb. While trying to get leveldb working I've seen that topic come up a few times even though I didn't have the problem myself on Ubuntu. But it might be an issue on other system.

justusranvier

legendary

Activity: 1400

Merit: 1013

Quote from: etotheipi on September 09, 2013, 04:59:43 PM

Did you recompile?

This is Gentoo, so there is no installation method that's not recompiling.

Quote from: etotheipi on September 09, 2013, 04:59:43 PM

Also, did you install the libleveldb package?

I have leveldb-1.9.0 installed.

Topic: Armory - Discussion Thread - page 104. (Read 521763 times)