Topic: Armory - Discussion Thread - page 100. (Read 521763 times)

2.2x the blockchain size is a lot of space especially when the blockchain is 50 GB.  But if you're going to go put 128 GB of RAM in your server, it'll be a lot cheaper to just buy a single 2 TB HDD for $80 which will sustain you for another year or two even at absolute maximum blockchain growth.

Also, the alternative is that you only hold 1x the blockchain.  Obviously, it's less than 2.2x, but it's the same order of magnitude, which means if 2.2x is enough for you to be concerned, 1x is still a lot. 

A workaround that doesn't involve any work for me:  make yourself a RAM disk and put all the data on there.  It should be fast as hell.  But of course, everything goes poof on a power-cycle. 

As I said, I'll think about how it might be done.  Especially if DB performance becomes a real issue.  But I might wait until that happens before I put too much effort into the RAM solution.

It's probably a seriously non-trivial thing to do, code wise.  Mostly the reason I was curious is that this kind of update would effectively double the hard drive demands of bitcoin since you'd have the 11.9 GB of the blockchain that bitcoin-qt manages, and the (presumably also) 11.9 GB of space that Armory would require from what you mentioned earlier.  24 GB is kind of a lot of space just to be able to keep Armory running happily, even in the era of hard drives with a lot of space.

Got it!  The error was misreported making me think it was a problem with python-twisted, but it was a problem with a the zope.interface module which is used by python-twisted.   The "zope" module doesn't have an __init__.py file and py2exe can't detect it to put it into the exe bundle.  The solution was to make an empty __init__.py in the C:\Python27\Libs\site-packages\zope directory after installing zope.  That's it!  Will need to remember to put that in the build instructions...

248 MB RAM usage on Windows.  Solid.  I guess it's still high for a "regular" desktop app, but it's nothing out of this world anymore



Importing and sweeping appears to be working correctly.  Wallet restore -- it works, but you have to restart Armory, or do it in offline mode then go online.  There's also some important updates to the backup system that needs to put in before anyone goes and uses it.  So all that is in the works tonight and tomorrow, then I'll create a standalone executable (signed) that people can use for pre-testing and funds recovery.  Should be in the next 48 hours.

Andy (currently my first and only team member) is helping setup a new Windows installer (using NSIS) that will Do The Right Thing when it installs Armory -- no more mysterious errors that require uninstalling and reinstalling Armory.   It will also have more installation options.  And it will look slick

cx_Freeze is an option.  But py2exe has been pretty darned reliable once I get it working.  Every Windows release I've made since Armory v0.5 has used py2exe and I'm happy with it.  And of course, it's already integrated into my build system.  If I'm desperate, I may investigate other freeze-app options, but I think I just need to figure out once and for all why py2exe is not behaving the same on all my build systems.

Would cx_Freeze help here?

EDIT: Also, pyinstaller might be an option.

The py2exe thing is not your problem.  Definitely mine.  And it's not related to the .dlls, I have all that under control.  It's that py2exe bundles up a working python project, and all its dependencies into a standalone directory with an .exe file.  I assume it embeds the python interpreter in there.  Technically the .exe works and links properly, but py2exe is not bundling some of the needed python libraries.  Armory successfully loads all the SWIG modules and all the related .dlls, but can't finish initializing because the py2exe never put the python-twisted libraries in the embedded environment and I get a python error that twisted is not found.

At the moment, I'm going to see if I can manually inject the twisted libraries into the output library.zip (which holds all the python dependencies).  If that works, it will give me an option for producing Windows .exe until I have time to fight py2exe.

I'm glad to hear the Windows build went through properly.

Regarding the speed issue and LevelDB: I suspect it's due to the implementation of atomic types and routines used in leveldbwin. That project is made for msvc 9 and 10, which don't support atomics and as such rely on full memory barriers and software mutexes, which are considerably slower than atomics. LevelDB is natively a C11 POSIX library.

I am going to try building the port folder on msvc11, with native atomics support. As I said before, I suspect compiling this part is all it'll take to get any version of LevelDB to build properly.

Regarding py2exe: I have no experience with Python, however Windows development is all about DLL compartmentalization, i.e. maintaining the armory core and the Python GUI part separately. I didn't get the impression that this is what SWIG does. I'd assume if there was a way to simply link the Python code to DLL binaries, it would considerably simplify the py2exe part, by limited its scope to the GUI code and an exporting header of sorts. Then again I may be totally out of my league and talking out of my a**.

I guess, for now you could just keep using the old version.  But going forward, it will be kind of tough to do what you just suggested.  However, there will be lighter-weight modes implemented that will use less RAM and HDD, though, it might have slightly watered-down functionality (not being able view arbitrary transactions or fees on some incoming tx).  I'll keep it in mind, though.  I'm sure there's lot of server applications where the user has more than enough RAM, and would prefer the performance.

At the moment, while there's only one version of Armory that everyone uses (no super-node, regular-node, lite-node split), RAM is much more scarce than HDD space.  But these kinds of things are great to keep in mind so I don't burn any code bridges if there's a chance they will be in high demand in the future (when I've got different modes).

I have a question about the upcoming RAM-reduction update.  Will there be any option for people to turn off the more hard-drive intensive version of Armory and keep the RAM-intensive version for people who have a lot of RAM?  I don't know if this would be a major pain to code or not, but it'd be a nice feature to have.

I forgot to mention that RAM usage on Windows looks to be about the same as Linux.  600-700 MB while the DB is building, then about 300 MB under normal operation.  In exchanges for the improved load time and RAM usage is 14 GB of disk space used for databases (which will be reduced in the future, but this was the easiest implementation for now).  So it's only a tad more than Bitcoin-Qt itself uses, which is about 11-12 GB currently (i.e. the storage efficiency of all that data in Armory's LevelDB database is only about 10-20% overhead).

Great! Thanks for the update. Waiting for this with much anticipation (OSX and Windows).

Updates:

Linux:  The app technically works, but it's got some serious usability bugs.  Importing, sweeping and restoring wallets causes the app to freeze until it finishes.  It also frequently freezes when new blocks come in or it checks for a new version.  Quite a few other quirks, too.  But all these issues are related to usability -- the core functionality seems to be in excellent shape!  The app successfully saves its entire state between loads, and only takes about 10-30 seconds to startup and pickup where it left off (longer if you haven't loaded it in a while and Bitcoin-Qt/bitcoind needs to catch up). 

Also, building and scanning (on all OS) is much slower than I was expecting.  Again, it's only a usability problem, and I am aware of multiple places that I missed optimizations that would improve all this.  I plan to dig into those when I get out a testing version. 

Windows:  The good news is that I got everything compiling and working in Windows using native python (thanks goatpig!).  I haven't gotten very far into Windows testing, but so far it appears to work the same as the Linux version though even slower (probably related to the ancient version of LevelDB that i hope goatpig will help me figure out how to upgrade for another 1 BTC ).

The bad news is that the process of creating a .exe file is failing, and may take a bit of work to fix.  py2exe has given me quite a bit of trouble in the past, and when I did get it working I had no idea what I did.  Therefore, it technically works on Windows, but only if you install python and like eight different python packages.  When I do finally get py2exe working, I can put out a pre-testing-testing version that should let people get access to their funds but not use it for regular use (it's going to be very rough around the edges). 

For those inclined to compile on Windows, I can tell you that part of the process has been restructuring everything.  All the old MSVS 2005/2008 projects have been destroyed, and rebuilt with MSVS 2012 Express.  This makes it free to compile (if you register MSVS w/ Microsoft), but it's still going to take a bit of tinkering.  If someone wants to try it, I can post a list of packages that need to be installed.

OSX:  If anyone wants to help with this, please let me know.  The new LevelDB stuff needs to be compiled & tested on the OSX side, and I have no idea how much of a PITA it is.  Perhaps Red Emerald is in good shape to try this out. 

I would guess it won't be in the next two months.  We have our plate pretty full, and we'd prefer to work out the higher priority stuff (like RAM reduction, new wallets, HW wallets) without dealing with the interface upgrades.  Well, and training our new employees

Ok. Any timeline of when that might get done? Working on some Armory video tutorials with someone to reduce the technical hurdle for people who want to use Armory. These will of course have to be updated as the interface changes. If I can get a very rough idea then I can kinda plan ahead.

This is in our plans.  One of the investors has a software company and is planning to contribute some manpower to redesigning it.  Right now it uses all the vanilla PyQt buttons and widgets, etc.  But it should be easy to subclass everything and make prettier versions of them (like buttons with images instead of raw text), which will make it look better.  And also we will be redoing some of the dialogs and interfaces.  It's just not high priority for me or my new developers, but it is part of our plan going forward as we try to implement a lite-version that is better tailored to new users.

Possibly. I never do that, I just link the include folder.

I selected "Use of ATL -- Yes".  Maybe it figured it out based on that?

That's odd, i had to link atl.lib to it. Regardless, as long as it's built, the rest is irrelevant.

It actually complied the leveldb.lib without linking the lib folder to the project.  I only included the header and it was happy.

Ok my bad the global include folder registering through msvc GUI seems to be limited to pay versions. Have to do it through the registry.