Topic: Armory - Discussion Thread - page 106. (Read 521829 times)

I'm not 100% convinced of that.  The issue is not slowness (well, it is an issue), but that these systems may be running out of RAM.  For instance, it just can't be run on 32-bit because of the lack of address space.  But if you virtualize a system that runs 64-bit OS with 16 GB of RAM (using 14 GB of disk), it may be possible to actually run it even though it takes 3 hours to sync. 

I'd be interested to see someone try it, though I would bet 2:1 that it still doesn't work at all.  But I can see why it might work.

When your computer needs more memory than it actually has, Windows uses a file on your hard drive as virtual memory (Unix uses a swap partition). As you state, it's incredibly slow. So no, using a VM doesn't benefit you at all.

I think this depends on your virtualisation software, certainly any version 8 or less VMWare software doesn't have a hypervisor swap-file scheme like this, but perhaps the newer releases do.

haven't tried it, but when you're just talking RAM, a virtual pc should be able to have 64 GB of ram, with the vm actually having it as a file somewhere, no?

i'm aware that it would be incredibly slow

Armory 0.88.1, 64bit, Win7, 16GB ram, i7 1st gen.
I don't know if this has been reported
I recently deleted my wallet and restored it.

Now Armory hangs at 98% - 15 seconds every day.
A close and restart gets it working again.

I don't have debug info.

I ran the installer and repaired the installation, fingers crossed, it has not shown any errors since.

That is actually the plan.  Or rather, when I talk about "super-node-lite-node split" that's what I'm talking about.  The lite version will be the same thing as the super version, but its calls to the DB engine will instead go over a socket to request it from another process or system.

It's a pretty big jump from where I currently am, but it's already part of my long-term plans

One thing you might consider doing with the large DB (aka server version?) is to allow the DB to be shared among several clients on a local network.  Ideally, you could have one version of the server version running on a host machine and several clients on remote machines utilizing the DB on the remote, removing the large space requirement for the lightweight clients but still offering the full functionality. 

I have a couple of environments where this would be really handy and I can imagine a few other uses for this type of distributed model, while not going full on cloud model.

P.S. - ACK, I totally forgot something!  This only works in Linux!  I've taken some small stabs at getting it to work in Windows, but leveldb is a complete b**** in Windows.  Luckily, the core devs have done a lot of LevelDB stuff in Windows already, which I/we may be able to cannibalize, but there's no question it's going to be painful.

So when I talk about "testing this weekend", I'm most definitely talking about Linux testing.  If you're not on Linux, I won't have anything for you yet.

I will offer a 2 BTC bounty to anyone who can get this new stuff integrated and compiling in MSVS 2008 (it must be 2008, for now).  If it's too difficult, it may be possible to do it mingw, though Armory project itself would have to be ported to mingw and I'm not too keen on that.  But at the moment, any solution is a sufficient one.  It is probably possible to compile the leveldb & boost stuff into a .dll and access it from the existing MSVS project with a lot less pain (I recommend that because the core devs got it working using mingw but I really need to run the Armory C++ utilities in MSVS for debugging).

If you want to try this, you must follow the directions on the Building Armory from Source page, and get it working with the master branch.  You should be able to compile the C++ utilities in MSVS 2008 and run Armory off the compiled product.  Then switch to the ramreduceleveldb branch and... have fun.  If you can compile BlockUtils.cpp on the ramreduceleveldb branch in MSVS, then I can figure out the rest from there.

PLEASE document your process if you try it.  I've gotta understand how to integrate leveldb into my project.  Knowing that the core devs had gotten it working in Windows was sufficient for me to know I can get it working in Windows, but I forgot I still have to go through that process!

Well, it's not done yet.  But those last two tests were the most exciting, because they demonstrate bootstrapping the old wallet system with the database data.  i.e. I actually left most of the thoroughly-tested, thoroughly-complex wallet infrastructure in place, and simply seed it with data from the database and it works smoothly.  That means that most of the GUI will work without modification (except for changing some function names).

I'm going to start trying to run the GUI with the new DB stuff and see what's still missing or what needs to be tweaked.  Before I'm totally done with the full upgrade, I might ask people to run it on mainnet but with a different home dir, to let me know how the initial DB build goes.  I've had mixed results with it -- the last test I did took 6 hours to build, but I've had a lot that took 3 hours.  I could use some extra data points on that one (by the way, I have one more optimization to put in place that should cut down that time even more, but at the moment I consider it "usable").

I'll let you know when I have something.  

Same here!

I'd be happy to help test. Is there anything in particular you need, or do you want people to just start using the new version?

Just another update, because I know people are anxious, and I'm making substantial progress.  I'll probably also need people to help with some testing this weekend


I don't have all the reliability features in there.  Meaning, this should probably work 98% of the time, but it really needs to be more like 99.999%.  Things like DB/Armory crashes mid-build or while on a block that is eventually orphaned, etc, are not well-tested.  But I gotta start somewhere...

P.S. - Did I mention how much I love GoogleTest?  it's made my life sooooo much easier...

Armory - Fort Knox of Bitcoin

Your progress sounds damn cool!!!!!  Better honestly than i was expecting!!  Ive been fretting on other threads about this in the past, but the world urgently needs a solution that allows local blockchain browsing a-la blockchain.info.  So if you can also deliver an engine that efficiently does this without a horrible rescan every time, then Armory "server" will have a killer app that others will lust for!  :-)   This is totally above and beyond the existing core coolness from your current feature set!

Woo hoo, ill tip you again fat style when the day comes that i can do an arbitrary blockchain query without waiting ages for a chain scan, and without telling everyone who is spying on web traffic the addresses i'm interested in by visiting blockchain.info.  They have a great service but frankly it's dangerous.  A query there is enough to publicly link your real world identity to those addresses that you are interested in.  Tor has its limits in terms of performance and market penetration.

No reason my isp or random law agencies deserve this info, its enough that the spies of the world could filter and log my transactions if they wanted to.  They don't deserve to also know all the other addresses i might peek into.

Don't do that. VMWare will become unstable with those settings whether you try to use Armory or not. Armory will run worse than if you ran in it in the host OS, if it even verifys a quarter of the recent block updates. Then repeat this mantra: " VMWare does not turn my PC into the computing equivalent of Doctor Who's more-capacious-on-the-inside space ship telephone booth "

*before* i attempt this, just wondering if anyone has ever tried it yet:

armory within vmware

virtual machine set to '12gb RAM' but host pc only physically has 2gb


...of course in the end the experiment/discussion will become moot if newer versions of armory use less ram

This process all sounds super exciting!
Great work, I can't wait for all the goodies there'll be soon! :-)

Armory, fck yeah!

Ente

Kind of.  At the moment there's a very minimal set of operations supported, and it will crash whenever there's a Bitcion-Qt/bitcoind blockfile split (just like the GUI does).  If you don't mind restarting and rescanning every couple days, and only the need to super-limited functionality it has -- yes, it's usable.

Right now it basically can only get you new addresses from wallets, and if it's in online mode, it can "listtransactions" and "getledger".  The listtransactions function makes an attempt to replicate what Bitcoin-Qt produces in response to the same command, but I didn't like what it produces.  It seems... sloppy.  So I implement "getledger" and "getledger_simple" which returns basically the same thing that you see on the "Transactions" tab of the GUI.  If you add up all the items returned by getledger, you will get your current balance without, and all items returned are non-overlapping.  The same cannot be said for "listtransactions."

I had started implementing more functionality, but decided it wasn't worth it until I got the new database stuff done which will make it stable.  A lot of the use cases for it require it not to crash every few days, and my efforts to fix the crash-on-blockfile-split were futile (I even made what I thought was a stressful unit-test for it, which passed, but it still segfaults when run on the network).  But I'm confident that will go away with the new DB stuff.

Yup.  And at least you get something useful out of all that extra storage space:  you can get the entire history for any address (including multisig TxOuts) nearly instantaneously.  This will make armoryd.py quite a bit more powerful than bitcoind, since it will ultimately implement everything bitcoind does plus all the arbitrary addr/script lookups.  It also means that you can import addresses and wallets without having to rescan.

And with that, it may make sense to implement a basic python shell into the expert interface, that allows users to issue commands/requests directly from a "debug" window.  For instance, you'd run "Server Version" GUI (not armoryd.py), and open the "Armory CLI" window, and type in "getAddressHistory('')", "importAddress('', '')", "findAllMultisig('')", etc.  I could make the same interface available through RPC, but that will take considerably longer because of how security-sensitive that is.

This super-node version also enables a very specific functionality that is not otherwise possible:  encrypted watching-only wallets.  Without a full address index, and without rescanning the blockchain every load, there's no way to hide the addresses in your watching-only wallet.  My new wallet design (which was side-tracked by this RAM-reduction effort) actually has a way to encrypt the public keys in a WO wallet, but it's not very useful if your database is storing only the transactions related to your wallets.  There's things you could do to improve the situation, but it won't be perfect unless you do a rescan every time you unlock your wallet and hold everything in RAM.  Or save the whole tx history as this super-node version is doing.  

So if you want absolute security and privacy, you can run this "server version" which will use a lot of HDD space, but it will run smoothly once the DB is built and you can hide the public keys in your wallet.  A lot of people have requested this functionality, because of concerns that a stolen laptop could lead to worst things than losing money.  For instance: the thief may not get your $100k worth of Bitcoin since he only has your WO wallet, but he can see that you have $100k worth of Bitcoin and can probably find your address somewhere else on the computer.   Many of my more paranoid/rich users would like to avoid that.

Profiling Update: my initial DB build is up to 221,000 blocks after 1.5 hours, and using 295 MB RAM (15 MB more than at 180k blocks).  So it looks like it may take about 3-4 hours to rebuild the whole DB and about 300 MB RAM.  Those are some solid, usable numbers!

Not only that, but it's very likely that I can build the initial database while Bitcoin-Qt is downloading the initial transaction history, meaning that Armory's DB building can be "hidden" and the whole new-user-startup may not be any worse than running vanilla Bitcoin-Qt.  I will not complicate it with that parallelization right now, but those numbers are very encouraging for a future upgrade.

Okay, enough rambling, more coding ...

Alright!  Back to where I was 10 days ago...



The difference is that now the DB operations are much better optimized for handling excessively-reused addresses like 1VayNert and all the SatoshiDice buckets (at the expense of a ton of extra DB complexity).  I'm in the middle of profiling the new stuff, but the dramatic improvement is immediately obvious:  My previous design was bottlenecked in RAM and speed by the address being updated with the most historical transactions.  Now processing time is fairly independent of that, and only dependent on blockchain size in MB.

So far, the DB itself is a bit bigger than I expected (about 1.7x the .bitcoin/blocks directory).  But that's because this is running in "super-node" mode which tracks the entire history of all addresses/scripts, which could be more appropriately named "Armory - Server Version".  But it was easier to implement than the "regular" version which needs extra logic to reliably track only a specific set of addresses (i.e. your wallets).  When this version is done, I will implement the "regular" mode which use less space and even less RAM, and may be able to work with a remote bitcoind/bitcoin-qt instance.  For now, you'll need a lot of disk space to run the new version, but RAM should no longer be a constraint.

And in case you're wondering about the RAM: it looks good (finally!).  As I type this, my testing run is already deep into SatoshiDice territory and it's been using only 276-279 MB of RAM the whole time.  And this should be the most intense part of the whole process.  It might go up a little bit once I create wallets, etc, but will also go down when it's done batching and updating the DB with thousands of tx per second.

Of course plenty of work left to do, but progress is being made!