Topic: Armory - Discussion Thread - page 144. (Read 521829 times)

Have a look at this gist for my idea about using Decimal values (or converting ints to decimals first) and then dumping in the json so it doesn't have floaty rounding display happening

https://gist.github.com/4536693

Will get back in a few hours with the other stuff I'm looking into.

edit: I am inclined to agree with you on the using integers and correct money handling rather than shitty floats or even decimals. At least in this case, when the value is received and loaded into a Decimal, it will be displayed and loaded completely accurately and not rely on the client correctly implementing rounding, which we cannot assume. As I have previously stated, I think following bitcoind api where possible is best since it makes for the least dicking around when migrating to use armory from bitcoind.

edit2: The option to set integer / decimal / string encoding is a great idea. Don't forget to allow users to set it back to the default for decimal if they desire.

edit3: I had a crack at getting the listtransactions data for the various scenarios you describe (and thus the 'way it works'), however I couldn't get any results. Frustratingly, I got raw transactions to sign (which is the second-last step of many) and then the last step is to broadcast the transaction, but my transactions were always getting rejected. I've got my progress thus far well documented and is definitely repeatable so I might try to work it out later in the week. It's worth asking on the development subform for clarification on the way the listtransactions call works, they will hopefully be able to answer it. Anyway I was creating raw transactions to match your scenarios based off this guide and testnet-in-a-box - https://people.xiph.org/~greg/signdemo.txt

The problem is that those values should be transferred as either long-integer (satoshis) or string... in which case the value can be coded exactly.  But someone decided that they'll have bitcoind should transfer them as non-exact floating point (doubles), and 0.0005000000001 for the fee is the closest float that can be encoded.  I'm fairly confident it's really just a display issue, not an armoryd.py issue.

Luckily, round()'ing will always work because doubles have enough bits to represent 8 places without rounding errors.  But I wouldn't mind if we can find a way to have the json library print out the rounded version.  I noticed in the few lines of code I stole from jgarzik (and now part of armoryd.py), he defined a "UniversalEncoder" and was able to pass it to the json.dumps() method to explain how to handle decimal.Decimal objects.  Though I mostly removed that in favor of following  Proper money handling in JSON-RPC, even though it should all give the same answers (it's less characters and more readable).  Either way, maybe there's a hint there about how to tell UniversalEncoder to print "float" values.

Anyone think it's a bad/good idea to do what I recommended:  "python armoryd.py setamountencoding COIN" or "python armoryd.py setamountencoding STRING" to set the daemon to transfer long-int-satoshis or exact strings, respectively?  That way default behavior matches bitcoind that most are expecting, but they have the option to change it with one line at the top of their script.

-----

By the way (mostly for mav), I added the capability for armoryd.py to detect if there's already an instance of armoryd.py running, and if there is, it will connect to it, pass the command line arguments to it, print the reply, then disconnect.  I believe that's exactly what bitcoind does.

Also for anyone who wants to use this, you need to have a "/home/username/.armory/armoryd.conf" file or "C:\Users\username\AppData\Roaming\Armory\armoryd.conf" in order to use it.  The file contains a single line:  "username:password".  Make sure to "chmod 600 armoryd.conf"

I would have thought there's away to convert those floats to the correct decimal values, although maybe that was just a client... I'll look into it tonight cause those floats are horrid! Also I'll look at bitcoind listtransactions output if I get a chance.

Sounds like it's going well so far for the most part.

I implemented a crappy version of listtransactions that probably doesn't behave much like the bitcoind version, but it is just as confusing (so at least there's something in common).  For each incoming transaction it gives one entry per txOut you own.  For each outgoing transaction, it gives one entry for the total output value (i.e. for your own contributions minus change) and one entry for each non-change receiving address (which can be sending or receiving depending on whose/howmany outputs there are).  I'm sure people will try to use it and complain.  Then I'll fix it to match what they were expecting...

However, I added my own version of listtransactions called "getledger" and "getledgersimple".  It has one entry per transaction and is a pure partition of your wallet activity.

For each transaction "getledgersimple" returns:

• "direction" is {'send', 'receive', 'toself'}.  To self is any tx for which you own all inputs and outputs.
• "amount" is the total BTC exchanged (inputs - outputs + change + fee)).  It is always positive.  For example, if I send someone 10 BTC with 0.05 BTC fee and 5 BTC change to myself, "amount is 10".  For to-self tx, change address is assumed to the be the address with the higher generation order (since the wallets always generate change addr after recipients are selected)
• "netdiff" is the change in your wallet balance induced by the transaction.  For the above transaction, netdiff would be -10.05. If you send money to yourself, netdiff will be the negative of the fee paid.  Summing all returned "netdiff" values returned will give you your wallet balance
• "fee" is the fee paid for the transaction.  It is always positive, and the caller only has to pay attention to it on outgoing tx if they want (and make it negative).  Or they may want to know how much fee was paid to determine how safe an incoming tx is. (returns zero in bitcoind for incoming transactions)
• "txsize" is size in bytes of the tx (included for same reason as fee)
• "firstrecip" is the first "target" of the transaction.  For simple transactions this is your own address for incoming tx, or the other person's address for outgoing tx
• "changerecip" is the first address that may reasonably be interpretted as change.  Someone sending you money as part of a multi-output tx, this will return the first output address that isn't yours.  Or if the transaction is outgoing, it's the first txout with address in your wallet.

For each transaction "getledger" returns the same as the simple one, but with the following additional output.  This accommodates users that want to handle more-complex, multi-recipient transactions and kind of just want all the information:

• "senderme":  list of all [address,value] pairs on the input side that are part of this wallet
• "senderother" list of all [address,value] pairs on the input side that are not part of this wallet
• "recipme":  list of all [address,value] pairs on the output side that are part of this wallet
• "recipother" list of all [address,value] pairs on the output side that are not part of this wallet

mav originally setup the thing to handle watching-only wallets only.  I will eventually upgrade it to work on any wallet, and maybe even multiwallet.  But for now it's actually starting to have usefulness, and I'll hold off digging into too many more calls until I get some feedback.   For now, I want to dig into fixing the instability issues I referred to earlier with blocksplits and simultaneous blocks.

P.S. -- This is all on the armoryd branch, and you must have a armoryd.conf file in your Armory home dir specifying "username:password", then use that in the URL for the connecting client.  

---

---

Example of simple and nonsimple getledger calls:

getledgersimple

getledger



P.S -- As far as I can tell, the ultra-high-precision values/amounts are due to the python-JSON library display.  It's a display issue caused by the fact that someone a long time ago decided the JSON interface should transport floats for amounts, instead of strings or satoshis, and the JSON library just prints the raw floats.  Luckily, double-precision values have enough precision that this will never screw up from a simple round() call.  But clearly, this is sub-optimal.  And since I'm using a built-in json.dumps call in python, I can't really modify the behavior.  

I'm thinking I could add a "setamtcoding COIN" or "setamtcoding STRING" to tell the RPC server to always encode values as longs (in Satoshis) or string values with at most 8 decimal places.  I could also make the last argument to every RPC call be "COIN" or "STRING" to tell the server what to output for that particular call.  But that seems kind of messy.

Can someone please help me out understanding how listtransactions really works in Bitcoin-Qt?  This is weird and non-intuitive, and I've spend ungodly amounts of time trying to figure out what they're doing when I just want to spit out my pre-computed ledger (which is what I would expect users to want).  I'm going to add my own version (probably "listledger"), but I feel like I need to have some matching behavior to Bitcoin-Qt, if I don't go insane first.

I'm going to give some hypothetical transactions, and maybe someone can tell me how to break them into outputs of the RPC call "listtransactions".  I'll pay 1.0 BTC for the answers because I've wasted way too much time on this already (1 BTC for all answers, not each)!  It's easily worth that much to have someone tell me (or do the research for me) and save my sanity!  I'm not looking for "the gist", I need details!  Mainly it has to do with transactions that have multiple outputs.  Specifically, how many "results" are produced by listtransactions for each one of these tx, and what are the {"address", "amount", "category", and "fee"}  (and signs on them).  

Addresses A, B, C and D are in my own wallet.  X,Y,Z are other addresses.  

Transaction #1 (SIMPLE Out, No Fee)
------
IN:  A  10
IN:  B   5
---
OUT: X   3
OUT: C  12
-----

Transaction #2 (SIMPLE Out, With Fee)
-----
IN:  A  10
IN:  B   5
---
OUT: X   3
OUT: C  11
--
FEE:     1
-----

Transaction #3 (SIMPLE In, No fee)
-----
IN:  X  10
IN:  Y   5
---
OUT: A   3
OUT: Z  12
-----

Transaction #4 (SIMPLE In, With Fee)
-----
IN:  X  10
IN:  Y   5
---
OUT: A   3
OUT: Z  11
--
FEE:     1
-----

Now the not-so-simple ones (Multi output)

Transaction #5  
-----
IN:  A  10
IN:  B   5
---
OUT: C   3
OUT: D   5
OUT: Z   6
--
Fee:    1
-----

Transaction #6
-----
IN:  A  10
IN:  B   5
---
OUT: C   3
OUT: X   5
OUT: Z   7
-----


Transaction #7
-----
IN:  X  10
IN:  Y   5
---
OUT: C   3
OUT: D   5
OUT: Z   6
--
FEE:     1
-----

Transaction #8
-----
IN:  A  10
IN:  B   5
---
OUT: C   3
OUT: D  12
-----

Transaction #9
-----
IN:  A  10
IN:  B   5
---
OUT: C   3
OUT: D  10
OUT: D   1
--
FEE:     1
-----

Transaction #10
-----
IN:  A  10
IN:  B   5
---
OUT: C  15
-----

Armory version 0.87-beta is released.  You should get notifications next time you restart.  If not, Get it here!

@justusranvier:  this time I remembered to tag and sign the repo on this release.  "git tag -v v0.87-beta" should give you a thumbs up.

However, I have found two subtle bugs that trigger rarely, but clearly need to be fixed.  I don't think it's critical, but since I'll be setting up a JSON-RPC server and it will be expected to run reliably, I'm going to have to fix them.  They are:

(1) Armory crashes just after a blockfile split -- the logic to handle a split in real-time was working at one point, but the upgrade to accommodate Bitcoin-Qt 0.8+ seems to have broken it.
(2) If Armory receives two blocks in the same heartbeat (2 sec), it apparently loses sync and no longer see new blocks.  If you leave Armory running long enough, it will happen just by chance.  If you have tx that are stuck at X confirmations, then you can right click on the tx and "Show on Blockchain.info" to see if you are out of sync.  

If you are hit by (2), simply restart.  Though, you should still be able to send and receive coins in this state, but none of the transactions will get past 0 zero confirmations.

Spent the day yesterday exercising and socializing, so I'm sore and introverted today.  I plan to spend most of the day hiding in my office and fixing these bugs, and hopefully also get a working RPC server going.  Thanks again to mav, who gave me an excellent template for the RPC server, and also got it setup with authentication.  Looks like I just gotta fill in the RPC commands and fix those bugs!

THEN back to the new wallets

Well, I was quite inconvenienced by the upload interface at googlecode, but then I found their upload script.  Now it's a breeze! 
A short python script, and I can upload all the installers correctly tagged, and even spit out a list of forum-formatted strings like below! 

Version 0.87-beta for Ubuntu/Debian 64-bit
Version 0.87-beta for Ubuntu/Debian 32-bit
Version 0.87-beta for Windows 64-bit
Version 0.87-beta for Windows 32- and 64-bit
SHA256 hashes of all installers

Hopefully, this will streamline my release process, since I've always found it to be quite a pain to manually select and upload each individual file, tag them consistently, and then copy the links to a post like this

Please do a quick sanity check on the download and install process.  If there are no problems with running these, I'll merge into master which will trigger notifications!

It's ultimately 64 bytes:  32-bytes for the root key and 32 bytes for the chaincode.  It's not necessary to have that much entropy, but that's how I designed it a year ago and then it stuck.  The new wallets will probably use 24 bytes of entropy (192 bits) for the root seed, and then you back that up instead of the rootkey&chaincode (those still exist, but they will be derived from the 24-byte seed).


Wow, that was fast!  Glad to see that fixed the problem for you

Speaking of storing wallets - how many bits is the seed?

Oh, and the new testing build has raw txs working fine. Thanks

@mav

You're amazing!  You made that look so easy... I guess you just have experience with this to realize that I didn't need all that extra garbage to do something so simple.  Thanks so much -- now I can get around to actually implementing all the RPC commands.  Ideas are welcome for how to deal with it for things that aren't the same for bitcoind and Armory:  for instance, bitcoind has accounts, Armory has wallets.  I was thinking it could be a multi-wallet interface, instead of accounts and you give it the wallet ID where you would otherwise put an "account" in the JSON interface.  Mapping other functionality over for watch-only wallets may be kind of fuzzy.  There's also things where Armory can provide a bit more than bitcoind could -- for instance, the ability to sweep or import private keys could be added (I'm thinking I could have that happen in another thread so that it wouldn't have to go offline... [wait, I could do that in Armory GUI, too... ...?]).



@Ploo
Yes!  I actually was just messing with something and found this bug and fixed it.  I don't know if I fixed it in the testing branch yet, but I'll definitely try to get it in, soon.  I was annoyed, myself, that I couldn't get the raw tx out of it...



@wachtwoord
The alternative you describe is really not much better than just maintaining an encrypted wallet on your hot computer.  What it prevents is someone who has gotten remote access to your machine and manually digging around your filesystem looking for wallet files.  That's not to say that such things don't happen, but I believe the real threats are viruses that get on your computer and siphons off data and send it back to "home base" when it finds it (it will farm the data and send it back the next time it's online).  In this case it probably doesn't even have to send any data back... it just reads your wallet when decrypted and creates a transaction sending all BTC to its owner the next time it's online.

Any computer with access to the internet is vulnerable to these viruses.  You can get them from various exploits usually relating to your browser, or opening PDF/.xls documents with embedded exploits, etc.  However, in order to compromise a truly-offline computer, there's a couple orders-of-magnitude more work for the attacker to do.  They must (1) find a way to hide data on your USB key that (2) exploits an auto-run vulnerability when plugged into the offline computer, and then (3) be able to automatically find the data and copy it back, hidden, on the USB key to get it back to the offline computer.  And these exploits are much more complicated when they don't even know what OS the offline system is.  And if I can find another way to transfer data between systems not using USB keys, then this would even an order of magnitude better...

I would look into this idea posted by N.Z..   It's a bit more work but is definitely a very reasonable solution for a single-computer setup (in fact, I've been thinking about bundling the tails packages like N.Z. suggested, to make this easier).  This means that your wallet only ever touches this sandbox that has no access to internet, and is reset to default configuration on every boot.

As for question 3 (paper backups), you just said in question 2 that you believe flash storage is frail.  This is exactly why you make paper backups.  If you print a paper backup and put it in a safe-deposit box or fold it into a book on your bookshelf, it will still be readable 20+ years from now . It doesn't even have to survive "well", as you could pull the data out of a terribly-faded copy with a bit of work.  I mean, you've seen books that are 50+ years old and their pages are still readable.

Now compare that to any kind of digital media.  You can drop a flash drive, or sit on it or bend it, or put it too close to a magnet, or it just decays and it's no longer usable.  What's your confidence level that even a well-treated USB drive sitting in a metal box will still work 5 years from now?  20 years?    Even CDs and DVDs have expected lifetimes of 2-10 years, but that varies widely depending on lots of factors.  With the paper backup, unless it physically burns to ash, you know it will still provide the data you need to recover your wallet in 20 years.... probably much longer than that.


@picobit
This is probably strictly better than just having an encrypted hot wallet, since it definitely much more complicated to automatically find data in a VM.  It's not as good as an offline solution, but I bet a lot of viruses sit around scanning directories looking for wallets, and it would have to be kind of advanced to be able to pick apart the VM filesystem and/or memory space.  It's definitely doable, but probably not deterministic (especially if you use some obscure OS in your VM).   
(EDIT: though, I'd still place this at substantially less-secure than a real offline solution)

I'm looking to generate a raw plain hex signed tx that's ready to push later via blockchain/pushtx or bitcoind/sendrawtx.

I've noticed that copy raw tx returns the same value pre and post signing, which i presume to be the unsigned data. Is this a bug?

Another possibility would be to have the offline wallet in a virtual machine without internet access.  Of course a virus could still get access to the file in which the virtual machine harddisk is placed, and thus the "offline" wallet, but the virus would more or less need to be tailored to attack your personal configuration.  So if you keep thousands of bitcoins, you definitely want a dedicated offline machine, but for anything less I would guess that the virtual machine - or even the setup you suggested above - would for all practical purposes be secure.

I'm not etotheipi, but no this is not as secure as an offline computer. You could have a virus on your computer that gathers your wallet information while you are offline and the wallet is unlocked, and then transmits it once the computer is back online.


I think the scenario I outlined above is a bigger risk. At least with an offline wallet, you know that it cannot be attacked except by someone in the same physical location.


You can make archival quality prints, which is I think mostly a function of the paper used. The reason for backing up on paper is that digital media can always fail. Paper is long lasting and easily securable.


If you have a virus, it can log your passphrase when you enter it, and then empty your wallet at its convenience. This is impossible with an offline computer keeping your wallet. I recommend the Raspberry Pi. It is cheap, low powered, and eminently hackable.

Hi,

I am not sure whether this the right place for this. Etotheipi, if you prefer I ask this in a separate thread just say so and I'll move it.

What I am asking? I am looking to improve my security and so want to start using Armory in the online<-->Offline wallet configuration. However I do not have a suitable system for being an offline system and honestly, this is not the configuration I would choose. Having a system dedicated only to my wallet that is only ever booted up for getting coins from my cold wallet seem awfully wasteful and there is a risk the system doesn't even boot when I finally have to use it. if that happens of course I can recover the wallet and recover on a new system but this only proves that a dedicated offline wallet is unnecessary. It is only necessary for the private keys (aka the cold wallet) to never reside on a computer that is at that time connected to the internet.

So what I am proposing is the following configuration. I only have 1 pc with an internet connection and two installations of Armory.

1) An "offline" Armory that only gets started by me when I have physically disconnected the pc from the internet (As  required remove any Ethernet cable, any wireless network card and any wireless dongle, even disale Bluetooth if this is present). Furthermore the wallet file resides on removable medium (most likely flash drive) which is only ever plugged in when the pc is disconnected from the internet.

2) A normal online wallet with a watch-only version of the wallet in cold storage and hot-wallets as required.

Questions I have:

1) Do you agree this configuration is equally secure as a configuration in which a dedicated offline system?
2) The biggest risk here (but also in the proposed configuration with the dedicated offline system) is keeping the offline wallet files physically secure. Especially as you will want to keep backups (flash storage is frail). Do you agree?
3) Why do you recommend making a paper backup? Personally I really do not trust paper (it decays even faster than any other sort of media). I would like to hear your thoughts because you have obviously put more thought into this than I.
4) Finally, atm I already use Armory (although not yet for the majority of my funds) in a configuration with 1 (hot) wallet on a system connected to the internet. I have encrypted the wallet with a very long pass-phrase (>50 characters to give an indication). Will using a offline configuration with a real and watch only wallet really improve my security? What exact use cases do I protect myself against?

Thank you

The changes to add auth are in the link below. Manually tested only. Returns the string "Unauthorized" if the user is unauthorized (no username or wrong password), otherwise returns the desired json. Credit must go to the author of txjsonrpc for including a nice wrapper for auth.

https://github.com/thedawnrider/BitcoinArmory-Daemon/commit/32cfd4697fed5546c57f5815b15a37f79988db74

inspired entirely by

https://github.com/bbqsrc/txjsonrpc/blob/master/examples/webAuth/server.tac

and if you want to do password storage differently than a text file (which I know you don't)

https://twistedmatrix.com/documents/8.1.0/api/twisted.cred.checkers.ICredentialsChecker.html

Hope this helps.

Alright, so I finally have a "release candidate" for 0.87 (it is 0.86.9-testing).  I just uploaded the installers to the code.google downloads page.  As soon as I get a sanity check, I'll turn it into 0.87-beta and finally release it with all the extra goodies...

http://code.google.com/p/bitcoinarmory/downloads/detail?name=armory_0.86.9-testing_win64.msi
http://code.google.com/p/bitcoinarmory/downloads/detail?name=armory_0.86.9-testing_windows_all.msi
http://code.google.com/p/bitcoinarmory/downloads/detail?name=armory_0.86.9-1_amd64.deb
http://code.google.com/p/bitcoinarmory/downloads/detail?name=armory_0.86.9-1_i386.deb

---

Now, someone with networking/JSON/python-twisted experience please help me.  I'm working on creating the JSON-RPC server, and I have all the pieces together except for the authentication.  I know I just need very simple Digest authentication using username and password in a file only readable by the user.  However, for such a "simple" thing, there's like 800 different ways to get there, and I have no idea how to do it.

From my research, I think I have found all the pieces I need, but I'm not sure how to put them together yet ... it's like a jigsaw puzzle.   I see that I need:

    twisted.web.guard.resource.Resource to be protected
    twisted.cred.portal.IRealm that creates "avatars" (incarnations of the user on the server)
    twisted.web.guard.DigestCredentialFactory specifying the realm
    twisted.cred.portal.Portal to that links the realm to a ...
    twisted.cred.checkers.FilePasswordDB object specifying the user/pass list
    twisted.web.guard.HTTPAuthSessionWrapper



The server code and launch without authentication looks like:

You can see the full source code as mav gave me here (with slight modifications, and adding all the complicated pieces above)

Anyone want to help me figure out how to create and hook together all these things?  Guarded resources, realms, avatar factories, a portals, etc? (sounds like I'm playing a RPG computer game with knights and wizards and magic, doesn't it?)

Very good to hear!!!
Thanks mav and etotheipi.

This way Armory is more valuable for back-end solutions and gives a lot of opportunities to add functionality for bitcoin (business) users with the network security and blockchain of the original bitcoin client and the wallet security of Armory.

Standing ovation.
Keep up the good work!

Concerning the dependencies, is txjson-rpc still required for armoryd.py?
If so, it is preferable to add it to the dependencies (to be able to use armoryd.py)


Just started right now!

Just pushed 0.86.8 to the testing branch.  At this point will just be released as 0.87.  Fixed tons of bugs I can't even enumerate (most of them tiny). 

@ HostFat:  I think you'll be pleasantly surprised.  All QR codes are inflatable!  It works in Linux, at least...

Also got permission from mav to add his JSON-RPC code to the project (committed as armoryd.py).  The code is not up to my quality standards yet, but is fantastic skeleton for me to finally get that support in there.  I may be posting questions here for others more knowledgeable about JSON servers.  So far, it looks like what he's got works, but it needs some more security features, and some more commands.  Will be making that a priority, actually...

Please test!

me too.

Armory source code is not in the correct condition for translations.  There is currently no unicode support, and I never implemented any of the hooks for adding translations/localizations...

That said, the new wallet files will have unicode support, and I'll take the opportunity to upgrade the GUI to go with it.  At the same time, I will add those hooks needed for translations.  THEN, I can start soliciting help doing translations.  Until then, there's not much to that can be done (any unicode characters added by the user, just about anywhere, crash Armory).