Pages:
Author

Topic: [ANN] BitcoinArmory-Daemon - armory on web servers - page 4. (Read 20542 times)

hero member
Activity: 868
Merit: 1000
Thanks for the response mav. Although I never touched python in my life nor am I a programmer, I added some codes from ArmoryQt and manage to get getbalance and listtransactions working (for this last only a few items are returned and some are wrong so I need to work more on that).

Here is yur modified code so far, ugly as hell lol:
Code:
################################################################################
#
# Copyright (C) 2012, Ian Coleman
# Distributed under the GNU Affero General Public License (AGPL v3)
# See http://www.gnu.org/licenses/agpl.html
#
################################################################################

from twisted.web import server
from twisted.internet import reactor
from txjsonrpc.web import jsonrpc

import decimal
import os

RPC_PORT = 7070
STANDARD_FEE = 0.0005 # BTC
       
class Wallet_Json_Rpc_Server(jsonrpc.JSONRPC):

    def __init__(self, wallet):
        self.wallet = wallet

    def jsonrpc_getnewaddress(self):
        addr = self.wallet.getNextUnusedAddress()
        return addr.getAddrStr()

    def jsonrpc_getbalance(self):
        int_balance = self.wallet.getBalance()
        decimal_balance = decimal.Decimal(int_balance) / decimal.Decimal(ONE_BTC)
        return float(decimal_balance)

    def jsonrpc_getreceivedbyaddress(self, address):
        if CLI_OPTIONS.offline:
            raise ValueError('Cannot get received amount when offline')
        # Only gets correct amount for addresses in the wallet, otherwise 0
        addr160 = addrStr_to_hash160(address)
        txs = self.wallet.getAddrTxLedger(addr160)
        balance = sum([x.getValue() for x in txs if x.getValue() > 0])
        decimal_balance = decimal.Decimal(balance) / decimal.Decimal(ONE_BTC)
        float_balance = float(decimal_balance)
        return float_balance

    def jsonrpc_sendtoaddress(self, bitcoinaddress, amount):
        if CLI_OPTIONS.offline:
            raise ValueError('Cannot create transactions when offline')
        return self.create_unsigned_transaction(bitcoinaddress, amount)

    def jsonrpc_listtransactions(self, p_count=10, p_from=0):
        #TODO this needs more work
        result2 = []
        txs = self.wallet.getTxLedger('blk')
        #txs = json.dumps(txs)
        #id_le_pairs = [[le] for le in txs]
        for x in txs:
            #result.append('{')
            account = ''
            txHashBin = x.getTxHash()#hex_to_binary(x.getTxHash())   
            cppTx = TheBDM.getTxByHash(txHashBin)
            pytx = PyTx().unserialize(cppTx.serialize())
            for txout in pytx.outputs:
                scrType = getTxOutScriptType(txout.binScript)
                if not scrType in (TXOUT_SCRIPT_STANDARD, TXOUT_SCRIPT_COINBASE):
                  continue
                address = hash160_to_addrStr(TxOutScriptExtractAddr160(txout.binScript))
            if x.getValue() < 0:
            category = 'send'
            else:
            category = 'receive'
            amount = x.getValue()
            confirmations = TheBDM.getTopBlockHeader().getBlockHeight() - x.getBlockNum()+1
            blockhash = ''
            blockindex = ''#x.getBlockNum()
            txid = str(binary_to_hex(x.getTxHash()))
            time = ''#x.getTxTime()
            result = {'account':account,'address':address,'category':category,'amount':amount,'confirmations':confirmations,'blockhash':blockhash,'blockindex':blockindex,'txid':txid,'time:':time}
            #result.append('}')
            result2.append(result)
            #print addr
            #dumpObj(x)   
        #result = json.dumps(result)
        return result2

       
    # https://bitcointalk.org/index.php?topic=92496.msg1126310#msg1126310
    def create_unsigned_transaction(self, bitcoinaddress_str, amount_to_send_btc):
        # Get unspent TxOutList and select the coins
        addr160_recipient = addrStr_to_hash160(bitcoinaddress_str)
        totalSend, fee = long(amount_to_send_btc * ONE_BTC), (STANDARD_FEE * ONE_BTC)
        spendBal = self.wallet.getBalance('Spendable')
        utxoList = self.wallet.getTxOutList('Spendable')
        utxoSelect = PySelectCoins(utxoList, totalSend, fee)

        minFeeRec = calcMinSuggestedFees(utxoSelect, totalSend, fee)[1]
        if fee            if totalSend + minFeeRec > spendBal:
                raise NotEnoughCoinsError, "You can't afford the fee!"
            utxoSelect = PySelectCoins(utxoList, totalSend, minFeeRec)
            fee = minFeeRec

        if len(utxoSelect)==0:
            raise CoinSelectError, "Somehow, coin selection failed.  This shouldn't happen"

        totalSelected = sum([u.getValue() for u in utxoSelect])
        totalChange = totalSelected - (totalSend  + fee)

        outputPairs = []
        outputPairs.append( [addr160_recipient, totalSend] )
        if totalChange > 0:
            outputPairs.append( [self.wallet.getNextUnusedAddress().getAddr160(), totalChange] )

        random.shuffle(outputPairs)
        txdp = PyTxDistProposal().createFromTxOutSelection(utxoSelect, outputPairs)

        return txdp.serializeAscii()
       
    def jsonrpc_createCombinedLedger(self, wltIDList=None, withZeroConf=True):
      """
      Create a ledger to display on the main screen, that consists of ledger
      entries of any SUBSET of available wallets.
      """

      start = RightNow()


      self.combinedLedger = []
      totalFunds  = 0
      spendFunds  = 0
      unconfFunds = 0
      currBlk = 0xffffffff
      if TheBDM.isInitialized():
         currBlk = TheBDM.getTopBlockHeader().getBlockHeight()

      wlt = self.waldlet
      totalFunds += wlt.getBalance('Total')
      spendFunds += wlt.getBalance('Spendable')
      unconfFunds += wlt.getBalance('Unconfirmed')


      self.combinedLedger.sort(key=lambda x: x[LEDGERCOLS.UnixTime], reverse=True)
      self.ledgerSize = len(self.combinedLedger)
     
      print totalFunds
      return

class Armory_Daemon():

    def __init__(self):

        print "Reading wallet file"
        self.wallet = self.find_wallet()
        self.loadBlockchain()
       
        use_blockchain = not CLI_OPTIONS.offline
       
        print "Initialising server"
        reactor.listenTCP(RPC_PORT, server.Site(Wallet_Json_Rpc_Server(self.wallet)))

        self.NetworkingFactory = ArmoryClientFactory( \
                                func_loseConnect=self.showOfflineMsg, \
                                func_madeConnect=self.showOnlineMsg, \
                                func_newTx=self.newTxFunc)
                               
        reactor.connectTCP('127.0.0.1', BITCOIN_PORT, self.NetworkingFactory)
        reactor.callLater(5, self.Heartbeat)
        self.start()

    def start(self):
        print "Server started"
        reactor.run()
           
    def newTxFunc(self, pytxObj):
        # Cut down version from ArmoryQt.py
        TheBDM.addNewZeroConfTx(pytxObj.serialize(), long(RightNow()), True)
        TheBDM.rescanWalletZeroConf(self.wallet.cppWallet)

        # TODO set up a 'subscribe' feature so these notifications can be
        # pushed out to interested parties.

        # From here down is display purposes only, copied from ArmoryQt.py
        message = "New TX"
        le = self.wallet.cppWallet.calcLedgerEntryForTxStr(pytxObj.serialize())
        if not le.isSentToSelf():
            txref = TheBDM.getTxByHash(le.getTxHash())
            nOut = txref.getNumTxOut()
            recips = [txref.getTxOut(i).getRecipientAddr() for i in range(nOut)]
            values = [txref.getTxOut(i).getValue()         for i in range(nOut)]
            idxMine  = filter(lambda i:     self.wallet.hasAddr(recips[i]), range(nOut))
            idxOther = filter(lambda i: not self.wallet.hasAddr(recips[i]), range(nOut))
            mine  = [(recips[i],values[i]) for i in idxMine]
            other = [(recips[i],values[i]) for i in idxOther]

            # Collected everything we need to display, now construct it and do it
            if le.getValue()>0:
               # Received!
               message = 'Bitcoins Received!'
               totalStr = coin2str( sum([mine[i][1] for i in range(len(mine))]), maxZeros=1)
               message += '\nAmount: \t%s BTC' % totalStr.strip()
               if len(mine)==1:
                  message += '\nAddress:\t%s' % hash160_to_addrStr(mine[0][0])
                  addrComment = self.wallet.getComment(mine[0][0])
                  #if addrComment:
                     #message += '\n%s...' % addrComment[:24]
               else:
                  message += '\n'
            elif le.getValue()<0:
               # Sent!
               message = 'Bitcoins Sent!'
               totalStr = coin2str( sum([other[i][1] for i in range(len(other))]), maxZeros=1)
               message += '\nAmount: \t%s BTC' % totalStr.strip()
               if len(other)==1:
                  message += 'Sent To:\t%s' % hash160_to_addrStr(other[0][0])
                  addrComment = self.wallet.getComment(other[0][0])
                  #if addrComment:
                     #message += '\n%s...' % addrComment[:24]
               else:
                  dispLines.append('')
        else:
            amt = self.determineSentToSelfAmt(le, self.wallet)[0]
            message = 'Wallet "%s" just sent %s BTC to itself!' % \
               (self.wallet.labelName, coin2str(amt,maxZeros=1).strip())
               
        print message

    def determineSentToSelfAmt(self, le, wlt):
      """
      NOTE:  this method works ONLY because we always generate a new address
             whenever creating a change-output, which means it must have a
             higher chainIndex than all other addresses.  If you did something
             creative with this tx, this may not actually work.
      """
      amt = 0
      if TheBDM.isInitialized() and le.isSentToSelf():
         txref = TheBDM.getTxByHash(le.getTxHash())
         if not txref.isInitialized():
            return (0, 0)
         if txref.getNumTxOut()==1:
            return (txref.getTxOut(0).getValue(), -1)
         maxChainIndex = -5
         txOutChangeVal = 0
         txOutIndex = -1
         valSum = 0
         for i in range(txref.getNumTxOut()):
            valSum += txref.getTxOut(i).getValue()
            addr160 = txref.getTxOut(i).getRecipientAddr()
            addr    = wlt.getAddrByHash160(addr160)
            if addr and addr.chainIndex > maxChainIndex:
               maxChainIndex = addr.chainIndex
               txOutChangeVal = txref.getTxOut(i).getValue()
               txOutIndex = i

         amt = valSum - txOutChangeVal
      return (amt, txOutIndex)

    def showOfflineMsg(self):
        print "Offline - not tracking blockchain"

    def showOnlineMsg(self):
        print "Online - tracking blockchain"

    def find_wallet(self):
        fnames = os.listdir(os.getcwd())
        for fname in fnames:
            is_wallet = fname[-7:] == ".wallet"
            is_watchonly = fname.find("watchonly") > -1
            is_backup = fname.find("backup") > -1
            if(is_wallet and is_watchonly and not is_backup):
                wallet = PyBtcWallet().readWalletFile(fname)
                # Register all wallets with TheBDM
                TheBDM.registerWallet( wallet.cppWallet )
                print "Using wallet file %s" % fname
                return wallet
        raise ValueError('Unable to locate a watch-only wallet in %s' % os.getcwd())
           
    def loadBlockchain(self):

      print "Loading blockchain"
      BDM_LoadBlockchainFile()
      self.latestBlockNum = TheBDM.getTopBlockHeader().getBlockHeight()
   
      # Now that theb blockchain is loaded, let's populate the wallet info
      if TheBDM.isInitialized():
         mempoolfile = os.path.join(ARMORY_HOME_DIR,'mempool.bin')
         self.checkMemoryPoolCorruption(mempoolfile)
         TheBDM.enableZeroConf(mempoolfile)
   
        # self.statusBar().showMessage('Syncing wallets with blockchain...')
         print "Syncing wallets with blockchain..."
         print "Syncing wallet: ", self.wallet.uniqueIDB58
         self.wallet.setBlockchainSyncFlag(BLOCKCHAIN_READONLY)
         self.wallet.syncWithBlockchain()
             
               
         #self.createCombinedLedger()
         #self.ledgerSize = len(self.combinedLedger)
         #self.statusBar().showMessage('Blockchain loaded, wallets sync\'d!', 10000)

    def checkMemoryPoolCorruption(self, mempoolname):
      if not os.path.exists(mempoolname):
         return

      memfile = open(mempoolname, 'r')
      memdata = memfile.read()
      memfile.close()

      binunpacker = BinaryUnpacker(memdata)
      try:
         while binunpacker.getRemainingSize() > 0:
            binunpacker.get(UINT64)
            PyTx().unserialize(binunpacker)
      except:
         os.remove(mempoolname);
         LOGWARN('Memory pool file was corrupt.  Deleted. (no further action is needed)') 
               
    def Heartbeat(self, nextBeatSec=2):
       """
       This method is invoked when the app is initialized, and will
       run every 2 seconds, or whatever is specified in the nextBeatSec
       argument.
       """
       # Check for new blocks in the blk0001.dat file
       if TheBDM.isInitialized():
          newBlks = TheBDM.readBlkFileUpdate()
          self.topTimestamp   = TheBDM.getTopBlockHeader().getTimestamp()
          if newBlks>0:
             self.latestBlockNum = TheBDM.getTopBlockHeader().getBlockHeight()
             LOGINFO('New Block! : %d', self.latestBlockNum)
             didAffectUs = False
             prevLedgerSize = len(self.wallet.getTxLedger())
             self.wallet.syncWithBlockchain()
             TheBDM.rescanWalletZeroConf(self.cppWallet)

       self.wallet.checkWalletLockTimeout()

       reactor.callLater(nextBeatSec, self.Heartbeat)
       
## {{{ http://code.activestate.com/recipes/137951/ (r1)
def printDict(di, format="%-25s %s"):
    for (key, val) in di.items():
        print format % (str(key)+':', val)

def delchars(str, chars):
    """Returns a string for which all occurrences of characters in
    chars have been removed."""

    # Translate demands a mapping string of 256 characters;
    # whip up a string that will leave all characters unmolested.
    identity = ''.join([chr(x) for x in range(256)])

    return str.translate(identity, chars)
## end of http://code.activestate.com/recipes/137951/ }}}


if __name__ == "__main__":
    from armoryengine import *
    rpc_server = Armory_Daemon()
mav
full member
Activity: 169
Merit: 107
Hi mav,

You daemon is very very useful. However I haven't been able to make any function beside getNextUnusedAddress work. getbalance() return 0 even though there's one adress with 1 btc on it, listransactions doesn't return anything (but seems like the function isn't developped yet in your code) and getreceivedbyaddress raises an unknown error.

Did you manage to make it work?

Any way I'm not a python programmer but I'll try to debug that and post what I found, if any Smiley

unclescrooge

Sorry for the slow response, I am subscribed to this thread but very rarely come on this forum any more so have not been updated on the new posts. I recently started at a new job which has sadly diverted my attention away from bitcoin dev.

I will be looking at this code on the weekend. I made some changes a while ago but never pushed them to git. By the end of the weekend check back on this thread, there should be updates. It's been a while since I've looked at this daemon, and certainly have not been using it in production on any of my systems. But from memory I also could only get getNextUnusedAddress to work properly, although the other calls did something slightly related to the required task, just not something completely useful yet.

The end of the weekend - I am sure some updates will be ready.
hero member
Activity: 868
Merit: 1000
Hi mav,

You daemon is very very useful. However I haven't been able to make any function beside getNextUnusedAddress work. getbalance() return 0 even though there's one adress with 1 btc on it, listransactions doesn't return anything (but seems like the function isn't developped yet in your code) and getreceivedbyaddress raises an unknown error.

Did you manage to make it work?

Any way I'm not a python programmer but I'll try to debug that and post what I found, if any Smiley

unclescrooge
newbie
Activity: 17
Merit: 0
Just thought to check this thread again, and that daemon looks like it could be extremely useful.  Thanks for working on this and putting it out for the public to use!

I was pointed to this thread by etothepi when I sent an email asking about generating addresses for my server to use, and ended up deciding that the best implementation for my needs was to generate chunks of addresses at a time with the method described in the first post, and putting them into a redis list (acting as a queue) which would get refilled when it dropped below a certain amount of addresses.  I'll probably stick with my current implementation as it is more than sufficient for what I need right now, but I will certainly keep an eye on this daemon and might decide to switch to it later on.  Smiley
mav
full member
Activity: 169
Merit: 107
The simple Armory Daemon is now on github. It aims to be as close to a direct replacement for the Satoshi interface as possible.

https://github.com/thedawnrider/BitcoinArmory-Daemon

Available json-rpc calls are based on the Satoshi rpc interface and are as follows:

getbalance
getnewaddress
getreceivedbyaddress
sendtoaddress

This looks very interesting. Combining this with bitcoinmonitor.net url callbacks (using the API) will provide you a complete solution to accept payments without the need to put your hot wallet online anywhere and without the hassle to set up/maintain a long list of pre-generated addresses with your bitcoinmonitor.net agent. Awesome!  Cool
As soon as I find some time I will set this scenario up for one of my sites.

The api for everything except getnewaddress depends on loading the blockchain. If you stripped it right down to just that one call, then yes it would be great when combined with bitcoinmonitor.net, and as you say, it's not even required to have bitcoin running. But for any functionality beyond getting new addresses from the wallet, you will still need to have bitcoin running so there's access to the blockchain.
hero member
Activity: 488
Merit: 500
The simple Armory Daemon is now on github. It aims to be as close to a direct replacement for the Satoshi interface as possible.

https://github.com/thedawnrider/BitcoinArmory-Daemon

Available json-rpc calls are based on the Satoshi rpc interface and are as follows:

getbalance
getnewaddress
getreceivedbyaddress
sendtoaddress

This looks very interesting. Combining this with bitcoinmonitor.net url callbacks (using the API) will provide you a complete solution to accept payments without the need to put your hot wallet online anywhere and without the hassle to set up/maintain a long list of pre-generated addresses with your bitcoinmonitor.net agent. Awesome!  Cool
As soon as I find some time I will set this scenario up for one of my sites.
legendary
Activity: 1428
Merit: 1093
Core Armory Developer
Clarifications required:

- Are incoming transactions which eventually do not become part of the blockchain dealt with correctly? I am unsure how / unable to test this.

- What happens if multiple transactions are made which would result in the balance being less than zero? Presumably when it comes to broadcasting the last of the signed transactions there will be an error from the Armory client which is performing the signing/broadcasting. Should tracking of the likely future balance in the daemon be enforced even though the transactions have not yet been broadcast and maybe never will be? How should this be managed, if at all?

-  There is full re-org/double-spend handling, which has been tested at the library level.   But it hasn't been tested at the GUI/interface level, because I never set up a way to test block/tx injection over the networking interface.  I see re-orgs happen all the time -- you'll see output to the console that says "Invalidating old chain tx -- Marking new chain tx valid".  For web-servers, this should be all you need. 

- Armory is pretty dumb when it comes to... lots of network-stuff.  It was written with the assumption that Bitcoin-Qt is going to feed it trustworthy, reasonable data.  I have noticed, under strange testing conditions, if a conflicting tx happens to make it in, it will show up in the ledger but it actually won't affect your balance at all.  Again, this is not all that relevant for web-servers, but it's worth noting:  Armory receives the conflicting tx, and immediately detects it's relevant, and adds it to the ledger on the GUI.  Then Armory tries to add it to it's pool of tx for each address, etc, but finds those outputs have already been spent.  So it gets dropped, and the underlying UTXO list is untouched.  It rarely ever matters, because Bitcoin-Qt/bitcoind can't pass it conflicting tx...

So I need to fix the auto-display-in-ledger bug.  But balances and spendability should maintain correctness in this situation.  Please let me know if they appear not to.

As for storing data between loads:  some people complain about the load times, etc, but it comes with a tremendous benefit of robustness, because there's no way for Armory to ever be "out of sync" with the blockchain.  One issue with Bitcoin-Qt is that it can mark outputs as spent, or believe that some transaction succeeded that wasn't accepted and get stuck in a state that isn't consistent with the rest of the network.  Armory avoids all this by having no memory.  At the expense of load times.

This functionality will eventually have to be added to do this (especially with the blockchain size getting so big), but right now it's not there, and it will probably require a lot of testing.  If the server is going to process lots and lots of transactions separately, you might consider taking my examples above for creating transactions, and add that memory/tracking around it (maintain the tracking yourself).  Maybe that's too much work...  I don't know.




Btw, I haven't looked at your stuff yet, but this sounds fantastic!  Thanks so much for doing this, and the JSON-RPC interface is something I've wanted to do for a while.  Perhaps we'll find a way to merge functionality together...
mav
full member
Activity: 169
Merit: 107
The simple Armory Daemon is now on github. It aims to be as close to a direct replacement for the Satoshi interface as possible.

https://github.com/thedawnrider/BitcoinArmory-Daemon

Available json-rpc calls are based on the Satoshi rpc interface and are as follows:

getbalance
getnewaddress
getreceivedbyaddress
sendtoaddress

getbalance:
Returns a decimal value in BTC for the total remaining balance in the wallet.

getnewaddress:
Returns the next address in the wallet as a string.

getreceivedbyaddress:
Returns a decimal value in BTC for the amount received by the address.

sendtoaddress:
Returns an unsigned transaction as a string. Implementation of signing and broadcasting is left to the client.



Features that may be included in the future:
User authentication
SSL
More API methods as required (the current set fills my needs). listtransactions is one method I am looking to include in the future.
Please suggest anything else you personally may find useful, as I have designed this only to fill my specific needs.



Clarifications required:

- Are incoming transactions which eventually do not become part of the blockchain dealt with correctly? I am unsure how / unable to test this.

- What happens if multiple transactions are made which would result in the balance being less than zero? Presumably when it comes to broadcasting the last of the signed transactions there will be an error from the Armory client which is performing the signing/broadcasting. Should tracking of the likely future balance in the daemon be enforced even though the transactions have not yet been broadcast and maybe never will be? How should this be managed, if at all?
hero member
Activity: 560
Merit: 500
I am the one who knocks
That's true. The thing is though, that I imagine IP connectivity opens up a lot more potential attack surfaces than a USB connection. I mean, any program on the key box listening for incoming connections will suddenly be an attack surface. And it's really hard to control which programs do this; a lot will. With the device acting as a USB client this won't be an issue. You can't - not as far as I know at least - "listen" for connections via USB, and so, there will only be one central point - one attack surface - that we need to secure, which is the USB driver and the program that communicates over this connection. If this is secure, the key box should be secure.

I see you point, but a slim install and a firewall would be all you need.  A lockdown of 'only allow connection to port XYZ from host ABC' would be sufficnt.
legendary
Activity: 980
Merit: 1008
That's true. The thing is though, that I imagine IP connectivity opens up a lot more potential attack surfaces than a USB connection. I mean, any program on the key box listening for incoming connections will suddenly be an attack surface. And it's really hard to control which programs do this; a lot will. With the device acting as a USB client this won't be an issue. You can't - not as far as I know at least - "listen" for connections via USB, and so, there will only be one central point - one attack surface - that we need to secure, which is the USB driver and the program that communicates over this connection. If this is secure, the key box should be secure.
hero member
Activity: 560
Merit: 500
I am the one who knocks
I would like to point out that you do not have to use USB. Ethernet would actually work fine for this. Just have to cards in the webserver; one for Internet and the other for the key-box.

Then you can just set static addresses (or even run a dhcp server on the keybof, but that increases your attack surface).

Anyway you achieve the same goal. There is only one connection to the keybox and it is not on the interwebz.
legendary
Activity: 980
Merit: 1008
Another possible web server setup that is possible with Armory (that I've actually thought of setting up myself), is having the web server connect via USB to a small Linux ARM device (just something cheap) that has a USB OTG port (allowing the device to act as a USB client). This USB client device stores the private keys and only responds to request over USB. With this setup, it's possible to simply send transaction requests to the web server that are signed with a certain private key, let the web server forward the request to the device, and then let the device - connected only via USB and not connected to the Internet - verify that request with the corresponding public key, and only return a signed transaction over USB if the signature on the request can be verified with the public key on the device.

This should - if implemented correctly - make the wallet unhackable. The only thing an attacker can do - even if he gains full access to the web server - is send request over the USB connection to the device that hold the keys, but without the proper private key the device will ignore the requests.

This could be useful for use cases such as an exchange needing to transfer coins from its cold storage wallet to it's online wallet. The exchange owner simply has to make sure that the private key used to sign requests is inaccessible to attackers.
Another use case could be a company exchanging fiat currency for bitcoins. Here the bank is in possession of the private key. It promises to sign all incoming wire transfers with this private key. If the wire transfer - in a comments-field of some sort - includes a receiving address, it's possible to let someone wire a certain amount of fiat currency to the business owners account, and let the business send out coins to that person - completely securely - given that the customer includes the send-to address in the wire transfer. If the bank can keep the private key secure, this - again - prevents an attacker from being able to do anything were he to gain full control of the web server.

I like the hardware idea, except I don't understand where the private key & signature comes from?  It's not stored on the online system, is it?
The private key (let's call it the master private key) would basically belong to the person who needs this setup (web server plus key-storage box (the latter having the private keys for the wallet (lets call these wallet private keys))) to produce and send out valid transactions, spending coins from the wallet(s) on the key-storing box. The key-storing box would then have the corresponding "master" public key, and it will only sign a transaction sent to it over the USB connection if it is signed using the master private key.

My initial use case was, as mentioned, me selling bitcoins for fiat currency. I have a site that informs the user to send an amount of fiat currency to my bank account, and include the address - to which the bitcoins are to be sent - in the "comments" field, when sending the wire transfer via his or her e-banking site. The bank I would use to provide my account would have the master private key. Whenever a new bank transfer to my account takes place, the bank sends me an email containing information on the transfer - including the destination bitcoin-address that the buyer included in the "comments" field - signed with the master private key. Then it's simply a matter of having the online web server read new incoming mails from the bank, and pass on the content of the mails to the key-storage box. The key-storage box checks that the wire transfer message is signed with the master private key, and if it is, constructs a transaction that sends an amount of bitcoins to the address specified in the wire transfer message. It passes this to the web server which publishes it. All automated, and - as far as I can see - unhackable, provided the software on the key storage box is trusted.
The only thing that struck me while writing this, is that we need some way of telling the storage box the amount to send. This can't really be included in the signed wire transfer message, as the customer shouldn't be in charge of this. It seems this would be an unsafe operation. Ie. a customer could place an order, hack into the web server, and change the amount being sent to his address, since the storage box can't know the bitcoin exchange rate, since it isn't online. I guess we'd need Mt. Gox to sign a piece of data containing the current time and exchange rate, so the box can know how many coins to send.

Quote
Why do you need any signing?  Why not just use that device to facilitate passing ASCII-only between the two systems?  Any non-ASCII buffers are rejected, and the offline computer will do nothing except collect the data and wait for the user to act on the received data.  Really, it would just behave like a serial port.
Are you imagining three devices? I only imagine two: the web server and the device. The device stores the keys, and does the signing - provided it receives a properly signed message that tells it where to send the coins.

Quote
Damnit!  I wish I could make the serial port idea work...
You and me both! I think I'll be looking into getting the Raspberry Pi to act as a USB client. It would then be necessary to use the USB protocol, which is a lot more complicated than just serial, but would not - as far as I know - allow login to the offline machine.

EDIT: I just discovered that the Raspberry Pi Model A (as opposed to Model B that is out now) will support acting as a USB device/client, because it uses a different communication-chip than Model B. Model A doesn't have an Ethernet port, and only has a single USB port, but I figure this might even be an advantage, since after the software has been installed on the device (using, for example, some USB-to-Ethernet type device to update packages), the Ethernet port will have to be removed because the USB connection needs to be used to communicate with the online computer. Perhaps one will be less tempted - and feel safer - if the device doesn't even have a port to plug in an Ethernet cable.
Additionally, it costs only $25. That's really not much for the ultimate in safe Bitcoin wallets. That being said, Model A isn't out now. And USB client mode won't necessarily work out-of-the-box. This is a software issue, as far as I can gather. Not OS level, I think, but firmware level - so it might not be that easy to implement. But still, looks like a device is coming to the market that fulfils the requirements of an offline hardware wallet. Off the top of my head, I can't see what else one would need other than a Raspberry Pi Model A, and perhaps a USB cable. But most people have USB cables lying around. Sure beats that $40 2xUSB-to-serial-plus-null-modem solution.
legendary
Activity: 1428
Merit: 1093
Core Armory Developer
Another possible web server setup that is possible with Armory (that I've actually thought of setting up myself), is having the web server connect via USB to a small Linux ARM device (just something cheap) that has a USB OTG port (allowing the device to act as a USB client). This USB client device stores the private keys and only responds to request over USB. With this setup, it's possible to simply send transaction requests to the web server that are signed with a certain private key, let the web server forward the request to the device, and then let the device - connected only via USB and not connected to the Internet - verify that request with the corresponding public key, and only return a signed transaction over USB if the signature on the request can be verified with the public key on the device.

This should - if implemented correctly - make the wallet unhackable. The only thing an attacker can do - even if he gains full access to the web server - is send request over the USB connection to the device that hold the keys, but without the proper private key the device will ignore the requests.

This could be useful for use cases such as an exchange needing to transfer coins from its cold storage wallet to it's online wallet. The exchange owner simply has to make sure that the private key used to sign requests is inaccessible to attackers.
Another use case could be a company exchanging fiat currency for bitcoins. Here the bank is in possession of the private key. It promises to sign all incoming wire transfers with this private key. If the wire transfer - in a comments-field of some sort - includes a receiving address, it's possible to let someone wire a certain amount of fiat currency to the business owners account, and let the business send out coins to that person - completely securely - given that the customer includes the send-to address in the wire transfer. If the bank can keep the private key secure, this - again - prevents an attacker from being able to do anything were he to gain full control of the web server.

This seems like a good idea to post on the Improving Offline Wallets Thread.

I like the hardware idea, except I don't understand where the private key & signature comes from?  It's not stored on the online system, is it? 

Why do you need any signing?  Why not just use that device to facilitate passing ASCII-only between the two systems?  Any non-ASCII buffers are rejected, and the offline computer will do nothing except collect the data and wait for the user to act on the received data.  Really, it would just behave like a serial port.

Damnit!  I wish I could make the serial port idea work...
legendary
Activity: 980
Merit: 1008
Another possible web server setup that is possible with Armory (that I've actually thought of setting up myself), is having the web server connect via USB to a small Linux ARM device (just something cheap) that has a USB OTG port (allowing the device to act as a USB client). This USB client device stores the private keys and only responds to request over USB. With this setup, it's possible to simply send transaction requests to the web server that are signed with a certain private key, let the web server forward the request to the device, and then let the device - connected only via USB and not connected to the Internet - verify that request with the corresponding public key, and only return a signed transaction over USB if the signature on the request can be verified with the public key on the device.

This should - if implemented correctly - make the wallet unhackable. The only thing an attacker can do - even if he gains full access to the web server - is send request over the USB connection to the device that hold the keys, but without the proper private key the device will ignore the requests.

This could be useful for use cases such as an exchange needing to transfer coins from its cold storage wallet to it's online wallet. The exchange owner simply has to make sure that the private key used to sign requests is inaccessible to attackers.
Another use case could be a company exchanging fiat currency for bitcoins. Here the bank is in possession of the private key. It promises to sign all incoming wire transfers with this private key. If the wire transfer - in a comments-field of some sort - includes a receiving address, it's possible to let someone wire a certain amount of fiat currency to the business owners account, and let the business send out coins to that person - completely securely - given that the customer includes the send-to address in the wire transfer. If the bank can keep the private key secure, this - again - prevents an attacker from being able to do anything were he to gain full control of the web server.
legendary
Activity: 1428
Merit: 1093
Core Armory Developer
I mentioned in my original post about this that this part is not trivial.  It's because selecting coins for a transaction is not a one-shot process -- you have to declare what your desired output amount is and your desired fee, then Armory will create the best transaction it can based on those parameters.  If the transaction has properties that require a bigger fee, then you need to re-select coins with the new fee.  Theoretically, this should be part of a while-loop or something (in case the new selection requires an even-bigger fee), but I designed the SelectCoins algorithm to use a selection that leaves some margin (if possible) so that a higher fee can be included without changing the coin selection.

The functions for reference are:
validateInputsGetTxDP() at qtdialogs.py:4867,
createTxAndBroadcast() at qtdialogs.py:4811
(Maybe)broadcastTransaction() in ArmoryQt.py:1889).  
PySelctCoins() in armoryengine.py:5049.  

There's a lot of code here, because it is also validating the data was entered properly on the Send-Tx dialog.  But what you need is there.  I'll try to summarize it here.

PySelectCoins arguments:
  • unspentTxOutInfo: a list of unspent TxOuts, retrieved via wallet.getTxOutList("Spendable") (qtdialogs.py:4968)
  • targetOutVal: total amount to send to (all) recipients(s), in satoshis  (i.e. 10.0*ONE_BTC for 10BTC)
  • minFee: you can always set this to 0 the first time, and re-call it with higher if the output suggests more
  • numRand (optional): number of randomized SelectCoins solutions to try (more about this in the appendix, below)
  • margin (optional): how much margin you'd like to build into the selection, for the reasons stated above

Actually executing the transaction -- you must get all the spendable coins, make a coin selection, then calculate the change and generate the recipient list (you will need to wallet.getNextUnusedAddress() generate a change address, or you can do something else).  Shuffle the list.   Then create the "PyTxDistProposal" -- this is an intermediate step where the transaction is constructed as a BIP 0010 packet, in case it needs to be signed by an offline computer.  In the case that you have the full wallet -- this TxDP will be signed immediately, then converted to a standard transaction and broadcast.

And finally, here's the code sample you've been waiting for:  Create one transaction from a wallet, to send 10.0 BTC to one recipient

Code:
      # Get unspent TxOutList and select the coins
      totalSend, fee = long(10.0 * ONE_BTC), 0
      spendBal = wallet.getBalance('Spendable')
      utxoList = wallet.getTxOutList('Spendable')
      utxoSelect = PySelectCoins(utxoList, totalSend, fee)

      # Check the minimum fee for the coin selection
      minFeeRec = calcMinSuggestedFees(utxoSelect, totalSend, fee)[1]
      if fee           if totalSend + minFeeRec > spendBal:
               raise NotEnoughCoinsError, "You can't afford the fee!"
           utxoSelect = PySelectCoins(utxoList, totalSend, minFeeRec)
           fee = minFeeRec

      if len(utxoSelect)==0:
         raise CoinSelectError, "Somehow, coin selection failed.  This shouldn't happen"

      # We have a good coinselection, now compute change and create recip list
      totalSelected = sum([u.getValue() for u in utxoSelect])
      totalChange = totalSelected - (totalSend  + fee)

      outputPairs = []
      outputPairs.append( [addr160_recipient, totalSend] )
      if totalChange > 0:
         outputPairs.append( [wallet.getNextUnusedAddress().getAddr160(), totalChange] )

      # Shuffle the list, create the TxDP
      random.shuffle(outputPairs)
      txdp = PyTxDistProposal().createFromTxOutSelection(utxoSelect, outputPairs)
      
      # Sign the TxDP
      wallet.signTxDistProposal(txdp)
      finalTx = txdp.prepareFinalTx()
      wallet.setComment(finalTx.getHash(), "This was the example tx created by the python script")
      finalTx.pprint()
      print "Hexified Tx: ", binary_to_hex(finalTx.serialize())
      
      # Broadcast the transaction
      ...  (will fill this part in later)
      

I suppose I could encapsulate much of this code in a more friendly way.  On the other hand, you can pretty much copy this code and make the appropriate adjustments, and you'll get a signed transaction.  You can modify it for multiple recipients, handling fees differently (but you should trust the calcMinSuggestedFee[1] value), and saving off copies of the transaction before sending.

I will include more about actually broadcasting the transaction, in a bit.  The problem is that you need to start the networking engine in order to actually broadcast it.  On the other hand, you have a PyTx object (finalTx), so you can call binary_to_hex(finalTx.serialize()) to get the raw hex version of it, which you might be able to broadcast in other ways.  (like "sendrawtransaction" RPC call with bitcoind)

I know you'll have questions.  Fire away!



Appendix: The way PySelectCoins works:  a bunch of deterministic coin-selections (solutions to the pseudo-knapsack problem) are generated based on a variety of standard coin pools (the types and sizes of unspent TxOuts for your wallet).  A few random solutions are thrown on top to generate some variety.  Usually, the deterministic algorithms find the best answer, but for strange coin-pool make-ups, some semi-randomized selections will produce decent answers.  In all, we get about a couple dozen possible solutions, and then each one is evaluated for "fitness" ... how big is the fee?  how many addresses are combine?  how obvious is it which output is the recipient and which one is change?  The relative weightings of each of these is defined in armoryengine.py:5000 -- the WEIGHTS variable defines how much relative weighting to use for each factor.  There's six of them, though it really could've been reduced to 3 important ones:  tx-fee/allow-free, input-anonymity, and output-anonymity.  

Code:
WEIGHTS[IDX_ALLOWFREE]  =  100000
WEIGHTS[IDX_NOZEROCONF] = 1000000  
WEIGHTS[IDX_PRIORITY]   =      50
WEIGHTS[IDX_NUMADDR]    =  100000
WEIGHTS[IDX_TXSIZE]     =     100
WEIGHTS[IDX_OUTANONYM]  =      30

You can manually modify these weightings, or find a way to inject a new weights matrix into the PyEvalCoinSelect (I apologize, I didn't make that easy, you might be better to just manually change armoryengine.py, for now).  What do these mean?  The default behavior is to avoid using zero-conf coins at all costs.  Any solution with no zero-conf coins will be scored higher than any solution with zero-conf.  The next two biggest weights are "ALLOWFREE" and "NUMADDR".  This means that equal weighting is given to "please give me a transaction that can be free" and "please use as few addresses as possible on the input side".   "TXSIZE" and "PRIORITY" are next, but probably shouldn't even be there, because they are already tied into the ALLOWFREE and NOZEROCONF scores.  

Finally, OUTANONYM is the last critieria.  Given a set of coin-selects that are basically all the same for the other 5 criteria, it will select the solution that has better output-anonymity.  i.e. -- if the outputs are 10.0 and 1.3482023, it's fairly obvious to someone observing the blockchain which one is the recipient, which is the change-back-to-self (1.3482023).  However, if a solution is found that gives 11.34 and 5.11 -- it's not so obvious anymore, and that solution will get a higher score.  It gets extra credit if it's deceptive:  if the recipient is getting 1.384 and it finds a solution that has a change of 8.1:  then most would observers would make the wrong conclusion.

The point of this description was not only to describe the weightings, but suggest you change them for your own usage.  I had planned to make this customizable in Armory, I just hadn't gotten around to it.  If all you care about is input-anonymity and output-anonymity, I suggest changing NUMADDR and OUTANONYM to high numbers, and change the other 4 to something tiny.  You might pay a fee when you otherwise wouldn't have to, but it will be more anonymous.

DISCLAIMER:  I make no guarantees about transaction anonymity at all.  Use of this feature is completely at your own risk.  This is implemented solely for the purposes of declaring a rough ordering for an under-defined set of solutions.  Setting any particular weight to 1010 and others to 0 does not mean you are guaranteed a "good" solution -- there may only be one solution.
mav
full member
Activity: 169
Merit: 107
So i can basically create as many receiving addresses on the webserver as I want, but without any risk in case the server gets compromised? Awesome!

This functionality is very simple to achieve and is outlined in the first post thanks to etotheipi providing the example. This speaks volumes about how nicely armory is designed, it's very easy to work with. I have started on an api interface which currently is able to get the next address in the watch-only wallet.

I have had difficulty in achieving my next two goals with armory which is to get the balance of an address, and to make a transaction for signing by the offline wallet. Unlike generating the next address, both of these require the BDM to be loaded (ie the blockchain to be buffered), which I have been able to achieve. However, implementing the functionality I desire has not yet happened, I have had too many other things which have taken priority, and I have not looked closely enough at the source to realise these goals.

If anyone is able to make this extra functionality work I would love to hear how you did it. I have written a very basic foundation which allows interface to armory wallets, however it's so basic that I have not yet uploaded it for public consumption. The project deserves a more complete implementation than what I have started. I don't seem to recall anything being in the example file about generating transactions.

Delay in upload is also due to considering whether to implement a satoshi-clone api or to create an api specific to armory. Any suggestions on this are also welcome.
legendary
Activity: 1428
Merit: 1093
Core Armory Developer
So i can basically create as many receiving addresses on the webserver as I want, but without any risk in case the server gets compromised? Awesome!

Exactly.

I recommend using an offline computer to make the full wallet, then make a watching-only copy.  Put it on your webserver, and also import it into Armory on your primary system.  Then your webserver will distribute addresses, and you can see the full transaction history from your desktop.  If someone gets the watching-only wallet... well all they can do is watch, but they can't take any funds!

I don't recommend generating addresses from the watching-only wallet, while the webserver is doing the same.  There won't be any fireworks, but you might end up re-using an address by accident.  If you want to also be able to distribute addresses from your primary system, I recommend making a second wallet on the offline computer just for that purpose.  You can watch them both from Armory, but only distribute addresses from the one the server isn't using.  Then you can also execute offline transactions from either wallet using the online computer.
hero member
Activity: 488
Merit: 500
So i can basically create as many receiving addresses on the webserver as I want, but without any risk in case the server gets compromised? Awesome!
legendary
Activity: 1428
Merit: 1093
Core Armory Developer
I have been looking for this information on the Armory website (like a developer API section), but I'm sure it will be up there at some point.

Thanks for this! It will be highly useful.

That kind of documentation has not made it into my priority list, yet.  However, threads like this are specifically for users to share ideas about how they successfully leveraged armoryengine for their purposes.  And while I don't have time to make such documentation (yet), I will respond to such threads and help folks accomplish their goals. 

Also, if you checkout the git project, there is a file called "extras/sample_armory_code.py" which has a lot of example code.  It's not as good as actual documentation, but working example code is quite useful.
hero member
Activity: 548
Merit: 502
So much code.
I have been looking for this information on the Armory website (like a developer API section), but I'm sure it will be up there at some point.

Thanks for this! It will be highly useful.
Pages:
Jump to: