Topic: New wallet file ideas (Read 2631 times)

Is it?  What if you want your watching-only wallet encrypted but you don't have disk encryption setup?  What if you don't know how to setup disk encryption?  Wouldn't the same argument apply to the private keys, too?

Tell me if I'm wrong, but I think there's another key advantage (but I don't know much about the way these things are implemented).  If you use FS encryption, then once you've booted and your OS has requested and unlocked the filesystem encryption, then anyone who gains access to your system while it is running will be able to read that file, even if they got access to your system remotely (the OS will decrypt for them).  But if it's done at the application level, then the file will always be encrypted, and the only way for them to get the info will be to pull it out of your memory space.  I'm pretty sure that is "difficult."  But I don't know for sure...

That sounds like duplicating operating systems existing support for disk encryption.

Another idea was just proposed to me, which I really like.  Out of scope for the first two ideas I presented, but completely relevant for a new wallet design:

Encrypted watching-only wallets

Some users have expressed concern that, although their BTC is safe if their online laptop is stolen because they keep their coins offline, the person that stole their laptop now knows their identity and the fact that they own $50k in BTC.  It would be best if the thief had no information at all...

As such, I'm thinking I could add an extra layer of encryption, to encrypt the public keys, addresses and transaction IDs stored in the wallet.  This "outer" encryption would use the same encryption scheme as the one used for private keys, but it would only have to be entered once when the app is started and would have no timeout .  This means that the data is always stored on disk encrypted, and the decrypted version only exists in RAM.

It would also be nice if all such wallets (per-computer?) had the same encryption.  So if you have multiple wallets, you still only type your passphrase once.  I can see it getting out of hand when you have multiple devices with different addresses, and then start juggling wallets around.  But there's only so much I can do about that...

The answer is:  you just seeded an excellent discussion about the far future of Bitcoin escrow/contracts, but I think that future hasn't arrived yet.  And when it does arrive, I'd love to see that discussed and developed into an inter-client standard.  Until then, the M-of-N cases are dramatically simpler, and can actually be used right now, thus it warrants a separate [simpler] treatment   (plus, there will probably be some lessons learned from M-of-N execution to be applied to the more-general case)

That works for me and am satisfied just by that you understand what I propose well enough to decide whether it's appropriate for inclusion in your client, even if the answer is no.

@Casascius,

I recognize that you are proposing something of ultimate flexibility and versatility.  And I actually really like the idea for arbitrarily complex transaction types.  But M-of-N transactions are a degenerate case of that large space of possible transaction types.  Combined with the fact it will be spectacularly more-common than those other cases, I think it's not unreasonable to have a separate "shortcut" method for setting them up.  

However, I do like where your idea is going, and worth some further discussion.  I'd really like to have a way to document and execute these crazy transaction types, I just think a fully developed solution for that is out-of-scope for now.  I think I will leave room in my design to accommodate "relationship" entries in the wallet.  When it is fully developed, I can switch all multi-sig tx over to that...

Though, why can't such relationships be developed by a central organizer?  Rather than distributing the relationship object and letting each person pick a place... there is at least one person that already knows how it should be setup.  Have everyone send that person a public key/wallet, and he will set it all up.  Perhaps the final result can be signed by the private keys of all included public keys, or something like that.

But for now, I think it makes sense to shortcut the couple degenerate cases of 1-of-2, 2-of-2, 2-of-3, 3-of-3, and handle all the details automatically for the user.  A user simply wants the transaction to require 2 of 3 signatures, and has his own private chain and the other two public chains.  He doesn't care about the internal ordering of keys, but he does require everyone to use the same ordering.  He also wants to avoid address collisions in the event that multiple parties try to create transactions at the same time with the same ordering.  I think this technique resolves all of that, and can be represented quite simply to the user.

---

The issue with look-ahead in deterministic wallets is an annoying problem.  Most regular users, really don't need more than 10 addresses lookahead, 100 to be safe.  However, e-commerce users/merchants might need significantly more lookahead than that.  For that reason, I defaulted to 100 address lookahead with Armory, and added a button to let users extend the lookahead as far as they want.  I expect that any business users interested in this application, will have the knowledge to know which knobs to tweak -- i.e: set lookahead to 10,000 addresses & disable second chain.  

Therefore, Armory users using two-factor auth as I have described, only have 100 extra addresses in their wallet.  One time.  If the second chain is never used, then it never increases.  The way Armory scans the blockchain, search time is O(log N) in the number of addresses in the wallet (because they're stored in a BST in RAM, so that's how long it takes to check if an address is in your wallet).  As such, 100 extra addresses is trivial, both in HDD space, and extra computation time.  If the user needs more than 100 lookahead, they are also likely knowledgeable enough to know what options to change for their application.

This.

For occasional interactions, I think that each party should just accept the burden of keeping track of everything they need.  For p2sh, it means keeping a copy of the completed redeemScript (which generally means having all of the pubkeys).  The whole point of using pubkeys is that they can be public, so the service that collects keys can also distribute them when done.

For regular interactions, and particularly when you want to define a stream, each party should generate a new set (seed and chain secrets), and then the problem is reduced a bit to merely passing around names and indexes.  If ABCDEFGHIJK are keys in some complicated scheme, they can be taken to mean AiBiCiDiEiFiGiHiJiKi, when given i, so everyone knows how to generate their part of it.

By the way, for regular X-of-Y multisig, I find it VERY useful to sort the pubkeys before assembly.  In fact, I wish that the default behaviors of createmultisig and addmultisigaddress in the stock client were to sort, possibly with an option to not-sort.

With what I had in mind, you wouldn't be restricted to any specific pattern of multisig transactions, the data structure could support any kind of multisig relationship imaginable.  You would still restrict users' ability to create such relationships to the ones you consider in scope of course, but the file format itself would accept any.

Here is how I imagine a "relationship" data structure, let's do something weird: (A AND B) OR (C AND D AND E) OR (F AND A) OR ANY5OF (A,B,C,D,E,F), and C,D,E are the ones who can issue addresses.

This structure would originally be created by A, but with "unknowns" for the public keys for B,C,D,E,F, and serialized into a block of text.  That block of text would be sent to the five other parties, who would recognize "OK, this is a multisig proposal with vacancies, which one should I join? First Officer, Flight Crew, or Air Marshal?".  The clients of those parties would emit a record proposing to join the position they chose.

When all six parties have their own record, plus the five of the other parties, they should all recognize the group relationship and be able to start participating in it.


At some point, one of the parties must take the initiative and decide "I'm creating a multisig wallet relationship, and the style will be 2-of-3".  I am imagining the process similar to the CSR request for SSL certificates.  At some point, when setting up SSL on a website, one party (web site operator) initiates the deal generating a CSR that says "I'm so-and-so, here is my public key, and I propose the creation of a certificate that looks like x".  The other parties to the deal accept the proposal and respond with "Here is the certificate you asked for".  Someone always has to go first, and there is nothing wrong with tying special responsibilities - like defining the proposed relationship - to the party who does.

At some point, all three parties must exchange some record so that they learn the public chain codes of the other parties.  If they didn't do this, they would never be able to recognize incoming transactions belonging to the relationship.  I see this step as unavoidable.

However, I do see a possibility that several of the roles could be equivalent.  My example above has three "Flight Attendants".  It could be defined that these positions are equivalent in purpose, and that which specific relationship member gets spot C, D, or E, is dictated by numerical order of a hash, rather than forcing the user to pick a spot deliberately hoping another member didn't pick the same one.

At some point, all three parties software must have the ability to express to the human user that they in fact all have the same parameters and keys in mind.  I propose they do this by emitting a hash of the serialized relationship record containing all the public keys in the relationship.  When the clients of all parties to the relationship say "Relationship Successfully Formed: Relationship ID F1175A23", the users can know that all parties are on the same page, and that (for example) party C didn't accidentally use a key that came from somewhere else.



I am not sure I understand how the resource usage is small.  I understand a chain can be used to generate unlimited addresses.  How should the software come to know that of all the addresses generated by party B (for example), the one with sequence number 10,000 has a transaction that needs to be checked for, even if all addresses 0 thru 9999 don't?  I see no way other than actually generating all those addresses just in case and checking them all, and then there is no guarantee that 10000 or any other number is enough if you can't assume how the other party will be using addresses they generate.  While such sparse utilization seems counterintuitive on the surface, it's not all that unusual if you consider that someone setting up an e-commerce web server might pre-generate a huge number of addresses assuming most will be wasted, and then give them out in an arbitrary order (e.g. alphabetical sort instead of the order in which they were actually generated).  If my assumption that the only way for party A to truly know if party B has never received funds at addresses he has issue is to perform an operation that is unbounded in resource usage, it seems prudent to clip it from the start if the use case makes it clearly unlikely to be used.  This of course discounts your assertion that "the resource usage of watching for a few addresses is trivial" - if I have overlooked something and this assertion is guaranteed to always be true for a reason I haven't considered, then yes I agree that it would be a waste to bother with an option that disables address generation roles for parties.

I already have an "extraAttributes" field which is intended to accommodate non-Armory addresses and address chains.  I don't know who else would be leveraging the code base, but at least it gives me room to, say, import and maintain an Electrum-deterministic wallet chain and store any necessary extra information in that field that doesn't fit into the Armory-specific structure.   I'm not sure of exact implementation details, yet, but I do want to try to leave room for things I haven't thought about, yet.  I suppose colored-coin attributes could go there, too.

While you're doing a rewrite, you may consider adding support for output "attributes".
E.g. projects like the colored bitcoins would vastly benefit from code like that (https://bitcointalksearch.org/topic/chromawallet-colored-coins-issue-and-trade-private-currenciesstocksbonds-106373), because their algorithms needs to keep track of the color state of individual outputs.

My understanding is that we don't have real capability to do ((A and B) or C) yet (or at least, it's not standard), and that only M-of-N multisig is really enabled.  I know you could execute any arbitrary script if you find a miner to do it for you, but that's not in scope for me -- I'm sticking with transactions that will be accepted by the network without special actions.  (I just confirmed with Gavin that non-M-of-N transactions are not standard; redeeming such a P2SH script would require a miner's help)

However, I agree that the design here should be extensible to these cases, when they become available in the future.  But it also seems unnecessary to require extra user-interaction for regular M-of-N transactions where the specific ordering is actually irrelevant to the users, as long as it is deterministic and accessible to all participants.    As a user entering a 2-of-3 wallet with two other parties, I don't actually care what order they go in, and asking the user to specify a totally arbitrary ordering is not only confusing, but ripe for people to do it wrong -- i.e. user A chooses {A,B,C}, but user B accidentally sets up their wallet with {B,A,C}, and then things get all out of whack...

Also, I don't see a reason why there should be a special case for desktop+second-factor (smartphone) vs shared-spouse-wallets.  Just have all 2-of-2 wallets pretend that both chains could be used, because the resource usage of watching for a few extra addresses is trivial.  If it's really a concern, there can be an option in "Expert" usermode to disable the second device chain if you know it won't be used.  

With regard to ordering of wallets I never imagined this to be a problem. Mainly because in my mind, somebody would be proposing the relationship with a specific number of vacancies which may or may not be equivalent in purpose. If A were to propose (A AND B) OR C, someone entering that relationship needs to explicitly be choosing slot B or slot C, there would be no ambiguity as to what party is in what slot and they cannot just simply take the first available slot.  Further, I imagine that when the relationship was defined, each of the roles would either be allowed or not allowed to generate addresses. (If B is just a smartphone validating A's transactions, then it would be a waste of resources to assign a chain for B it will never use, but which A will have to walk up and down that chain looking for transactions at great CPU expense just in case it somehow did).

If I use those assumptions, then the field containing the list of parties able to generate addresses would dictate a fixed order and also dictate how to manage the chains so that all parties know where they generate from, assuming they are set to be able to.

I have more reading to do on the contracts and chains and how they work, so it is probable that my assumptions reflect a deficient understanding of those.

Ahh, so it does make sense to use P2SH for paired/linked wallets, and regular multi-sig for escrow tx, etc.  For some reason, I was under the impression that P2SH was intended for everything.  I do prefer plain multi-sig for those escrow tx.

@Mike,

I'm not sure the payment protocol changes this much.  It means that the decision about which form to use will be irrelevant to those using the payment protocol, but there will still be users who will do it the "old" way, which should still be accommodated.  Though it sounds like we already have a consensus ... it makes sense to use P2SH for linked wallets, and regular multi-sig for isolated escrows and contracts.

---

@Casascius:  I was just about to post an expanded idea for linked wallets, but I think you already did it.  I'll restate it, though, for clarity, and maybe add a little.

Let's say I make wallet A/j, and my spouse creates wallet B/j.  We exchange the root of the external/internal chains.  Meaning, we exchange the data needed to not just generate one chain of addresses, but both the primary chain and change chain.  

So, I give her A'/j and she gives me B'/j.  All addresses generated by my wallet will be of the form 2of2([A'/j/0/x, B'/j/0/x]) and all change will go to 2of2([A'/j/1/y, B'/j/1/y]) -- external chain is 0, internal chain is 1.  The order will be switched for her wallet:  she will create addresses of the form 2of2([B'/j/0/x, A'/j/0/x]) and change will go to 2of2([B'/j/1/y, A'/j/1/y]).  So each wallet will only generate addresses from its own two chains, but watching for addresses on all four chains.  

I'm pretty sure that's very similar to what you suggested.  


EDIT: to avoid ordering issues for 3+ wallets, the addresses should be ordered by wallet fingerprint order, and each wallet always generates addresses starting with their own in that list, wrapping around.  So if the wallet fingerprints end up ordering the wallets  B, A, C,  then all 2-of-3 addresses generated by wallet B will be {B,A,C}, wallet A will be {A,C,B} and wallet C will be {C,B,A}.  That makes sure there are only N possible address orderings (one per party), instead of N-factorial

You can use plain multisig if you like, they WERE added to IsStandard.

Alan, did you look at the payment protocol discussions happening on bitcoin-development? Some of it may be relevant to this work.

Re: P2SH/normal, I agree with Mike that P2SH is not always the right tool for the job. It was designed on the assumption that people will want to use short addresses and that may well turn out to be true, but if we successfully move most payments to use a payment protocol (as we're going to try) then addresses will start to fade away as a user-visible part of Bitcoin and that rationale goes away. It leaves P2SH being purely a UTXO set size optimization, which like any optimization must be weighed against the additional complexity it introduces. If it's going to make the user experience worse, it's better to just use regular scripts.

Mike, escrow is indeed meant to work as you describe but using 2-of-3 multisig which boils down to (A AND E) or (B AND E) or (A AND B):

  https://en.bitcoin.it/wiki/Contracts#Example_2:_Escrow_and_dispute_mediation

A think a lot of these things become easier to reason about when you assume a payment protocol.

Agreed - I don't mean the user shouldn't have to do backups, I just mean we should reduce the number of transaction types that require the user to make an immediate backup or else risk a financial loss in the event of a data loss... the whole purpose of having the deterministic key chains in the first place, and what I understood a core element of the original problem to be as described in the OP.

Another thing that occurred to me, is if Eddie is in the professional escrow business, P2SH could still be used, with Eddie having primary responsibility for backing up the script (given that he already has agreed to accept primary responsibility for releasing the transaction in the first place).  Eddie could still propose a multi sig transaction, but instead of Alice sending coins to a huge string, instead, Alice would receive a record allowing her to confirm that P2SH destination X is really what she thinks it is (which could be done on a public website instead of being built into any given client), and then she just pay the P2SH address.  We could leave the backup responsibility to Eddie, and relieve Alice and Bob of the burden.

Let's assume for a moment that Eddie only has backed up his deterministic wallet chain code and not the individual P2SH script belonging to Alice and Bob, and suffers a data loss.  While not trivial, it would still be possible for Eddie to recover that P2SH script from the hash just by taking Alice and Bob's submitted bitcoin addresses, and combining them with all of the keys coming from Eddie's chain code until the result is a match.  Even that potential burden could be completely eliminated simply by e-mailing the P2SH "destination verification" data to Alice and Bob, as in the event of Eddie's hard drive crash, Alice and Bob could recover the script from their e-mail.

Seems to me the ideal escrow transaction is (A AND E) or (B AND E) or (A AND B).  This way, in the event of disappearance of the escrow agent, Alice and/or Bob can release the coins to one another by submitting a transaction paying the other, requiring that other person's signature.

I don't want to get side-tracked on the philosophy and use-cases of P2SH (though, I do agree, I much prefer plain multi-sig for escrow/contracts).  Even if users agreed to use plain multi-sig and avoid necessitating such backups, there's probably plenty of information in their wallet that is nearly-critical to have backed up -- perhaps contact information stored with the wallet, order numbers of various transactions, identities associated with non-wallet addresses, etc.  A lot of this isn't widely used now, but I expect it will be with the URI spec -- I've been pushing for contact information and order numbers to be included in URI strings so that when a merchant requests payment for something, they can auto-document the transaction for the buyer by pre-filling those fields which will end up in the user's wallet.  Even without P2SH, the user has a strong interest in securing that data (and users always fail if effort expenditure is required for a seemingly-non-essential operation).

If plain multi-sig transactions became the de facto standard for contracts and one offs, it's unlikely they'd accumulate on the block chain.  An unspent transaction of that sort represents unspendable money that somebody somewhere would like to have unencumbered, so they're unlikely to leave it sitting in a state like that for a long time.  The moment Eddie releasing the transaction that commits to Bob getting the money, the fat transaction becomes ripe for pruning.

The IsStandard could be resolved simply by offering an option that says "make sure my transaction gets sent to this/these miner(s)" and getting somebody somewhere to agree to mine them into a block.  That of course ignores the overwhelming likelihood that the whole Bitcoin community will immediately realize the value and will accept defining such a transaction as worth relaying.

I was under the impression that P2SH was intended for all multi-sig transactions, though you are right there is no reason plain-text multi-sig can't be used (is there? they were probably never added to isStandard...).  One reason for desiring P2SH, in general, is that very long scripts become part of the TxIns instead of the TxOuts.  This means that they will never bloat a pruned version of the blockchain.  If everyone only used P2SH, then no TxOut script would ever be longer than 22 bytes.

For escrow transactions and one-offs where creating and backing up a relationship would be a burden, I see P2SH as the wrong tool for the job.  This might be more suited for a simple regular multisig transaction where the redemption script goes into the block chain and is based on normal keys in a normal wallet.  The purpose of P2SH as I understand it is to push the multisig burden off the sender and onto the receiver and to make sure that users and websites not interested in using multisig can always send coins to people who are, with a normal-looking and normal-sized destination address.  This burden shifting doesn't make much sense if three consenting individuals want to deliberately participate in a complicated transaction.

Such a transaction could be formed much like a relationship (in the sense that there's a "proposal" and "acceptance"), but would be far simpler.  If Alice wants to send coins to Bob and use escrow agent Eddie, then Alice and Bob could each send a fresh bitcoin address of their own to Eddie, and then Eddie could formulate a large and cumbersome "address" that contained the entire multisig script serialized within, and then Alice would just send the coins to it.  Both Alice and Bob's clients would automatically recognize an incoming but "encumbered" transaction, and both Alice and Bob would have the option to create a transaction proposal that would have to be sent to Eddie, signed, and released by him for the funds to go anywhere.