But for a multisig output that is being signed by multiple signers and where the signers are using different SIGHASH types, I think there needs to be a defined way of figuring out what SIGHASH mode is in effect.
From what I understand, the SIGHASH flag is used to determine what the signature actually signs.
transaction --- SIGHASH ---> signature-target --- ECDSA ---> Signature.
Since each signature has a different sighash byte, they can also use a different signature target.
There are only 2 steps. The bottom 5 bits select one of the output options and the 6th bit selects if you want to blank out the other inputs.
Unless the bottom 5 bits are 0x02 (SIGHASH_NONE) or 0x03 (SIGHASH_SINGLE), all the outputs are included. If the bit for 0x20 is set, then all inputs are blanked except the current input (SIGHASH_ANYONE_CAN_PAY).
See here.
While it could potentially be expanded, that would be a hard fork to include it in the basic script. P2SH or SW could include new sighash operations in their script (and SW is expandable).
That is the part about how to make the modified tx to sign, but how do I treat a set of sigs for a tx where the different multisig signers use different SIGHASH modes?
I can verify it is a valid signature, but what happens if there is SIGHASH_ALL for one sig, SIGHASH_SINGLE for another and SIGHASH_ANYONE_CAN_PAY for the third sig for the same input. Things are complicated enough with the different vins for a tx, which might or might not be dependent on each other, but do I do an M of N and count any mismatched SIGHASH as invalid? and which ones are mismatched if there are three different ones? Maybe depending on other clues, it can be determined which SIGHASH mode is valid for a specific tx, but what if more than one could be valid? It seems there would need to be a precedence or at least much clearer explanation on what should happen in these mixed SIGHASH vins.
Then I can start to worry about what happens when we have a single tx that contains mixed SIGHASH vins. As it is, it is tempting to just require SIGHASH_ALL across the board.
James