If you are sending to P2SH addresses then your initial transaction works just like normal. The usual size, fees and dust limits apply. This is because the P2SH addresses are just regular addresses other than the fact they are a hash of a script instead of the hash of a public key.
Where things can get trickier is in redeeming any of those outputs in subsequent transactions. As stated earlier, the redemption script for a 7-of-11 might be too large.
This is a critical point. P2SH makes things easier. For someone to send you funds you can just provide an address. The outputs are also smaller (as the output just contains a fixed 32 byte script which contains the hash of the redeemScript). However P2SH is a double edged sword. It can make it more difficult for you to spend funds you receive if you create scripts which are non-standard and worse you can even created scripts which are invalid which the network can't verify until funds have already been 'locked up' at that address. See note at the bottom on testing and risk of losing of funds.
'Sending' funds TO a P2SH addressP2SH hash is in the output of the tx
You can have a nearly unlimited number of P2SH addresses as outputs
The output is a fixed length hash of the script.
Actual script not "known" to the network, only "normal" tx validation applies.
The tx is not significantly different than a "normal" tx. Instead of putting a hash of the pubkey in the output you are putting a hash of the script in the output.
'Spending' funds FROM a P2SH addressThe redeemScript which hashes to the hash in the unspent output's PkScript is added to the ScriptSig portion of the 'spending' transaction's input.
In tx validation the script is hashed and compared to script hash in the prior tx's unspent output.
If that validates then the ScriptSig is validated normally.
The validation rules for scripts apply (see below). Actually they apply for all scripts but a "normal" (P2PkH script can't exceed the limit).
1. It is possible to have multiple m-of-n contracts as outputs for a transaction? From what I understand this should be possible but costly due to the bytes required to script the transaction.
This is where I think you may still be confused. With P2SH, an n of m address is simply the HASH of the script. 3 of 3 keys, 7 of 11 keys, 15 of 15 keys, any other arbitrary script. Instead of the conditions for spending the output being in the output just a hash of that script it placed in the output. So the direct answer is yes; the size of the script doesn't matter as the output contains a fixed sized hash of the script not the script itself. However pushes are limited to 520 bytes and since the whole redeemScript needs to be pushed to the stack to validate it's hash any address produced from the hash of a script larger than 520 bytes is effectively unspendable (without a hardfork).
2. Is there a size above which a miner (by consensus) will not mine a transaction to a block? You mention the number of bytes that are allowed? Is there a hard limit here or does the fee just increase above a certain byte size in relation to how many extra bytes are required to write the transaction to the blockchain?
Once again it is important to distinguish funding TO a P2SH address and spending FROM a P2SH address.
For funding a P2SH address (i.e. sending value to the P2SH address, the hash of the script is on the output side of the tx) in all but the most extreme edge cases there are no constraints. Transactions have no hard size limit (other than they will need to fit in a 1MB block). To be relayed by most nodes the tx will need to pass IsStandard which limits you to "only" 100KB but with the output being only ~40 bytes you could have thousands of such outputs.
Now SPENDING FROM a P2SH address is a little more complex as there are restrictions on what scripts are valid and/or standard. Also see the note at the end about the risk of losing funds due to delayed script validation.
A tx is invalid if any of the following are true - Block Size is >1,000 KB (this is a block level check but obviously a tx which can't fit into a block <=1MB could never be confirmed at least not until the 1MB limit is raised).
- A script is >10KB (this is per script so tx can be larger if it contains multiple scripts each less than 10KB).
- The size of the value being pushed in a script is >520 bytes (effectively limits P2SH scripts to 520 bytes as the redeemScript is pushed to the stack).
- The script contains more than 201 operations (excluding pushes). OP_CHECKMULTISIG with m keys is considered m operations for the purpose of this check.
- The number of keys (n) for a OP_CHECKMULTISIG operation is > 20
- The number of signatures (m) for a OP_CHECKMULTISIG operation is > than number of keys (n)
Your script generation code should validate the script against all validation rules to avoid an "oh shit" moment. See the warning about loss of funds at the bottom and be sure you understand how the 'delayed validation' of P2SH scripts makes it easier to lose funds. This doesn't make P2SH bad but it does give you enough rope to hang yourself. The bolded limitation is the easiest to violate. Since redeemScripts are pushed to the stack in P2SH input validation the entire script must be less than 520 bytes. Always do a hard check on the length of your redeemScript. Don't trust any rule of thumbs. For example a m-of-15 multisig script is <520 bytes if all keys are uncompressed however if your code accidentally generated one or more uncompressed keys you have now violated the 520 byte limit and created an address which is unspendable.
If the tx is valid but not standard it won't be relayed by most (virtually all) nodes and won't be included in a block by most miners. To get it confirmed you will need to bypass the peer to peer network and push the transaction directly to a miner who accepts non-standard transactions. There is no "miner discovery" mechanism in the protocol so you will need to manually check with each miner. Also the miner may not accept any non-standard txn and may have specific fee requirements so this should only be done if you absolutely can't accomplish the task with standard transactions. If the transaction is standard it will be relayed automatically by nodes and included in consideration for the next block by most miners so special handling is needed.
A transaction is not standard if any of the following are true- The tx size > 100KB
- The scriptSig size is >
500 now 1650 bytes. - The value of any output is less than the dust threshold (
5,460 now 546 satoshis). - The tx is low priority and doesn't include the minimum fee to relay (
10,000 now 1,000 satoshis).** - The tx is not final (nLockTime block has not been created yet).***
- The tx version is greater than the current version (which is 1).
- For 'native' multisig, the number of keys is > 3.
- For P2SH transactions, the number of SigOps is > 15.
** Technically this doesn't make the txn non-standard but even if standard the txn won't be relayed by most nodes if it has insufficient fee or priority so it may help to just consider it part of the IsStandard check.
*** The txn itself is only considered non-standard not invalid but if you include a nlocktime txn with a future nlocktime in a block then the block is invalid.
Note: It is possible to lose funds with P2SH so extensive testing on testnet is strongly recommended. The risk comes from the fact that the network is unaware of the script contents at the time the funds are "sent" to the P2SH address. For a simplistic example create an otherwise valid Bitcoin script which is greater than 10KB in size. Generate a P2SH address from the hash of the script. Make a payment to the P2SH address. The tx sending funds to the P2SH address will validated and be processed by the network without issue. However any attempt to "spend" the coins sent to the P2SH address will fail as the only valid method of "redeeming" the output is the script and the script breaks the 10KB validation rule. Those funds can never be spent (or at least can't be spent until the 10KB limit is raised or removed).