This is definitely interesting topic to think about. I think the device will have some known hard limit of inputs and outputs, depending on used chip and amount of available RAM. This information can be provided over the API (message Features?), so desktop client will have a chance to handle it properly (offer to split transaction to more smaller ones, for example).
But maybe we find some algorithm how to do signing in some streaming way so the device won't need to have everything loaded into the memory at a time...
FWIW, here's approximately the method I use:
1. Supporting transactions (i.e. transactions whose outputs are being spent) precede the spending transaction in the data stream.
2. Supporting transactions are serialised as they would be in the blockchain.
3. The spending transaction is serialised as it would be in OP_CHECKSIG (with all inputs removed except for the signing input, which has its script replaced).
4. Before each supporting transaction, the output number of the output being spent is explicitly specified.
This allows the bunch of supporting and spending transactions to be parsed by a streaming parser. Each output in the spending transaction can be displayed and approved as it is parsed.
The supporting transactions are there to calculate the transaction fee. The parser does this to calculate the transaction fee:
1. For each supporting transaction, the hash of the transaction, tx_hash, is calculated.
2. For each supporting transaction, the nominated output (from (4) above) has its amount added to the transaction fee.
3. The list of tx_hash and output numbers is itself hashed into ref_hash_compare.
4. When the parser sees the spending transaction, it takes the list of input reference transaction hashes and input reference numbers and hashes them into ref_hash. ref_hash should == ref_hash_compare. This step is necessary to confirm that the correct supporting transactions are being included.
5. For each output in the spending transaction, the output amount is subtracted from the transaction fee.
BIP 0010 is relevant to this discussion. What do you think of BIP 0010? It might be a good idea to talk to etotheipi about it, if you haven't already.