"how must the last transaction on the mainchain look like" to be able to peg-in on the sidechain, which prevents me (=Alice, the "originator" of some sidechain funds) moving the funds causing chaos on the chain, invalidating tons of transactions, etc.
There is no strict enforcement, because there are many options:
1) You can use P2PK. Then, to make it safe, you have to create it with someone else in a way, where you know the public key, but nobody alone knows the private key. Multiplication of ECDSA points is a partial answer, because then nobody can sign it. A full answer is for example 2P-ECDSA, described here:
https://duo.com/labs/tech-notes/2p-ecdsa-explained2) You can use P2PKH. It is possible to encrypt things first, then compute the key as in P2PK case, and then hash it. After encrypted hash will be computed, it can be decrypted and sent to the network, then even nobody knows the combined public key behind this hash, because it was available only under homomorphic encryption. Note that having unencrypted signature will reveal that key, but you can also accept other proofs inside your sidechain, and then keep it hidden.
3) You can use P2SH. Then, if both parties know that things are standard, and evaluate into OP_TRUE, they can send it on-chain. In that case, the whole script is unknown. Also, OP_CODESEPARATOR can be used to sign only part of the script, and accept anything that comes before that opcode, but in this case it is better to use P2WSH or P2TR, because P2SH with OP_CODESEPARATOR inside will be non-standard.
4) You can use P2WSH. Then, it is similar case as with P2SH, but you can create more scripts, and keep it standard. Also, it is cheaper, because of witness discount, but the whole way of encrypting things first, executing them in encrypted form, getting needed proofs, and then decrypting and sending on-chain, is still the same.
5) You can use P2WPKH. Then, to make it standard, you can always use compressed public key.
6) You can use P2TR. Then, you can prepare N-of-N multisig as a base, and then you can safely experiment with TapScript.
A lot of flexibility is needed, because people will not accept sidechains as a soft-fork, and because they should be upgradeable, when other features will be enabled on-chain. Also, it increases privacy, so it is a good thing to allow broadcasting all current standard scripts, because a no-fork way means that in the best case, nobody should exactly know, when it started, and if those things were active in the past. For example, what if there were active sidechains, and if some users joined them, and then all of them moved their coins back to the mainchain or lost them? If possible, the starting point should be unknown, and it should remain as "always was possible, always was active, since 2009". I think that kind of sidechain will have no Genesis Block, it could always start earlier than all nodes assume, but all proofs were simplified into OP_TRUE and discarded.
The crucial part where I fail to see a solution using Bitcoin Script is the penalty mechanism.
Look at Drivechain proposal, and try to figure it out, how sidechain miners can always steal funds by constantly signalling the wrong state of the sidechain, continuously for many months. If you compare LN user broadcasting some previous closing transaction, with sidechain user doing the same, it looks very similar. The main difference is timing: in LN it could be two weeks, in sidechains it could be three months.
In an "open" sidechain, where you transact directly with all users, it can't be like with Lightning, because you're not updating a state with a single multisig partner and could thus invalidate previous states like in LN.
Why not? On-chain interactions are prepared in advance. It is not like in LN, where you can open a channel instantly with anyone. Here, in sidechains, on-chain updates can happen only from time to time, it was proposed to happen every three months. So, if you have on-chain coins now, and want to move them into the sidechain, you join a larger group by connecting to the sidechain network, and adding your signature. It will not be instantly visible on-chain as a new transaction with zero confirmations. It will go through a long batching process, and will be finally published as a single, large, on-chain transaction, where many coins will be pegged in, many coins will be pegged out, and a single sidechain output will represent those who will stay inside the sidechain. Also note that if you choose Taproot, then signatures can be batched, so a single signature can represent many users. And note that things will be executed in encrypted form, so you will not know exactly, how many people are inside, because nodes can reveal proofs provided by other nodes, and because partial scripts that will evaluate into OP_TRUE can be discarded. Also, only some users will go through this on-chain process, because most will use swaps.
I also looked at an "overcollaterized sidechain" model, where extra funds for penalties are required in addition to the funds used for peg-ins, but this didn't solve my problem - to ensure, via a mainchain contract, that only the last person in a chain of payments can peg-out (and previous intents are properly penalized).
If you have a CoinJoin transaction, then what is the cost to penalize misbehaving node? There is a choice between time-based cost, and satoshi-based cost. I think sidechains will reach some equilibrium somewhere in the middle, because using coins is one way, but you can use less coins if you want to wait longer. I don't know exactly how large will be sidechain transactions, and how many signatures will be joined in a multisig, or how many coins will be allocated for penalties, it has to be tested in practice.
This of course has the disadvantage that the sidechain cannot "onboard" new participants who never used Bitcoin, all have to have funds locked on the mainchain, otherwise the penalty mechanism would not work.
Why not? If that penalty will be satoshi-based, then it can affect those who have some satoshis. But for those who don't, there is still a time-based penalty. For example, I can safely sign a transaction that will give you all my coins, and will be valid after 10 years. Because I am almost 100% sure that I will move those coins by then, and the transaction I gave you will be invalid, when that timelock will expire.
So yes, you can send someone some coins inside the sidechain, when that person will have no on-chain coins. They will be quickly valid inside the sidechain, but to get any mainchain coins, that person will need to go through peg out, that is difficult, it consumes a lot of time, and then sidechain miners will have a lot of time to get all of those coins from that person.
There is no contract condition I can imagine that only Dave can fulfill (the last participant of the chain) but not Bob (intermediate) nor Alice (always talking about Bitcoin Script).
You assume that earlier participants know the script that was used, and you assume that they have all needed keys, so they can move those funds alone. There is no need to strictly require that. Because to make it generic enough, it is needed to accept things in a form they exist on the sidechain. So, Alice joining the sidechain from the mainchain directly is not the only use case. Another valid way is to leave one sidechain, and join another sidechain. As long as proofs are compatible, they can be accepted. So, if Alice can provide a proof that she is going to leave sidechain A, and join sidechain B, then Bob from sidechain B can check that proof, and accept it. Also, proofs from other kinds of swaps can be accepted, so a swap can be batched into the sidechain transaction, that will be finally visible on-chain. There are many possibilities, someone going from LN into the sidechain by forming a different channel closing transaction is also an option.
You can't prevent someone who wants to steal coins convincing him that there is some way to peg-out "faster".
You also cannot prevent some malicious miner who wants to steal coins, convincing him that mining on top of some earlier block is not going to work. And you cannot prevent some LN user from broadcasting some earlier closing channel transaction. All systems have their assumptions: Bitcoin assumes that the heaviest Proof of Work chain is formed by honest nodes, break that assumption, and Bitcoin no longer works. LN assumes that the honest participants will grab the coins, before expiration of the timelock of some earlier closing channel transaction. And sidechains assume that their users will inform the mainchain with sufficient proofs about the correct state of the sidechain, before the thief will grab those on-chain coins (but again: assume that someone created a malicious Drivechain, and read what Paul Sztorc said about sidechain miners stealing those coins).
Again, how exactly would this penalty mechanism work?
1) Alice will encrypt her penalty transaction, and send to the sidechain, with sufficient proofs.
2) Malice will send some earlier transaction, trying to steal the coins.
3) Every sidechain user will have a proof that Malice published transaction that should be invalidated.
4) Only if absolutely no sidechain user will react, or if all Bitcoin miners will decide to accept malicious transaction, those funds can be stolen.
Wouldn't that mean that payments could only be done between of the members of this multisig contract?
No, it only means that their action is needed to initiate it, but later it can be passed to anyone. If someone will decide to leave the sidechain, then that transaction will be a typical Bitcoin transaction, so it can contain any outputs. If you have Alice and Bob in transaction input, you can still have Charlie in transaction output. Also, if Dave wants to join the sidechain, while Charlie wants to leave it, then Dave will provide some on-chain transaction input, that can be used to create transaction output for Charlie. In the end, everything will be batched, so all sidechain interactions can be cut-through in the final on-chain transaction.
You're right, this is not the same thing as a federation, but it's quite similar in that only the users participating in this multisig contract have control over peg-outs.
Not only, because people entering the sidechain can also provide their inputs, so those who want to exit, can use those coins.