So in order to send/spend BTC I will need:
either the 12 word seed phrase (to recreate the wallet including all its contents from scratch)
or the actual wallet file (only the small 2kb file inside the wallets folder, right?) and the password, correct?
...
Yes, that is correct.
Another follow up question: If at some point I open the wallet file in electrum and connect to the internet, if my computer is not compromised presumably nothing will actually change in the setup when I disconnect. The offline wallet will have spent some time online but assuming nobody stole it or found the private keys it can still go back to cold storage and be safe from theft, correct?
Yes but true definition of COLD STORAGE is when the wallet NEVER come online (once came online it's not fully a cold storage).
Sum up:
Anybody who gets access to either:
1-your unencrypted wallet file (encryption must be strong otherwise it's not big deal to break it)
2-your wallet seed phrase
Can spend all your bitcoins ( no matter how safe you stored the other thing !!!)
So the weakest security point is at the moment of creation the wallet so if your PC was compromised that time (and the hacker had gain access to your private keys) he can just wait till you store some BTC there.
But honestly I think this entire offline/cold wallet thing is mostly for people who keep their entire wealth in BTCs and loosing them would ruin their life. IMO using simple home PC with proper antivirus, not visiting doubtful site ... just keeping general online safe habits is enough to be safe.