We should not trust any devices that contain secure elements either, we can't rely on SE because the information stored in these elements can be obtained in other ways. For example, hackers don't need to get physical access to our SIM card, instead, they can easily convince mobile phone employees to clone the SIM.
That is because your SIM Card contains the information required to authenticate with the cell towers. Operators can permit an alternate SIM card to use your number but that doesn't mean that the SIM card has failed it's purpose. It's primarily purpose is to
resist any bruteforce attacks to get the keys and CMIIW, most modern SIM card bricks itself during a bruteforce attacks.
Our payment cards have all sensitive information written on them, which makes them vulnerable to phishing, hacking, stealing, etc. If you were to lose it or get robbed, you wouldn't rely on SE to protect your funds, you would block your card instead thus making it completely useless.
The secure element on a debit card is used to protect against cloning attacks. It is difficult to clone the chips in an EMV cards while it is easily to clone the magnetic stripe cards. The scenario that you present isn't the point of EMV cards in the first place, it's designed primarily to combat swiping attacks.
When I lose my hardware wallet, I trust that the secure element can resist the attacks until I can transfer my funds out.
The one time you can rely on the secure element is when the information that you need is inside the secure element itself. There is no one else to do social engineering to get the information within the SE unless the owner chooses to reveal it himself. When used properly, the secure element will not reveal the information held within and that is it's primarily purpose inside a hardware wallet, and inside a sim card, payment cards etc. For the scenario as stated, HW wallets containing SE still provides an additional layer of security.
) like this:
