For me personally, this is not a big problem - I have my own PC, but the goal was to warn others who might share a computer with someone. It is just about privacy protection, and I agree that there are many bigger issues that need to be addressed.
It is not about tray icon, it goes away in my case also - but it can still be found in Notifications area where it shows balance of the last opened wallet. nc50lc is confirmed same issue on W10, but as you say, you use portable version.
I wonder, if the notification is still hanging there, does it update your balance if you receive a tx even if you have closed the program?
Anyhow, it seems we all use 3.3.8, ...maybe...
It is still there, after 24 hours I open wallet last time - but it is not updating balance and it shows old BTC price from yesterday. Yes, I am using 3.3.7 with Nano S, but i will make an update to 3.3.8 and select option "Hide icon and notifications" for Electrum, then I'll see if there is any difference.
I was thinking this is only related to W7, but some basic things stay the same in all versions. However, I think only one who would really seek it would find such information, Electrum has been around for so many years and no one has ever reported this (at least as far as I know).