Yes, the point is as long as we do not involve or harm others in any activities or affairs that we have, then you are free to not care about anything that other people say to you, but maybe on the other hand it doesn't matter if you want to tell them a little by providing a little evidence that what other people say is not true.
And regarding gambling, yes, as you said, one of the reasons why we don't have to care about anything that other people say is because we gamble with our own money, but of course it would be a different story if it turned out that you gambled using money borrowed from other people, because if so, the fault is on your side.
But on the other hand, you are also free to tell or hide your habits from other people, and if it turns out that you are basically one of those people who don't like it when people criticize anything you do, then clearly hiding it will be the best decision you can make.
If we lived in a society that very strict about gambling, we don't have to gives a little evidence to them because they will not want to accept anything. They will deny that and will not tolerate that thing so we shuld be better to keep that for our secret and not have to tells to anyone. We playing gambling using our money that we have so we must responsible with that and not let ourselves spends too much money. Besides that, by not telling to other people about our gambling activity will makes us calm enjoying the gambling games.
If we don't want to get critic from other people, we must be selective to tells about something to other people and not tells to random people especially if we don't know much about that people. We better to hide our gambling activities than gives us the problem.
I would say we should adapt ourselves to society. Gambling is legal in many countries. So if I gamble openly in these countries then there is no problem. And no one will make any bad comments about gambling to me in these countries.
However, gambling is not legal in many countries. So we must adapt ourselves to the social system. We should not openly gamble in these places. First, since gambling is not legal in these countries, I may face legal hurdles. Secondly, most people in these countries consider gambling as a crime. So I must hide my gambling habit. I certainly don't want people to have a bad idea about me. So I always try to adapt myself easily to the people and environment around me and conduct myself accordingly.
If you playing gambling in a country that allow gambling, you will not getting a problem but that will be different when you playing gambling in a country that prohibit gambling. You will get the problem and maybe the government or police will arrest you and punish you. We must adapts with the situation where we lived and if we thinks that playing gambling in our country can give the problem, we should not try that. We have the other choice to playing gambling which is we can play gambling in the online casino.
We don't have to think about what other people will say to us if they know that we are playing gambling. We can responsible to ourselves when playing gambling so we don't have to afraid or worry unless we can not control ourselves and playing gambling only gives a problem to us.