You could say that because you are not addicted to gambling, what he said was that those people who are addicted on gambling can't even stay away from it even for just a single day, I guess you misunderstood it. You are a lucky person though because you could control yourself when playing gambling, playing gambling for fun is not that addicted since you are not after the money.
Yes, if we are already addicted to gambling at a heavy level, gamble will be hard to avoid. I've felt how bitter being an addict and my life is almost messy about it. But fortunately, right now, I've been able to change my perspective on gambling and I'm slow but certainly able to reduce my gambling activities.