Replace gambling with: playing computer games, going to a club dancing, having sex, biking, going to the gym.
All cost money (yup some of us use condoms), all are taking time but you have fun doing it, so if you have fun gambling why should that be different?
I would argue that the restaurant is a bit different than the others since you eat, a thing you would have to do anyhow, so despite the difference in price it's still something you would have done in order to keep living, but yeah, I don't understand why everyone keeps focusing on the fiances and not on the entertainment aspect of it.