I agree with you. Your reasoning reminded me of business trainings when "trainers" are advised to give up small regular expenses (such as a daily latte) and say that saving on this in a year turns into a large amount. In my opinion, giving up small pleasant spending (be it latte or gambling) will bring more disappointment than joy from the amount of money saved.
When we finally count on how much we saved, we tend think what if I have reduced my expenses more and saved more. It certainly happens. Often, its would be really so good to plan to spend on something which would just make us feel happy or just past time. But when that occasion gets over and we think back on it, we realize, how it be good enough if i have skipped that occasion. Later, I felt its ok and its necessary to spend on something which would certainly make us happy and entertain us.