You have a very sensitive conscience.
That is good, but you must also be careful about it. So you don’t beat yourself up too fast.
I have a hypersesitive conscience too and I have asked myself such questions.
I have asked myself things like, if I gave a beggar “only” some bread, am i allowed to buy myself some chocolate on the same day?
On the other hand, I have often bought somebody something that I rarely treated myself to, being on a kind of low budget… and actually had to learn to not do that often, as I must also watch that i eat healthy enough.
But it is very very difficult.
I don’t know if this is about sin really. Or more about walking the fine line between loving ones neighbor and taking care of oneself.
Which should not be a contradiction at all.
People like you and me struggle with such things.
Here is another example (I don’t remember it exactly, it’s been a couple years or so, so this is an approximate rendering

I went to visit an old lady at a home and then told her I had to go, and left… On the way back to the bus stop however I did stop at a used books stand and bought myself a book. Then I felt guilty and wasn’t sure whether or not I was allowed to read that book now, since it was like time stolen from the old lady.
A priest I told this to later quickly caught on and said I had to be careful to not go into scrupulosity.
It really depends on te situation, often.
Yeah, I know what you mean, you buy yourself a treat but then maybe he would have needed your money for something necessary, right?
But then, a lot of people would need our money for something necessary… does that mean, we are never allowed to buy anything we like?
Are we allowed to turn on the heater when it is not really freezing freezing cold when we could send the money we spend on electriciy respectively gas to a charity instead?
It may lead to… neverending questions.
Or total ascetism. Which can be something holy. But probably only if done for the right reasons.
I am sorry, probably I am throwing in more questions than i am answering. I basically wanted to say: I understand you; been there…
Maybe an option would be to just give him some change or buy him a cup of hot coffee when it’s cold and still leave yourself enough to buy what you needed.
Unless you are really CALLED (and not by scrupulosity, but with love in your heart) to spend the money on something else. Like, a relevation: I do not really need this.
Maybe I am wrong; I struggle myself with such things.
But you know what?

At least you felt for the guy. A lot of people would not even have thought about it.
Anxway maybe it helps you that you are not alone with such questions
