Heartless or justified?

If you haven't any charity in your heart, you have the worst kind of heart trouble.
~Bob Hope

We've all experienced that moment when a stranger asks, "Can you spare some change?" Clearly, these are indivduals who look disheveled and low on luck - barely covered in tattered clothes, sometimes wheeling a grocery cart full of aluminum cans or carrying a soild bagpack. How do you react? I can say I have never offered any kind of funds in this situation. I don't know if I should feel guilty or not - I would consider myself a kind-hearted individual, but I'm starting to second guess the notion. On the other hand, the more I think about it, the more I rationalize my decision to seal up my pockets to folks on the streets, though I may need to change my approach.
My previous job required me to commute to an area of high poverty. On my drive home, it was very common for me to encounter individuals asking drivers for money at red lights. It seemed like I was always stopped by the freeway on-ramp where one particularly crabby man would walk past car windows, his bony, dirty hands pressed together in a pleading manner. After checking to be sure my doors were locked, I would often fiddle with my cellphone or radio, not daring to make eye contact. I'll never forgot one day when I barely missed the light and was once again trapped near my transient friend. As per usual, I looked away as he sauntered up to my window. Apparently, he was not pleased with my lack of attention in his directly, and began shouting profanities at me. My first thought was, "I might be safer if I just blow this light!" Then, I thought about my actions - why wouldn't I at least look this man in the eye and give him a gesture of apology? Instead, I was ignoring him, trying to pretend he didn't exist. Perhaps I deserved his deratement, for not even treating him as a fellow human being.
Today, JC and I were walking out of a store when I noticed two women conversing near our car. As we passed, I overheard one woman saying to the other, "You can't even spare a few pennies?" and notice the other woman clutching her purse and shaking her head. I thought, "Great, she's going to ask us next." Luckily, we made it to the car before we were engaged in conversation, so I thought we were home-free. Then, out of the corner of my eye, I saw the woman walking towards our car, waving her arms. JC must not have overheard the conversation like I had, because he said, "Oh look, she's trying to stop us. Did you drop something?" I quickly backed out of the parking space with a quick reply, "No, she just wants money." As I backed the car up, the woman continued walking toward us, making desperate attempts to stop me. When I was finally able to turn and leave, I did so without even giving the woman a second glance. As we pulled away, I felt a strange tinge of guilt - even if I wasn't going to give her money, should I have at least acknowledged her?
At the same time, what gives her the right to pursue me in a somewhat agressive manner? And why do I really deserve to be sworn at by a man on the streets? I guess I can't even picture how the situation would play out if the tables were turned. I may not be living high on the hog, but I know I have many advantages that too many people do not have. I am definitely thankful that I am not in their shoes, but does that make me responsible for helping them? I guess I'm a big conflicted on how to react, but I think I do need to be more generous with the way I treat these people.