Yeah it is annoying but I'll do my best to ignore it. I'll admit at first I would send a neg rep back with my own insult but this last time I sent a positive rep back with a "LOL I ain't mad at ya".
LOL most of the stuff I get is very positive. I have way more positive reps then I deserve. I have plenty of neg worthy stuff that doesn't get responses so I guess it evens out.
Thats the best response. To be honest, I don't think there are many on here who hover their cursors over your, or anyones, green bar to check on their reputation. Mine says I'm a baller, whatever the hell that means.