It depends on the context. Obviously it is offensive, people take offense to it. Black boy or white boy aren't any different to cracker or nigger.
Paddy or Taig can be offensive depending on the context, what I don't get is Fenian. I mean....it's a compliment to call an Irish peron a Fenian.