Rory is a bit of an introvert, so that's a big part of why he seems "off" to some. Others just hate him because he doesn't dress or act like a "bro". The ones who hate him because he beat their boy BJ - well, I can respect that. That's how it goes in sports.
That, or he's a narcissistic sociopath using mixed martial arts as a tool to act out his sick fantasies until this doesn't do it for him anymore and he starts to **** and murder people. Either is a fair opinion to have though.