You don't have issues with "casual fans", which is thrown around way too much, and in essence shouldn't even be used in a negative sense.
You have issues with retards.
The hardcore/casual divide really doesn't have much to do in the way of knowledge, as it does enthusiasm. Whether we like it or not, "that guy" that won't shut up about MMA/UFC, watches every event and lets the world know how stupid he is at every turn is a hardcore fan, not a casual one. He's just a hardcore idiot. And on the other hand the person that appreciates a good fight for what it is and shows interest in the technique used ect, but doesn't make time to watch every show, doesn't feel it's worth it to buy a PPV and mostly just watches free shows + internet clips, is a "casual" fan. Just a smart one.
Allow me to make a graph: