I'm a smaller guy and anytime someone says that to me, I cant help but to smile. It means that they assumed something that was wrong. In MMA you always want your opponent to be caught off guard. Just sayin...
Same I've heard this phrase myself many times for the same reasons. I've always been athletic and I've been lifting weights since I was young.
Most of the time its when im clearly superior to someone who expected much less of me.