You're generalizing. If Condit wins, then he wins. Plain and simple. We'd be foolish to ignore factors such as ring rust if they're blatantly apparent, but I expect ring rust to be a non-issue. If Carlos wins, it's more likely to be because he's a damn good fighter in his own right.
As for the size difference between GSP and Silva, turnabout is fair play. One of GSP's most dominant performances came against BJ Penn, and to this day, many people claim, 'Well, he only beat up on a lightweight'. That road, much as I dislike it, goes both ways. But again, that would be a factor I'd completely ignore unless it were blatantly obvious. There's a difference between Anderson KOing GSP in ten seconds and outmuscling him (and let's be honest... if he can hold off much bigger men like Chael and Bonnar, he can probably do it to GSP).
I don't see this as being too big a deal regardless. If Silva stomps GSP, most fans will see it as his being the better fighter. You're associating a handful of GSP fans with the lot of us, and to be fair... every fan base has its 'go to' excuses and defence mechanisms. Let us not pretend that this isn't especially true of Silva fans.