It would depend on how he loses and to who he loses to.
If he loses to Jones at LHW, it wouldn't be a big deal, because he would had lost to a much younger, bigger fighter outside his weight class.
If someone like Rashad comes down and wrecks his shit, and Sonnen goes up in weight class and gets blasted by mid tier guys, then that would definitely hurt his legacy. People will start to claim that he ruled a weak division whether it is true or not.
People hate on Fedor for ducking fighters, but he never ducked a fight. He was willing to make the fights happen if a financial agreement that satisfied him and his team was met.
GSP, Silva, and Jones on the other hand don't want to fight because it would hurt their legacy and sponsorship money. If the UFC is not careful, we will end up with the same thing that happens in boxing, the big names will hand pick their opponents and not risk fighting guys that can beat them.
Fedor's management team ducked plenty of people... like Brock and Overeem.