Depends who it is who gets that W over him. I think it would, just like any loss typically does but given how he's gone unbeaten in so many UFC fights it will go down as him coming to the end of his career and being over the hill.
The fact is if he retires now he retires an invincible fighter who demolished every top middleweight. Put a loss on the end of his career and suddenly he wasn't invincible just dominant for a very long period of time.
I say let Rashad fight him and if Silva wins, retire.