I don't see why.
GSP, BJ Penn, Mike Pierce and even Chris Wilson had success scrambling back to their feet when Fitch took them down. His ground control isn't otherworldly like a Jacob Volkmann.
Especially considering Erick Silva is genuinely good at it.
I don't think "he couldn't keep BJ Penn on his back" is a good argument ever for anyone.