I have low T and to be honest it does not affect my daily life. Then again my daily life is sitting at a computer working. When I play with my sons I don't ever get tired or feel like I don't have enough energy. And by Low T I mean low, low enough that I am borderline TRT recommendation.
I have to imagine for guys who run through their peek fitness everyday that it is life affecting. Being slightly lower than normal could probably affect you over both the short and long term.
My honest opinion though? Competitive people will do ANYTHING for a competitive edge. While I believe there are a few guys who suffer from this and need TRT, my honest belief is that majority are gaming the system for any edge they can get.