You're not proud of your heritage, but have ZERO problem with other races being proud of theirs? Weak. Everyone is entitled to be proud of their heritage as long as they don't hate people of other heritages.
I think he was more pointing out that the association is what matters. Hitler didnt invent the swastika, he just adopted it as a symbol. It is just associated as a symbol of hate due to the historical context in which it became the most infamous. Saying something like white pride is not neccesarily a racist comment, but that is how it is almost always going to be taken.