There's a fair amount of variation even regarding sex. Intersex/ambiguously gendered people are born all the time... even chromosomally speaking there are people who do not fit within XX and XY. Doctors and parents in the past have tried to force these children into one sex or the other and it can have disastrous consequences.
That's why I chose to word it as 'rather rigid.' Comparatively speaking, that's exactly what it is. I certainly wasn't trying to disregard people with sexual differences at birth either