I think asking trans folks of colour would be a place to start.
Probably that would be the most definitive way to answer the question.
Jas it's an interesting idea, but I'm thinking the way your analogy falls down is that the physical differences between people of different races is much, much smaller than the physical differences between men and women.