So gender roles are based on physical characteristics. So is race.
Black people have a collective experience of surviving slavery and thriving even though USA culture treats the as a permanent underclass, based on physical characteristics. Their ancestors survived because they worked hard, endured much, and managed to find joy and meaning through it all.
Sure -- all of our ancestors survived similarly. However, the experience of slavery and apartheid for many generations has shaped their lives and culture, and we need to recognize and respect that if ever the racial divide in this country is to be healed.
What about other social constructs -- acudemia, workplace hierarchies, our economic system, etc? Do you suggest that we ignore these also? Try that and you probably won't last too long.