There is a fundamental question of fairness. That is why mens and womens sports were separated to start with.
You have weight classes in boxing, wrestling, and some other events. its the same with sex.
It is not fair for a 200-pound person to wrestle a 100-pound person. Its the same with other sports. A male body has a physical advantage in sports, and sports is about the physical body.
If we have trans women competing against women, womens sports will cease to exist.
Colleges will start giving scholarships to trans athletes because it will give them an advantage on the field or court. If one does it the rest will too, and womens sports will be no more.