Practical Considerations, Not Hate

It's a practical issue. When I was working as a bartender, if a man had tried to enter the female bathroom, I was on firm grounds to eject him, and had the ability to call the police to remove him, with the practical side-effect of NOT having to have evidence that he had caused trouble. His male genitalia was evidence enough that he was in the wrong.
Without that power, you have to wait until AFTER he has caused harm. There was a book written about the West Side Rapist and one of the most interesting points is that the guy had been picked by the police several times, for "Peeping Tom" behavior - looking in windows at women.
Harmless, right?
Wrong.
James Neff explains this activity as "pre-rape behavior" - the scouting around due to the perp's preferring to ascertain whether the intended victim is sufficiently isolated.
THAT'S what the law is meant to deter.

NOT a female-appearing person entering the ladies room, doing his business, and leaving.  Not a female?  If you can pass successfully, who cares?

Not someone who has gone "the Full Monty", and no longer has male parts.

But, the physically-male person that hangs around, looks for victims, and takes advantage of the isolation.

And, without a firm line drawn between men - those with "outies", and women - those with innies, those opposing the law will unwittingly facilitate rape.