My husband's solution to anyone making an issue of it is to call everyone bastard when they are there.
Reasonable people actually ask politely for you to try and remember but don't make a fuss if you dont.
I've known a few very androgynous people. I have never called them by a pronoun other than you or they. Names didn't help guess on some. Took over a year to learn that one was a guy.
It's the morons who aren't actually transgender and like to fake it for attention because it makes them feel special. They bitch and cry for attention and throw fits. Then in a few years when it's not good for attention they move on to the next thing.
I went to school with someone like that. ADHD was the thing her and her attention whore Mom decided she had because she'd read fashion mags instead of listening in class. Then it was "No I actually have aspergers." And she proceeded to be a cruel bitch and claim anyone calling her on it was violating her rights. She's hopped on the trans wagon recently. Didn't change how she dressed or acted, just wanted to cause scenes if anyone called her ma'am.