As you know, Tom, I have rather serious misgivings with the notion of "identity", and sexual identity in particular. I wholly agree with your last sentence. However, the operation that consists in defining an identity is, imho, is a case of the intellect capitulating, of confusing a thing with the representation of that thing.
Although I am spontaneously aroused sexually only by men, whenever someone represents me as gay, then I fear that (s)he has started to see me as somebody different from the real me. (Am I making sense ?)
I think you're making perfect sense....but keep in mind that the fact that you are, as you stated, 'aroused sexually only by men', IS part of who you are

but I get that you're afraid that people are going to get the wrong impression about who you REALLY are ( because of your sexuality ) just because most of them can't get pass the 'tags' and think that being gay is EVERYTHING you are, that it's something that defines you as a human being entirely, as a person. but you and I both know that there's nothing 'wrong' about being gay or bi, hell, I myself am bi ( admitted it to myself a few months ago after realising deep down that all people are bi

) so I perfectly get where your fear is coming from. but you need to learn to be ABOVE it. you need to make that fear your friend and except it as part of who you are- that's the only way you'll be able to act against it ( there's a book called 'fell fear and do it anyway'- the title says it all. haven't read it though, the title just came to mind and seemed to fit here perfectly

). make it your friend, not your enemy, don't push it away

just be yourself. I'm trying to do that myself, but the problem with me is- I 'know' a lot of these things in 'theory', but when it comes to the actual realisation of them, well, let's just say I suck! but I'm most definitely working on it, because I don't want to live my entire life deluding myself and convincing myself that I'm not afraid when I truly AM! the further I keep pushing fear ( IN GENERAL! ) a way, the longer it will stay there, and I will never evolve.