You don't have to be one thing. You can't help your chemistry--what you're drawn to, what repels you. You can't squeeze an ill-fitting hat onto your noggin, no matter how you wear your hair. You just end up grumpy with nothing but a wicked headache to show for it.
So throw it away (okay--donate it; don't throw it away--that's just needlessly dramatic and wasteful...) and try on another. And another. Keep a shelf of hats in your closet. Hell, keep a closet of hats in your house.
Love how you look (or how you're able to hide--there are hats for that, too) and don't give a flying fig to anyone who tells you otherwise.
(Or, better yet, buy them a sweet Indiana Jones hat and tell them to seek their own adventure--you're already living yours).