I’ve never felt “stressed” when I was around either smaller or larger women. Pretty much the only time women stressed me out was when I thought they might reject me… which was quite often.

I said stress for the sake of simplicity. I’d say straight men feel some degree of anxiety around women, whether it’s because they want something from them and are not sure if they’re gonna get it or because of the fear of being appraised. But usually someone who feels confident ignores women if none of them is of interest.
Gay men are probably not gay only because they’re “stressed”, whatever that means. It might be a combination of a couple of factors. But on average, they seem to be more stressed than their straight counterparts. Remember the Savic brain study? They had similar brain connections as straight women in some of the areas that manage stress too. The Safron-Bailey brainscan thing showed the bigger amygdala reaction in gays. And many, many other connections pointing in this direction. Check out this one on mood.
I thought that there’s a connection between male tallness and women’s need for protection (which I have a hunch it has a lot to do with how secure they feel, ie how their stress response is pitched). Comparably, gay men’s insecurity might have a say in why they’re attracted to guys.
There’s other dimensions on which women and gays seem close neighbours, like spatial orientation and narcissism. Arguably, there are many hetero men who are narcissists and so there are many passive gays who put themselves in second place in relation to others. But both gays and women are on average more careful with their bodies. On the contrary, straight men and lesbian women are less aware of their looks.
The rest of the gay-straight question might be answered by instrumental aggression, that straight men direct towards women sexually, whereas gays might have it lodged within themselves.

Men have a strong tendency to attach sexual meaning to women’s ambiguous cues (Abbey, 1987; Haselton, 2003) and are likely see sex in red regardless of the signaler’s intent

.
Let it be said that women know that “men have a strong tendency to attach sexual meaning” to everything. The old expression that men “think” with a certain part of their anatomy is not news to us!