Generally, it has been the case I have an easier time getting comfortable around men then women, outside whoever I am dating who has the pleasure of hearing me blather on forever about whatever random thought comes into my mind.
In general though, as much as I would like to say this is not the case, it does come down to the *idea* of sex. When talking to a guy, I have no worries that anything will lead to anything related to sex anything. With a woman, there is that sliver of uncertainty. What will being friendly be construed as? How should I construe someone being friendly to me? Without rattling on a million more examples of silly concerns, it comes down to me not really being confident in those unspoken social cues that come with interacting with people. There aren't many I worry about when talking to guys, but I worry myself with them when talking with women.