You may have a few of them: Situational friends. People whom you primarily see within the context of one activity. I've had many of them from work, classes, groups, volunteer activities, and Occupy. I see these people a lot, in some cases more frequently than I see my close friends. But no matter how much we chat, they still don't know a lot of personal stuff about me. (In fact, I'm a champ at knowing people for a really long time and having them know almost nothing about me as a person. I don't even know how I do it.)
This is on my mind, because at the last Occupy meeting, someone mentioned that "half of the working group is out [as gay]." I cynically thought, "I can't come out because everyone would make fun of me". Even though I've come out to positive responses a bunch of times, and I'm a big proponent of the action in general, it doesn't stop being scary. Although I doubt anyone cares what my orientation is, suddenly it felt like the elephant in the room to me. I am, indeed, the only person in that group who is not either out as gay or in a long-term heterosexual couple.
Situational friends can be the hardest people to come out to. The level of emotional investment is fairly low, and yet you still have to spend a lot of time with them, making things difficult if their reaction is negative. Situational friendships also often take place in groups, and it's much harder (and in my experience, not the best idea) to come out to a crowd. I don't want to come out, at the wrong time, just for its own sake. I won't lie about my sexuality--the one time a situational friend asked me, "Are you straight?" I said, "No, I'm asexual". But that rarely comes up. No one EVER asks me annoying stuff like, "do you have a boyfriend?" or "why are you single?" As aggravating as those questions (and the assumptions behind them) are, I'm left with no real segues.
It makes me bemoan the fact that there's no way to successfully drop hints about being asexual. I think it's that lack of cultural context that makes coming out so hard and so formal-seeming a lot of the time. In my experience, no one says "by the way, I'm gay" to their situational friends. They mention a girlfriend, wife, boyfriend, husband, or someone they find attractive (of course, there's room for misinterpretation here, too...another place where bisexuals easily share our experience). I can't say "that person is cute" without having people think I'm straight or gay. I think that's why I never told my friends about my rare crushes, even though I cultivated them (sometimes) in order to fit in.
If someone knows me for 10 years, they'll see that I never date or have sex, and they might start to realize that I'm "different" somehow. But even in the case of these sticky situational friends, I don't think I'm going to wait quite that long.