Open Question: Small Penis Problems?

I'm a guy, early 20s, and I have a small penis. I guess I'm asking females and other guys with this problem...but how should I go on with this? like it's probably 3-4 inches when hard, and it's a huge blow to my confidence. So it makes it tough for me to want to go out to meet people or to parties and stuff, because I know that if something happens I'll have to face that moment. Where she'll be like "oh". What do I do? Do women really care that much or am I overreacting? Are there any women out there with husbands or boyfriends that are small?