While I am definitely an advocate for sparring, I think you should look into a ref Self-Defense courses. Especially those that emphasize environmental awareness, and discretion (like the type they use for Firearms Safety training). I think the "fear of confrontation" you are feeling is something that isn't necessarily a bad thing. Likewise, learning how to disregard it can be life threatening.
While I can clearly attest to being willing to step into a sparring session at the Academy at drop of the hat, I would be weary of being "confronted by a 12 year-old anorexic girl" also. While I know full well that I may get beaten up while sparring (and I have jumped into sparring sessions expecting - and receiving - exactly that), I have no idea what this person in front of me has in store and that is unnerving for a very good reason.
The large difference being that even if I get put in a bad way at the hands of my training partners at the academy, I know they aren't going to hurt me overtly. While an anorexic 12 year-old girl may not be shit to me if she throws on the gloves, I don't know if she's got a knife, I don't know if she's got a gun, or I don't know she's got a group of her buddies or family members right around the corner/across the street waiting to jump on me and stomp me into the pavement. While my sparring partners don't intend to do me grievous bodily harm, I can not say that goes for XYZ person out on the street.
At times like this, I always mention Alex Gong
. While fearless and more capable than the majority of the people on earth in a fight, his lack of discretion cost him his life when he pursued a guy who hit his car. While confronting the man he was shot point-blank in the chest. While sparring will help lessen fear, it does not instill a large level of discretion. Even as martial artists who are confident in facing fear, you can not servive long in this world if choosing to forego discretion.