believe it or not it's quite common, i have seen an ad about a psychiatrist who was especialized in fear of driving (it's in brazil so not much help for you)...
i didn't drive until i was 21, because i was afraid yes but because it's kind of expesive to do make a licence here, at the time it was a couple hundred $ more then i made a month and i had a fairly high salary...
When i did got the guts and the money, it was difficult, i got nervous (i had two instructors and one was a really pretty girl so...) and the first time i went on the road i would frequently let the car die... but i tryied and i did it... i managed to get both my car and motorcycle lessons, i passed the car test in the first try, but i had to try two times with the motorcycle...
today i drive my faithful old car everyday, i wont say i'm the ace of the wheel, i drive carefully, i know my litmitations , i pay much attention and respect every single signal, it's been three years now and i did crashed or ran over someone.
I wont say i don't get nervous when a speedy gonzales keep tailing me, or when i'm in heavy traffic and i need to change lanes, but i'm careful , very very careful. my phylosofy is, when in doubt go slow and don't risk it... at least here, if someone hits your rear its their fault not yours...
Actually i think learning to drive made me much better, now i care much less about other's opinions...