I wish I had the same sort of freedom you have when I drive, that's even if I do have the guts to drive.
But yeah, I don't like to so much if drivers behind me looking, especially more so when you end up stalling! I fear or get anxious driving in and around town, seeing people I know and fearing I'll end up stalling or hitting a car and then they laughing at me!