When I was going through school, I had a part time job in retail. I didn't go to full time until I graduated. I just couldn't handle both at the same time. I was absolutely terrified to work in retail because it is largely based on people interaction and I sucked at it. But I ended up actually enjoying it very much. It wasn't someplace with crazy amounts of customers, like Walmart or anything, which was a major plus. I was able to ease myself into it without rushing.
If you are truly fed up, I would definitely consider leaving. Finding a new job may be stressful, but staying in the current one sounds like it is as well. I'm in a similar situation. I grew immeasurably annoyed with my boss, so I quit my job in October. I've had quite a few ups and downs in the last 4ish months and haven't found a new job yet. I'm trying to find something that suites my limits, which is hard. I think you need to find your limits and search for a job within them. I hope that made some sense.

Good luck!