I would say that's a self confidence thing. I do the same thing.
In the book I have, the anxiety and phobia workbook (SUCH a good book!), there's an entire chapter dedicated to asserting yourself. There's a few pages of situations where people usually can't say no and you have to write down ways of saying no, and it gives you suggestions of what to say.
I am similar to you, I would never say no to anyone or speak up if I wanted something, I would just feel soooo awkward about saying anything! It needs practice practice practice I would say, and some sort of confidence in your right to say what you want.
Now where do I get me some of that?!