While I'm no expert on carrying a conversation, the main thing is just to get them to elaborate on whatever they're talking about.
For example, if you go into work on a Monday, ask someone how their weekend was. They will probably just vaguely describe it as either good, fun, bad, etc. If they say it's good just follow that up with asking what they did. They might mention a bar or someplace they went to. Then just say something like "You went to _____? I've never been there but heard it's pretty cool place. How'd you like it?" If you keep it up eventually they might talk about something that really interests you. Once you find that it's much easier to carry a conversation.
That's what I've been trying to get myself to do this week, and so far it seems to be helping.
If you have trouble thinking of stuff to talk about, just make sure you follow the news. I usually check the news online every morning, especially for the funnier/odd news. It's always a good conversation starter for people you kind of know.