I used to get really anxious about it but not so much anymore (unless days and days have passed and I had send a message which invited further contact, such as if I suggested a meet up).
I used to reply to things straight away on the premise that I may forget later, but now I've become a lot slower at responding. Being busy is the main reason, but also because sometimes I'd like to reply to a message in detail, and take more time and care to write it, rather than just whittle something off quickly for the sake of replying sooner.
It's strange, the slower I've got at responding, the less anxious I am about how quickly people respond to me. It's common for people to worry about why so-and-so hasn't replied to a text, but the truth is that there are a huge number of reasons. To sit and worry about it is a waste of time. Most likely, they're busy doing something else.