It isn't important to make direct eye contact. Nobody is comfortable with that. In many animals it's experienced as a challenge or threat because it's uncomfortable. In many human cultures it's considered impolite as well.
Think about the intent instead: what you really want to communicate to the person is that you're paying attention. You don't need to look them directly in the eye or focus on anything at all to do this. You only need to look them in the face every once and a while as your eyes dart around naturally. Unless you're very close to them it's hard to tell whether someone is looking you dead in the eye or just somewhere on your face.
If you feel the need to look away when trying to form your thoughts into a sentence, that's entirely normal, too. You're generally trying to juggle watching, listening, and speaking, all of which require a lot of attention to do. Instead of worrying about keeping them all "in the air" at once, try to get used to the idea that you're going to be rapidly switching between them without thinking.
(When you're anxious, you're consuming most of your attention with very inward-focused thinking, which makes it hard to do any of those things adequately. You may be trying too hard to mimic what you think is normal, instead allowing the most natural thing to happen and accepting it.)
The other part is when you're speaking, you want to make just enough eye contact to gauge their reaction, and see if they are listening. I've noticed this usually happens right after you finish forming the sentence in your mind but before you finish speaking it. You don't "decide" to look, but it's like suddenly your mind nothing to do, so you're free to take a glance.