I think both can be equally bad. If you stare too much into a person's eye, especially that of the opposite sex, the other person might think you are interested in them. If you look away too often, like what the posters have mentioned earlier, it could make you appear disinterested in what the person's saying, and that's equally as bad.