If you haven't already (sorry, can't remember) I think both you and him need to attend therapy, separately, to resolve these issues.
I've had painful, toxic people in my own life that I've felt dragging me down, dragging me down, dragging me down... and then I'd look at their faults and...