Yah, you need to be out of this relationship cuz you are obviously not poly, and he's not poly either; he just wants his cake and eat it too. It takes a rare person to be truly poly.
It would be preferable if you could extract yourself from the whole situation so you will never see him again... like move out.
I am convinced that when anyone feels that they are in love with someone, what they are really feeling is that they love themselves better because of all the things they imagine about this person.