My first reaction is that you should report the matter to the police. The sooner and more you get it on record, the better case you will have to pursue legal action further down the road - if it should come to that.
Hopefully, just having the cops come knock on his door will make him decide to leave you alone and go pick on someone else. If it doesn't, at least you will have established that he knew he wasn't wanted there, and you can then get a restraining order if he continues to bother you.
Other than that, my only other suggestion is to call up some good friends who would be willing to beat the snot out of him.