Yes, I was berating and beating myself up after the encounter thinking why didn't I say exactly what you've typed here Ana, like NO its not f*cking ok for him to do this. Guys, all your advice has got me seriously pumped up, I actually can't wait for him to try me again or for some idiot to put their hands on my hair, I'll show them that I am not one to mess with going forward. I know it won't be easy but I'll need to just override the fear and speak up cause the sh*ts will continue and I'll hate myself for being such a p*ssy
You're not a pussy, it's because you are concerned about him knowing what you do and it's slightly overridden the fact he's a creepy weirdo that actually shouldn't be invading your space and making you feel uncomfortable and worse still, unsafe.
If you weren't escorting how would you react to him lingering around? You'd probably say something. He is WAY overstepping and should know better than to be on your doorstep unless specifically invited, that's just wrong on so many levels.
You don't need to be rude, you can ask him politely to leave as he's making you feel uncomfortable and it's YOUR house and he has not been invited. If he then continues you should take it further. As previously suggested, recording the initial conversation is a very good idea.