I'd suggest to record all conversations with this creep on your phone voice memo app & back it all up to you pc or external hard drive.

I'm gonna speak to him but ask a friend come with me just in case.

A witness is great too.

Reminds me of a situation that I was in nearly 20 years ago. Although I wasn't working from the flat, I was doing outcalls. There happened to be 2 guys on the same floor as me & the younger one was being paid by the older guy for sexual favours, I later found out & he'd been in prison previously. This younger guy stuck his number through my door. Who knows what his agenda was.  In the end I moved in with my boyfriend at the time to get away from them all.