Okay that was a poor choice of words, obviously you can't stereotype people by their nationality, but cultural influences do have in impact on people's interaction with each other, as I have noticed in the many many places I have rented, and various nationalities of people I have rented to/from, mostly outside of escorting. I definitely wasn't suggesting the OP is somehow to blame for this man's behaviour.
It seems very clear from the fact he spoke up in a defensive way "it's okay she just likes to play with the play with the pebbles" that he was well aware he was in the wrong; that is very defensive - not normal neighbourly conversation at all. He knows you know he's up to something.
Definitely move when your lease ends... even if nothing happens, staying put may cause you stress and paranoia. As far as what to do in the meantime, before the lease is up, that really depends on the potential consequences of this situation escalating: ie. depends on the legalities of escorting in your country.
One option would be to move and sublet the flat for the remainder of the contract term, again that depends on the legalities of subletting where you are. Seems like your neighbours would potentially grass you up to your letting agency or landlord if you sublet and tried to hide it.
More immediately, it might be worth putting up a dummy CCTV camera over the entrance way? They cost £5 i think, it might put him off hanging around.