I was in a similar situation and for this reason avoid leasehold and or small blocks of flats, LH usually contains clauses which means you can be asked to leave.
A friendly neighbour had a word with me, the residents had asked for reports on my activities. The problem was that I did also have a chaotic lifestyle due to an addiction problem, which was leading me into very poor company. I was not a great neighbour and would be clumsy when inebriated which added to the indiscretion of clients. Punters would let the front door bang which could be heard throughout the block, run up the stair waving cash shouting my working name in greeting, or back as they were leaving. A few parked in neighbours spaces (the car park had had a history of locals using it instead of paying for parking in town so residents already had an eye on it before I ever moved in), one or two ran into bollards I pre-warned about. When the buzzer of the entry panels was broken despite me explaining how to get to the other one, a few rang other doorbells asking to be let into 'my working name' flat. Even when not broken their were other similar indiscretions. The buzzer either wasn't or took a long time to be fixed by the landlord, the company doing the repair needed me to be in but didn't turn up at the times they asked me to be in.
Heck that's brought back memories.
I started looking for other properties and asked clients to park around the corner, around the time I found somewhere else I was at the same time given notice without any reason - because they didn't need to. I was glad to move although I then found another neighbour who would pounce on visitors, but that wasn't specifically targeted at me or any suspicion of sex work, and the few who were approached were just fine (they just acted as if they should be there which was absolutely correct).
Advice for the OP perhaps look for somewhere else, but my advice is unless you see somewhere you really want to move to, carry on and the best piece of advice I was ever given is "Don't piss the neighbours off"............and keep your dignity. That flat was spotless when I left it, had everything professionally cleaned and anything damaged I replaced with same or higher standard.