You own the property you live in. You work there alone, so you are not using your house as a brothel. In this instance, you are not doing anything illegal - sex work is perfectly illegal in these circumstances.
Next time they tell you they're going to call the police ask them why, what are they actually going to complain about/report you for? See what they say first, I bet they haven't a clue about the legalities.
Acceptance from neighbours isn't a given, and you were lucky with your previous neighbours. Not everyone is. It might be worth talking to them and finding out what they are objecting to. If it's footfall, strangers parking in your street, it might be worth asking your clients to park elsewhere if only to placate your neighbours as there is a possibility they could make your life difficult and hassle you relentlessly, even though you are doing nothing wrong.