You own the property you live in. You work there alone, so you are not using your house as a brothel. In this instance, you are not doing anything illegal - sex work is perfectly illegal in these circumstances.

Next time they tell you they're going to call the police ask them why, what are they actually going to complain about/report you for? See what they say first, I bet they haven't a clue about the legalities.

Acceptance from neighbours isn't a given, and you were lucky with your previous neighbours. Not everyone is. It might be worth talking to them and finding out what they are objecting to. If it's footfall, strangers parking in your street, it might be worth asking your clients to park elsewhere if only to placate your neighbours as there is a possibility they could make your life difficult and hassle you relentlessly, even though you are doing nothing wrong.

How did they find out you're working there?

I would be wary of winding them up since you'll have to declare any disputes with neighbours if you ever want to sell the place - this certainly includes anything that results in you reporting them. The danger is that when when they find out they have no legal footing at all, they'll start pissing about trying to put you out of business - standing around making comments to punters as they're arriving/leaving, photographing people and their cars and so on. Do you know if they're renting (so might not be there as long) or have they bought their place?

I'd agree about the parking elsewhere (so they can't whinge about it, and for no other reason) and just quietly get on with it; if they've not been there long it may well blow over when they see you're not interested and that nothing you're doing has any real impact on them. I daresay they'll find something else to froth about before long