The current situation is that you're selling a sexual service, so legally, you're a prostitute just as much as someone offering sexual intercourse for money. You're thinking about hiring other prostitutes.
If you actually did that, you'd be controlling prostitution for gain, just as with an escort agency. Have a place to do this in and you'd be controlling a brothel too. (If they've never done anything like this, you'd also be inciting prostitution, but that's easy to get around.)
Will anyone care? It depends on how much you annoy the neighbours as much as anything. As you've noticed, lots of people do much more than that without getting into trouble. They could do so, but there's a limited budget for policing prostitution and some rather higher priorities than going after small operations that don't annoy people. The police in the London borough I used to work in issued a planning document saying they would raid one brothel per.. I think it was per quarter. There were at least five brothels on our project's street...
Will you get into trouble? Very probably not. Could you? Yes.
It's up to you whether you want to take that risk. When it does go wrong with this sort of setup, it's usually because some police operation fails to find what it is looking for and decides that it's going to convict someone of something and brothels are an easy target for that.
(Not a lawyer.)