An awful lot of places are brothels. When what became the Sexual Offences Act 2003 was being debated, and the penalties for running a brothel were going to be increased, the owners of assorted 'sex on the premises' venues went 'oh shit' and did some lobbying. Hence there are now two offences: the rarely prosecuted 'owning or managing a brothel' and the 'it depends where you are' 'owning or managing a brothel involving prostitution'.
As far as duos are concerned, if both of you go to an outcall it's not a brothel (or, if it is - Max Mosely's flat probably counts, for example - it's not your problem). For incalls, it's who owns and/or controls the place. Who decides who works when? Who does the advertising? Who...? For standard classic brothels, the answers to those are easy and there's usually a clear manager. For two people sharing a flat or a serviced apartment, it can be harder. Certainly the one who signed the lease / contract is in the firing line, but it becomes easier to say 'you both are'.