For me it's about instinct. If I feel someone is just a pushy jack-the-lad boundary pushing wanker then I will be very firm and act pissed off and ask them to leave (even if I'm slightly intimidated I'll pretend not to be fazed and I'll say something like "I've already told you I don't do x,y, or z so leave now"). If however I feel really weirded out or I feel they are creepy and/or dangerous and that I'm actually potentially unsafe, then I will act firm but all sickly-sweet and nice and make an excuse like "I really need you to go, there's been an emergency in my family and my sister is coming right now, I'm so sorry , I will certainly make it up to you next time") and on the one occasion that didn't work, I had already called in my security and they were on the way and primed to call he police so I looked at the person square in the eyes (even though I was TERRIFIED inside) and told him that when I pressed that button on the phone the police had now been called and are on the way so I suggest you get out right now.
PP - I always have my security buddy either on the premises or within a button push away of being contacted. Even with regulars cause occasionally they can turn. If I feel the slightest bit on edge or suspicious or that feeling that something isn't right , I press the button to call them and they make their way to me asap and they have a key. If I missed call them again or obviously if they can't get in they call the police. I just do it for every booking as part of a routine.
xx