I am exactly the same. I live and work from home so have to be selective especially when I had a regular client in the past who somehow got into the building and came knocking on my door after I had blocked him for crossing the boundaries. I always go with my gut when screening as I hate having a stranger in my home making me feel uncomfortable or threatened.
If I don't like the sound of someone, the way they come across or just have a gut feeling I will straight up tell them "sorry I don't feel comfortable seeing you". A gentleman will understand this and not pester, but those who pester are just insecure and can't take no for an answer let alone respect my decision. So they start abusing and swearing-my gut is right when they behave like that, all the more reason not to see them. Trying to justify my reasons to them only pisses them off more, more pestering, more abuse etc. Best thing to do is just ignore them, they usually go away.
If they behave like that the. They're probably not the type of clients who I wish to see in the first place. This way the ones who I do see, I am able to mentally be with them, feel comfortable, be able to do my job without hesitation and actually enjoy it. Always stand your ground.