If you get bad vibes, don't do it. That is my motto.
There have been times even after putting the phone down, I thought, hell no, I don't feel comfortable so yes I would text (immediately I hasten to add!) and say, oops, just double checked diary and it's not possible. I know this isn't exactly highly professional but then again the bookings I turn away, I doubt the word professional means much to them.
At the end of the day, I have to go with the way I am feeling and if it means cancelling or saying no, then I do it. If I get bad vibes, I will try and keep a guy talking on the phone as long as possible to try to get a better feel for him. Ask him lots of questions about your profile, your site - to make sure he is fully aware of what you do or don't do.
Maybe I have lost some good bookings but at the end of the day, it's how I feel and perhaps being in the business a while, gives you a better feel for who you want to see and who you don't and you can almost categorize by hearing an accent/voice, if that makes sense.
Jenny