It's nothing mystical or magical - it's your brain making connections, subconsciously, to signals that the other person is giving off, which your brain associates with danger. It's an incredibly powerful survival tool which we all have, but many of us (especially women) are socially conditioned into ignoring, along the lines of "But I know nothing about him, he deserves a fair chance" or "I don't want to upset him by saying I'm going to leave".
I know it's tough when you're skint, but you have to consider - if you end up getting attacked, you'll be out of action for X days, in which time you can certainly get another booking to replace the one that's making you question his motives.
I now look at it as, if I have doubts about the booking, even if he's perfectly okay I'm going to be on edge and twitchy, and that's going to mean both of us have a shit time. I'd rather pass on this one and wait for the next who DOESN'T set off my internal alarms.