If somebody calls you at 4am and you answer it, then you can't possibly complain Joey. It's not a situation exclusive to you or male escorts, guys call us at all sorts of stupid times too. The difference is I don't answer phone calls at stupid o' clock and I've had weeks in the past where I've been desperate for a booking, so I do understand and empathise with you. But I have my boundaries, and having them is what's kept me sane all these years. If you pander to guys calling you at 4am, then yes, it's going to drive you nuts and you'll most likely end up with a twitch.
You and I know that guys call at stupid hours because they have an immediate urge they want satisfying. But as I said before, you've got to decide what you want to do. Stay home and answer the phone, or go out and leave the phone at home. Or at the very least, don't answer the phone when you're in company. What were you thinking?
You need to find a balance so that escorting isn't taking away from your personal life. You're mixing the two and it's not going down very well, hence your frustration.