This isn't an industry where you can always avoid conflict or please everyone. Unblock him, bite the bullet, apologise and stop taking bookings if you're on call. It's unfair to make commitments if you're unable to commit. We all mess up sometimes, just own up to it and you'll feel less anxious after.