I normally don't charge if they change their mind quickly (i.e. within the first 5 minutes). Anyway, with most people who don't want to proceed, you can tell pretty quickly and I give them an easy out telling them they are free to go with no payment required as I don't want to be in the company of someone who doesn't really want to be with me.
But in this case, seeing that he messed you around by arriving 45 minutes early, seemed to be okay, took a shower and then changed his mind, I would have kept the fee.