When I take deposits it's to secure the specific appointment being discussed. If the appointment doesn't go ahead due to something on their side (cancellation, ghosting, "confusion" etc) I keep the deposit. I'd only refund if I needed to cancel
So from my perspective he wouldn't have been entitled to a refund when he tried to change the nature of the booking to something you're not comfortable with. He could either go ahead with GFE or cancel and forfeit the deposit. Similarly, he would need to pay a 2nd deposit to secure the new booking, you're still down the balance of the 1st booking so I wouldn't give him the chance to do the same thing twice and waste my time.
I am similar, one of the most annoying things I find is a booking is made - outfit/roleplay/specifics discussed then it changes several times. Also I have accepted bookings without any suggestion of the BDSM services I offer, then a week or so before he contacts me to ask for BDSM. I now ask for any specific requests before I accept the deposit, specify requests have to be made either at the time of booking or no later than 1 week prior.
However if he cancels because I won't or cannot do something he asks for after the deposit, he loses the deposit - hasn't yet happened but I would not refund.
What I sometimes do if a person has been straightforward when setting up the booking, if he cancels due to say having a cold or covid I may agree to transfer but require a second deposit. I then subtract both deposits from total to pay, but if he cancels both deposits are lost. This is due to me being understanding in the past, then the rescheduled booking is either cancelled or once again changed.