IMHO, it depends how he is approaching the subject.
If he is a nice friendly guy who is just chancing his arm a bit and being a bit too pushy, I'd just tell him politely but firmly that it would be unprofessional for you to do that and you prefer to keep work and personal life seperate, but that you'd be happy to do XYZ with him at your social rate.
If, however, you think he's pushing too hard, is at risk of getting sinister with it or is making you uncomfortable to the point that it's not worth the fee from him each time, I'd find a nice, polite and non offensive way of stopping seeing him. Keep it as nice as possible though.
At the end of the day, it's all about how you feel. I probably wouldn't lose a regular just over him asking, but you know how uncomfortable all this is making you. If it's at the point where the discomfort of this isn't worth the fee, then you stop seeing him.