In my opinion the correct answer depends a lot on your circumstances.
For context, I do only outcalls and this is my only source of income. I live in a tiny place (20.000~ people), which neighbours another equally small town. It takes 1 hour to get to the nearby town, but I'm happy to take outcalls to the area assuming the appointment is long enough for my liking (2+ hours). All I personally ask for is to have my travel costs covered. This works just fine for me, because I'm not really missing out on any other income - in fact, I'd be missing out if I didn't take the outcalls to the other town.
With that said, as Miss K said, your client's a cheeky fucker and I'd plain refuse to see him again for docking my pay.