I'm not sure how long you have been doing this for, but I suspect you're rather new to it- I think most of us have a moment when we feel ripped off or we've blurred lines or not been clear about what we want. However savvy and in control we feel most of us have made mistakes especially in the early days and it's those experiences that make us better and make us realise that you have to set boundaries and stick to them, get money up front, establish the nature of the booking and never give them more time than they have paid for unless you are clear it is a one off. The nicest, sweetest clients will end up taking the piss or just not realising they are diddling you out of money.
Don't think you are silly at all, he was a using arsehole.
In a nutshell it's a lesson learned. Just don't do it again!
I don't think he'll pay you back to be honest; I hope I am wrong. Ask for the full amount, the full 6 hours, you can say as it was social time you will discount from your usual rate. If you get it all excellent, if he gives you some then that's still a small win and if you get nothing then move on and remember not to trust anyone again! Don't bother fighting him over it in case he gets nasty and maybe just say that neither of you should contact the other again.
If he does just ignore and this can be your practise in getting tough!