This really does sound like just a bit of casual fun. If you had met in any other circumstance, and it was just all about sex, would you still question if it was going anywhere?
I went on a date with a client last week, and that's exactly what it was: a date. He took me out. There was no mention of sex. He didn't look at me like a hooker. He wasn't sleazy. He's asked me out again, and has planned another amazing date. If the guy was interested in you really, he'd be taking you out and doing something wonderful that he knows you'd enjoy.
I'm sorry, but you need to call a spade, a spade.