If it is an outcall, he might have said that so that the neighbours think that you are a friend or colleague. Turning up in a tarty miniskirt, 4 inch heels and fishnets might get the neighbours talking.
If it is an incall, it might be because he fantasises about women he knows or works with - unlikely to wear Ann Summers lingerie and short skirts. As others have suggested, ask for more details.
When I have been asked for the same with no detail, I go for office work type clothes and probably a skirt rather than trousers - which I think is sexier and allows a stylish strip down to undies without looking slutty at the start. Opaque tights and flat heels bring civy realism rather than the standard hold-ups and heels of the profession.