I don't ask for any verification going to a client's home address. I go by how he sounds and the voice I go for is older, businesslike, polite etc etc etc. Never had a problem. I do Google Earth their address though just to see what area it's in. I've often wondered if I should be doing something else though.
With hotels, I get their room number and call them via reception.