Yes, I let a client out at the exact moment my neighbours came out of their flat. Luckily, I have a policy of saying goodbye away from my door as we walk down the hall and then I hold my fingers to my lips as I open the door while hiding behind the door. So they didn't see anything except a man walking out of my flat - I wasn't visible and he wasn't heard speaking to anyone.
I try to keep them as quiet as possible coming and going. If they see the very occasional person, it shouldn't add up to anything in their mind.
The bigger the building, and the more flats on each landing, the less such things are noticed. I also make a point of avoiding interacting with my neighbours. In real life, I'm likely to be friendly and neighbourly. I avoid talking and eye contact altogether.
I am also prepared to move flats at the end of my lease if I think such instances have occurred too often. However, while that could get me a clean slate in a new building, I may end up next to someone who is home all day and thinks they're the Neighbourhood Watch.