yeah this is the one thing I worry about, as I live in an apartment block. Like usually I am waiting by my door, tiny bit open and as soon as I see them through the peephole I let them in and shut the door quietly behind us but my biggest fear is someone coming in/out of their place as this happens. Did recently think though maybe if that happens, make it look more 'normal' by letting him knock on the door and depending on what he looks like/wearing etc, greet him in a certain way like 'thank you so much for coming to see me, please come in' or glad you found me ok, promise this wont take long, thanks again' etc etc. loads of things you could say to make it not look 'dodgy' and even make a point of waving or smiling at the neighbour who is maybe looking at you? I think you would be so 'open' or 'friendly' if you thought you were doing something 'wrong'.
Do know the uncomfortable feeling though as dont know about anyone else but I do care what people think about me, know I shouldnt but hey, thats me! lol. I just think now if anyone says anything my back up plan is, that I am a sex addict who likes to meet men for causal sex! lol Thats only my last resort back up plan though! haha x