Depends where. In central London you can easily get away with it. When I town terrace I rarely saw my neighbours, and would have new neighbuors every year or so. Most working people are so absorbed in their own life they wouldn't notice in some areas.
You may have to consider the thickness of the walls/floors/ceilings too!