Every. Single. Day? Hrm. I'll try to hold back my not-very-helpful urge to do a long-distance psychoanalytic review of this poor chap and just say that "too often" is any point where YOU feel that it's too often, because of discretion or obsession or neighbours or any reason.
Perhaps explain nicely but clearly that you're worried about your neighbours and you can only see him once or twice a week (or whatever amount suits you best). You don't have to see someone just cos they ask you. It sounds to me like you need to be *really* careful about being *really* discreet if you've already got gossipy neighbours who have literally asked you about him already. To be honest, their asking means they already have their suspicions, probably.
If you have children that you want to protect from the possible fallout of people finding out about your work, it's likely a good idea to perhaps stick to outcalls for a little while. Maybe even consider moving? Sorry, that probably all sounds very drastic, but there are other posts around this forum about nosy neighbours and the disasters that can too-easily occur because of them. You must be very careful.