In 15 odd years of working, I have only spotted 1 client outside of work...and it was in a totally random airport hotel, miles from where we originally met.
I have also only ever encountered 1 client I actually knew from anywhere in real life and it was a weird one because it was the dad of a childhood friend i'd long lost touch with.  so when they knew me I was 10 years old. They absolutely were in my opinion wondering, as if they could sense I 'knew' them from somewhere but obviously I looked a lot different - They booked a few times actually in the end  I never let on.
The reason for me is that I have never worked in the same town i live in, for the exact reasons you describe, I just don't want to be recognised or see anyone I know/friend of a friend etc, for me it is just too stressful and I don't want to be outed but everyone is different.
You could consider wearing a wig / disguise to work just in case and always, always check the peephole on the door before you answer