In 15 odd years of working, I have only spotted 1 client outside of work...and it was in a totally random airport hotel, miles from where we originally met.
I have also only ever encountered 1 client I actually knew from anywhere in real life and it was a weird one because it was the dad of a childhood friend i'd long lost touch with.  so when they knew me I was 10 years old. They absolutely were in my opinion wondering, as if they could sense I 'knew' them from somewhere but obviously I looked a lot different - They booked a few times actually in the end  I never let on.
The reason for me is that I have never worked in the same town i live in, for the exact reasons you describe, I just don't want to be recognised or see anyone I know/friend of a friend etc, for me it is just too stressful and I don't want to be outed but everyone is different.
You could consider wearing a wig / disguise to work just in case and always, always check the peephole on the door before you answer

I have worked in small communities, I have had indiscrete punters - one who hadn't seen me in the town I moved to spotted me shouting my working name across the road in a very sensitive place. When I ignored him he shouted again saying "I am sure it's you". This is a man who on the couple of times he booked me was terrified of being found, told me privacy and discretion were for him paramount. I had to go hide around the corner before heading to my destination because combined with his behaviour (after becoming unreliable, then a gap of a year or two had been contacting me for a while trying to get around some of my new booking requirements), and not understanding why this was all undesireable.

Someone not too far away who I was stupid enough to offer a small discount for minimal travel ran up to me in the street calling me by my working name, and despite being given a name to call me when next saw me decided to play a game of 'what name shall I call you'.

HOWEVER many clients travel into the area either for work or leisure reasons or paths simply do not cross. Infact I have bumped into clients a 90 minute drive away in completely random situation - also 90 minutes away from the place I know they live. Many live several hours drive away, others the other end of the country. For discretion and privacy reasons I actually prefer outcalls to homes rather than incalls or hotels. Historically I have had so many be indiscrete at incalls, and to some extent some hotels. I also have a discretion speech when accepting their booking, if they are localish I explain what to do if we do bump into each other.

My ongoing issues in this small pond  :-one punter, in building trade, with a large "showy" vehicle, with the trade name etc. deliberately park in the street (indiscreet-experience of previous harassment) -where I asked him to avoid going there-likely he wants some business (?!)    :'(  Next time, I might as well suggest him to use my parking space that everyone could see.  I wouldn't.

I began to think that if punters are asked to park away, then it might attract more attention (?).  I had some major grief in the neighbourhood over the years - I was made to feel as if I was a criminal - extremely low-level work, but with "professional" curtain twitchers, who got nothing better to do than passing judgement.