I'm speaking as a massive cynic when I say this (based on my escorting and personal experiences in the love/men/sex department), but fuck it - if you can get away with it and are happy enough then carry on and make sure you don't get caught. If possible, I'd divorce him in the long term though.
I don't know about others, but the majority of my clients have spouses and so for me, I don't see how anyone can pass judgment when we are making a leaving, either fully or partially, off (mostly) men that cheat.