They may take offense if you tell them the ugly truth. You might get some abuse or something else. I generally dislike "regulars" . Even if there was some chemistry to begin with, it's hard going, I find. Avoid having regulars.
Personally, just keep telling them you are not available. If they persist, avoid them and block if they go on. Men don't take reasons. They just go mad or give you some grief, blackmailing etc. just tell them you got so busy.