I think we're on extremely thin ice with the suggestion that punters are more likely to abuse women than non-punters; the abolitionist lobby have been trotting that old 'rapists with an outlet' shit out for years. Some punters will be abusive, and so will some cyclists, gamers, birdwatchers and people with all sorts of other ideas about how they spend their free time, and this idea that men are basically no better than farm animals and unable to control their behaviour really needs dissolving every time it comes up - it demeans us all as well as excusing the real arseholes.
If I had a single regular who I suspected would be an abuser (and I realise they're not that easy to spot), he wouldn't be seeing me again. I have no doubt that I've come across a few in the last twenty or so years, but if I spent my days in any job where most of the people I met were men I would expect the proportions to be about the same.