I don't keep punter numbers for any of those reasons; I keep them because I either don't want to see them again or because they're good, reliable regulars who I know well and will prioritise. I don't share numbers with anyone unless I believe a punter to be a risk.
There is no reason the police would be interested anyway, but then the majority of posters here are in the UK. If you are not then you may have more reason to be concerned about LE, but regardless of that, keeping your phone contents secure is your responsibility and nobody else's. How would 'scammers' get hold of your numbers unless you let them?