I can relate.
I was talking to an ex once (who I Used to sleep with on and off) and he said something like "I know things about you that you might think I don't know."
He wouldn't tell me what and it pissed me off and confused me because I didn't think he would be the kind of guy that would ever ever accept this job. I still don't know to this day and I don't think I want to.
If you want to shag your ex go for it! I use mine as recreational sex and when I'm feeling lonely or a bit fed up he's there. Do you want to get back with him?
You could always ask him outright if he knows?