PP, if you must keep your phone on then, even if silent, put it in another room. If you hadn't responded then he wouldn't have gotten a reaction (which he did).
I always try to keep it so that they won't go down the "change number" stage because then you are more paranoid about every number that comes in especially if you have nt spoken to him and don't know what his voice sounds like.
Even if it did start before midnight, if you are not working, then put your phone somewhere away from you you, even if it's just for headspace and time for yourself.
If I'm right, you don't work late so no need to have phone near you. We all have to relax. When I was whoring I very rarely answered a call after 9pm. If they wanted me they would get me another time.