I don't think being a bit older (at least up to 60ish) will mean you aren't busy, as there is definitely a significant market for 50+ escorts, both of younger men and those who feel more comfortable seeing someone closer to their own age.
What I would consider is:
1. The menopause
My libido, which was admittedly high beforehand, plummeted. I know we don't need to be turned on or feel 'sexy' to escort, but I think mentally it can make a big difference to motivation and job satisfaction. My natural lubrication also almost totally disappeared over 2–3 years, and while lube obviously fixes that, you may not be able to physically handle as much PIV sex, or at least prolonged, rougher etc.
2. Being 100% reliant on escorting for income
After being made redundant, I escorted full time for 4–5 months before setting up as a freelancer in my original profession, and since then have combined the two. It's worked very well for me, as I could escort more when my other work was slack, but drop it when I was busy. It also meant I had another stream of income, and I think that protects you a little from making bad decisions about who to see/what services to offer, e.g. seeing a client you don't like because you're desperate for cash.
3. Friends & family
More minor, but what would you tell friends, family, neighbours etc. you're doing for a living if you escort full-time for more than a few months. Any issues with a CV gap, mortgage, banks etc. etc.?