I am also of the opinion that dislike of wearing condoms is largely psychological.
We have all probably had some experience with clients who do not have a chronic problem but occasionally could not get hard, or could get hard but not come at all, due to nerves or stress.
I also recall a news article about a study that men can even orgasm mentally without physical stimulation. Sleep orgasms is an example, so obviously there is a large psychological aspect.