What I will say is, I (and I think many directors) love/like 70-80% of my job. That other 20-30% I don't really care for, but it's my job so I do it.
So my point: is your job really only 20% things/situations that you do NOT like or make you have anxiety? Can you truthfully answer that more than 50% of your time is bad/miserable? I know there are situations were that is true. And I know that one person's misery is another's challenge to overcome. It's up to you to decide what you can/will handle.