You are not crazy. In my opinion, if you determine that this is a good opportunity and you are offered the job, take it.
I understand that you don’t want to leave your current program in a difficult position, but it sounds to me like you have left things in good shape for your successor. This is part of the business of being a band director. Your administration should respect your decision and do their best to find a good replacement. It’s their part of their job. Good luck!