Parents don't want to make the effort to bring them to rehearsals and later, football games and contests. Parents are afraid that their kids "won't like them" if they make them follow through with something that they started.
I bet that all the same kids who are in athletics won't quit. If they even think of quitting, their parents will make them stay in so that the parents can live their lives through their kids.
I'm retired now, but for me, I got tired of the excuses, quitting for no reason, etc. Towards the end of my career, I never really counted on a lot of kids actually staying in band and being willing to work. That way, I would not be disappointed and depressed. I always planned my music and show as if most would quit. When more stayed it seemed like a bonus.
Also, the strange thing is, once I got fed up and stopped all the extra coaxing and just did standard recruiting, more kids stayed in band. But, it was a really weird football is everything type of town.
I wish I could give you advice, but I was stumped myself. I just retired as soon as I hit my magic number and never looked back. But I still feel for the band directors such as yourself who are going through all of the nonsense. And some of these towns wonder why they have trouble finding directors when the position comes open. Or maybe they just don't care. I don't know.