It will never ever ever ever grow if kids are having to make that choice.The key to success at the small school - for both band and athletics - is SHARE SHARE SHARE and encourage your kids to do both. In high school they may need to make some choices, but in junior high they should be able to try out lots of different things.
Perhaps there are some perks to the area/school that you are in that make it worth it to stick around, but unless they figure out a way for kids to be in athletics and band in junior high you are going to lose most of them and have a really tough time building any kind of a program. Sorry.
Good Luck to you.