Is it worth the extra practices and extra Saturdays you have to give up?
Drumline is a choice. If you do not want to do it, don't do it.
You need to make sure your administration is good with the decision and you need will need to explain to them why you want to eliminate it.
If you eliminate it, be prepared for the ill will it will cause for a period as the students and parents who want it will complain about not having it and possibly drop out of your program.
You can have a well rounded program with or without indoor drumline.