I make better money than I did as an associate director, and I have better retirement, health, and benefits. My current students couldn't hold a candle to my former students musically, but they are getting better everyday and still experiencing the benefits of a formal music education. We do marching band (competitive, but less so), concert band, jazz band, percussion ensemble, colorguard and winterguard.
This situation does require substantially more advocacy, and I am constantly battling archaic mindsets on the role of the arts in education. There are certainly down-sides, but spending time with my kids is NOT one of them...
There are other ways to exist in this career than chasing "fame" and recognition. It's a better life when you can enjoy seeing your own kids grow up too.