I know many perfectly well adjusted adults whose parents were band directors and they grew up in the band hall. They are good musicians but not necessarily all band directors.
However, you're not going to get the time back with your kids. If you want the time to be at home, that's fine. It doesn't hurt them to be around band events.