When all of us are on our death bed, 99.9999% of people are not going to say "Man I wish I worked harder/longer in life".
It will always be "I wish I spent more time with my family".
Also my first 5 years teaching I was an HS assistant and every head director I worked with had a bad personal life (divorce, couldnt see their own kids much, lots of regret) or wish they were doing MS.
My advice: Do it as soon as you can. Your wife and kids need you more than your students.