TSB4 said:
“Am I totally wrong for wanting him to leave?”
It’s quite natural for you to feel like that. On the other hand, if he hasn’t always been like this, there’s hope that he will change. Also, is it harder or easier for you when he’s gone for a week? If it’s harder, maybe he is doing stuff to help that you don’t notice. If it’s easier, if everything goes more smoothly, the housework gets done better, and the kids are easier to deal with, well that’s very interesting.
Does your husband understand that the more love, attention, and energy you and he put into interacting with the kids, the more pleasant and manageable they will be? Conversely, the more he ignores them, the more annoying and demanding and disobedient they will be. If he only pays attention to them when they’re bad, they will get worse and worse.
I would suggest reading “Transforming the Difficult Child” and “Don’t Shoot the Dog” (they’re both books on positive reinforcement, with the former focusing on children).