You switch off on holidays..............
There may not be a whole lot you can do about this holiday, but I strongly recommend that (in the future) you not try to go to both families in one day. Because with the schedule you just mentioned, it will only get more and more difficult as the years go by.
Just a suggestion: create a family tradition in YOUR (you and DH) family, like always spending Christmas morning at home, no matter what. No one, not even your extended families, can fault you for that. And even if you do hear some grumblings, ignore it.............we're talking about YOUR kids, not theirs. Then rotate holidays with extended families, like yours on Thanksgiving and his on Christmas, then vice versa the next year.
I never have figured out why there are families who make one feel they have to make it to their house on a holiday, then you end up almost spending more time traveling then you do with family. What's the fun and tradition in that? We're lucky, in that both DH's family and mine are all in the same area, but if holiday celebrations just happen to fall on the same day and same time (or in close proximity of time), we only go to ONE gathering, instead of trying to overextend ourselves. DH is one of 6 kids and it's rare that all of them and their families are together on a holiday because they also have other extended families to see and that's understood.
Have the gathering at your home and tell the in-laws, explaining that because of all the kids being together, it is that much more difficult to be so many places in such a short time, which is the truth. If they don't understand that, after raising a family of their own, then their refusal to come to your home isn't your problem, it's theirs. I think you also need to sit down and discuss this more with your DH, as you BOTH need to be on the same page with this or it will never work.