History, particularly in the last 30 years where political polarization has significantly increased:

https://en.wikipedia.org/wiki/United_States_midterm_election...

Trump lost 41 seats in the house in his first term and the administration wasn't anywhere near as polarizing. The voting bloc for mid-terms is very different from presidential elections.