It's a valid issue. You wouldn't need to worry about any component of your motion that's "constant velocity" (so, inertial straight-line motion), since all inertial reference frames are equally valid. But you would need to keep track of curving motion (like the rotation of the Earth, orbit around the Sun, Sun circling around the galactic core, etc.).
You might be able to wave your hands and claim that the time machine follows paths back in time along natural world-lines in curved space-time (gravitational geodesics), which will at least get things like orbiting around the Sun right (and galactic motion as well). But I think you'd still need some way to keep the machine's path rotating on the surface of the Earth rather than falling back and forth through the center of the planet.
I thought it might follow backwards in space as well as time but when the book The Physics of Time Reversal by Robert G. Sachs came out in 1987 it made me skeptical of that possibility. I haven't followed the theory of CP violation for many years so this may not be a pertinent point anymore.
For most "time machines" you read about, you aren't really going to run into the "CPT" issues from quantum field theory.
Unless you're considering a model of time travel where you're still literally in the same room and just living in reverse (while still visible to all the forward-living people around you and interacting with them as you go), this isn't equivalent to CP symmetry reflection.
Most time machine ideas instead picture some sort of wormhole or "stitching" between different points in space-time, in such a way that the local "forward time direction" remains continuous for observers taking the trip (and then connects back up with forward-directed time at some point in the past).
(I'm teaching a low-level course on "Time Travel in Science and Literature" this term. It's fun, but a challenge without using math.)
Maybe time travel needs some kind of tracks, like a railroad. That would also explain why we don't see any time travelers - they can't visit us as there are no tracks that lead back to 2014.
You might be able to wave your hands and claim that the time machine follows paths back in time along natural world-lines in curved space-time (gravitational geodesics), which will at least get things like orbiting around the Sun right (and galactic motion as well). But I think you'd still need some way to keep the machine's path rotating on the surface of the Earth rather than falling back and forth through the center of the planet.