Not at all. Having remote work experience has helped immensely with finding new and better companies specifically looking for remote workers.

Even if it had, I'd still take this extra time with my young kids over a raise. Every. Single. Time.