That's a compelling case for why we should not use "distance", because distance cannot be negative, but the error term can.

Just look at bog-standard linear regressions, say Y_i = m X_i + b + ε_i. It makes no sense to call the ε_i terms "distance".