>I feel the author failed to address the crux of why people argue that "non-statistical" bias is bad – that we should be judged by our actions, and not by factors out of our control such as race, class, the family we were born into, or where we were born.
This is exactly what the author is talking about. You are comparing the predictions against your fantasy of a world where these aspects do not matter because they're not "fair". When they don't match up, you call the predictions biased. But these factors outside our control do matter, accounting for them does not introduce bias, and averting our eyes will not change that fact.
I'm not claiming that we should ignore these aspects and lie to ourselves about reality, in fact I'm acknowledging that these relationships exist. My greater point is that the author is trying to use the meaning of statistical bias to dismiss what journalists/laypersons consider bias without addressing why the latter is concerned with bias in the first place.
My suggestion is that we should be using a better feature set that only looks at aspects that we can reasonably hold an individual responsible for rather than using demographic information which is out of an individual's control. If we have two convicted criminals with similar crimes, behaviors, and histories, but one is white and grew up in a wealthy neighborhood while the other is black and grew up in a poorer town, why should the former be granted a higher probability of parole than the latter? Why should either of them be held responsible for the actions of others? Even if in expectation people from the latter demographic were more likely to reoffend than the former, that is not justice – it undermines liberty.
This is exactly what the author is talking about. You are comparing the predictions against your fantasy of a world where these aspects do not matter because they're not "fair". When they don't match up, you call the predictions biased. But these factors outside our control do matter, accounting for them does not introduce bias, and averting our eyes will not change that fact.