I don't really care about being "fair" to the applicant, I care about getting the most competent doctor I can. Being a doctor is position of grave responsibility, it shouldn't be a reward that we give to people who deserve it because they tried really hard and had a tough life.
Those kids who had a tough life might be super smart and fully capable of higher scores but didn't achieve them because they didn't have the time to study or were exhausted from just trying to survive. But once in the school setting and then in medical practice can thrive.
It's like trying to find the fastest sprinter using a race but some of kids are wearing heavy backpacks and some aren't. How do you find the actual fastest sprinters?