This, along with several other "meta" objections, is a significant portion of the discussion in the paper.
They basically say two things. First, although the measurement is repeatable at face value, there are several factors that make it less impressive than assumed, and the model performs fairly poorly compared to likely prospective lawyers. Second, there is a number of reasons why the percentile on the test doesn't measure lawyering skills.
One of the other interesting points they bring up is that there is no incentive for humans to seek scores much above passing on the test, because your career outlook doesn't depend on it in any way. This is different from many other placement exams.
They basically say two things. First, although the measurement is repeatable at face value, there are several factors that make it less impressive than assumed, and the model performs fairly poorly compared to likely prospective lawyers. Second, there is a number of reasons why the percentile on the test doesn't measure lawyering skills.
One of the other interesting points they bring up is that there is no incentive for humans to seek scores much above passing on the test, because your career outlook doesn't depend on it in any way. This is different from many other placement exams.