Very interesting work, and well-explained in the post.
Like many others here, I suppose that in it's basic form this would mostly be used for cheating on homework; although it would certainly be useful for those (few?) students who are truly motivated to self-learn the material, rather than just pass the tests.
One thing which springs to mind is "Benny's Conception of Rules and Answers in IPI Mathematics" ( https://msu.edu/course/cep/953/readings/erlwanger.pdf ), which shows the problem of only focusing on answers, and on "general purpose" problem sets. Namely that incorrect rules or concepts might be learned, if they're reenforced by occasionally giving the right answer.
I think it would be interesting to have a system capable of some back-and-forth interactivity: the default mode would be the usual, going through some examples, have the student attempt some simple problems, then trickier ones, and so on.
At the same time, the system would be trying to guess what rules/strategies the student is following: looking for patterns, e.g. via something like inductive logic programming. We would treat the student as a "black box", which we can learn about by posing carefully crafted questions.
Each question can be treated as an experiment, where we want to learn the most information about the student's thinking: if strategies A and B could both lead to the answers given by the student, we construct a question which leads to different answers depending on whether A or B were used to solve it; that gives us information about which strategy is more likely to be used by the student, or maybe the answer we get is poorly explained by A and B, and we have to guess some other strategies they might be using.
Rather than viewing marking as a comparison between answer and a key, we can instead infer a model of the domain from those answers and compare that to an accurate model of the domain.
We can also use this approach the other way around, treating the domain as a black box (which it is, from the student's perspective) and choosing examples which give the student most information about it.
Like many others here, I suppose that in it's basic form this would mostly be used for cheating on homework; although it would certainly be useful for those (few?) students who are truly motivated to self-learn the material, rather than just pass the tests.
One thing which springs to mind is "Benny's Conception of Rules and Answers in IPI Mathematics" ( https://msu.edu/course/cep/953/readings/erlwanger.pdf ), which shows the problem of only focusing on answers, and on "general purpose" problem sets. Namely that incorrect rules or concepts might be learned, if they're reenforced by occasionally giving the right answer.
I think it would be interesting to have a system capable of some back-and-forth interactivity: the default mode would be the usual, going through some examples, have the student attempt some simple problems, then trickier ones, and so on.
At the same time, the system would be trying to guess what rules/strategies the student is following: looking for patterns, e.g. via something like inductive logic programming. We would treat the student as a "black box", which we can learn about by posing carefully crafted questions.
Each question can be treated as an experiment, where we want to learn the most information about the student's thinking: if strategies A and B could both lead to the answers given by the student, we construct a question which leads to different answers depending on whether A or B were used to solve it; that gives us information about which strategy is more likely to be used by the student, or maybe the answer we get is poorly explained by A and B, and we have to guess some other strategies they might be using.
Rather than viewing marking as a comparison between answer and a key, we can instead infer a model of the domain from those answers and compare that to an accurate model of the domain.
We can also use this approach the other way around, treating the domain as a black box (which it is, from the student's perspective) and choosing examples which give the student most information about it.