An interesting proposal from Horn & Fisher that fills in one of the most gaping holes in conventional quantitative research in education (specifically randomized controlled trials but also less rigorous efforts like A/B testing etc) by explicitly looking at the differences in those that do not fit in the average curve - the ones that do not benefit, or that benefit to an unusual degree, the outliers. As the authors say:

"... the ability to predict what works, for which students, in what circumstances, will be crucial for building effective, personalized-learning environments. The current education research paradigm, however, stops short of offering this predictive power and gets stuck measuring average student and sub-group outcomes and drawing conclusions based on correlations, with little insight into the discrete, particular contexts and causal factors that yield student success or failure. Those observations that do move toward a causal understanding often stop short of helping understand why a given intervention or methodology works in certain circumstances, but not in others."

I have mixed feelings about this. Yes, this process of iterative refinement is a much better idea than simply looking at improvements in averages (with no clear causal links) and they are entirely right to critique those that use such methods but:

a) I don't think it will ever succeed in the way it hopes, because every context is significantly different and this is a complex design problem, where even miniscule differences can have huge effects. Learning never repeats twice. Though much improved on what it replaces, it is still trying to make sense through tools of reductive materialism whereas what we are dealing with, and what the authors' critique implies, is a different kind of problem. Seeking this kind of answer is like seeking the formula for painting a masterpiece. It's only ever partially (at best) about methodologies and techniques, and it is always possible to invent new ones that change everything.

b) It relies on the assumption that we know exactly what we are looking for: that what we seek to measure is the thing that matters. It might be exactly what is needed for personalized education (where you find better ways to make students behave the way you want them to behave) but exactly the opposite for personal education (where every case is different, where education is seen as changing the whole person in unfathomably rich and complex ways).

That said, I welcome any attempts to stop the absurdity of trying to intervene in ways that benefit the (virtually non-existent) average student and that instead attempt to focus on each student. This is a step in the right direction.