There have been (at least) tens of thousands of comparative studies on the effects of 'technology' on learning performed over the past hundred years or so. Though some have been slightly more specific (the effects of computers, online learning, whiteboards, eportfolios, etc) and some more sensible authors use the term 'tech' to distinguish things with flashing lights from technologies in general, nowadays it is pretty common to just use the term 'technology' as though we all know what the authors mean. We don't. And neither do they.

It makes no more sense to ask whether (say) computers have a positive or negative effect on learning than to ask whether (say) pedagogies have a positive or negative effect on learning. Pedagogies (methods and principles of learning and teaching) are at least as much technologies as computers and their uses and forms are similarly diverse. Some work better than others, sometimes, in some contexts, for some people. All are soft technologies that demand we act as coparticipants in their orchestration, not just users of them. This means that we have to add stuff to them in order that they work. None do anything of interest by themselves - they must be orchestrated with (usually many) other tools, methods, structures, and so on in order to do anything at all. All can be orchestrated well (assuming we know what 'well' really means, and we seldom really do) or badly.

It is instructive to wonder why it is that, as far as I know, no one has yet tried to investigate the effects of transistors, or screws, or words, or cables on learning, even though they are an essential part of most technologies that we do see fit to research and are certainly prerequisite parts of many educational interventions. The answer is, I hope, obvious: we would be looking at the wrong level of detail. We would be examining a part of the assembly that is probably not materially significant to learning success, albeit that, without them, we would not have other technologies that interest us more. Transistors enable computers, but they do not entail them.

Likewise computers and pedagogies enable learning, but do not entail it (for more on enablement vs entailment, see Longo et al, 2012 or, for a fuller treatment, Kauffman, 2019). True, pedagogies and computers may orchestrate many more phenomena for us, and some of those orchestrations may have more consistent and partly causal effects on whether an intervention works than screws and cables but, without considering the entire specific assembly of which they are a part, those effects are no more generalizably relevant to whether learning is effective or not than the effects of words or transistors.

Technologies enable (or sometimes disable) a range of phenomena, but only rarely do they generalizably entail a fixed set of outcomes and, if they do, there are almost always ways that we can assemble them with other technologies that alter those outcomes. In the case of something as complex as education, which always involves thousands and usually millions of technological components assembled with one another by a vast number of people, not just the teacher, every part affects every other. It is irreducibly complex, not just complicated. There are butterfly's wing effects to consider - a single injudicious expletive, say, or a even a smile can transform the effectiveness or otherwise of teaching. There's emergence, too. A story is not just a collection of words, a lesson is not just a bunch of pedagogical methods, a learning community is not just a collection of people. And all of these things - parts and emergent or designed combinations of parts - interact with one another to lead to deterministic but unprestatable consequences (Kauffman, 2019).

Of course, any specific technology applied in a specific context can and will entail specific and (if hard enough) potentially repeatable outcomes. Hard technologies will do the same thing every time, as long as they work. I press the switch, the light comes on. But even for such a simple, hard technology, you cannot from that generalize that every time any switch is pressed a light will come on, even if you, without warrant, assume that the technology works as intended, because it can always be assembled with other phenomena, including those provided by other technologies, that alter its effects. I press many switches every day that do not turn on lights and, sometimes, even when I press a light switch the light does not come on (those that are assembled with smart switches, for instance). Soft technologies like computers, pedagogies, words, cables, and transistors are always assembled with other phenomena. They are incomplete, and do not do anything of interest at all without an indefinitely large number of things and processes that we add to them, or to which we add them, each subtly or less subtly different from the rest. Here's an example using the soft technology of language:

• There are countless ways I could say this.
• There are infinitely many ways to make this point.
• Wow, what a lot of ways to say the same thing!
• I could say this in a vast number of ways.
• There are indefinitely many ways to communicate the meaning of what I wish to express.
• I could state this in a shitload of ways.
• And so on, ad infinitum.

This is one tiny part of one tiny technology (this post). Imagine this variability multiplied by the very many people, tools, methods, techniques, content, and structures that go into even a typical lesson, let alone a course. And that is disregarding the countless other factors and technologies that affect learning, from institutional regulations to interesting news stories or conversations on a bus.

Reductive scientific methods like randomized controlled tests and null hypothesis significance testing can tell us things that might be useful to us as designers and enactors of teaching. We can, say, find out some fairly consistent things about how people learn (as natural phenomena), and we can find out useful things about how well different specific parts compare with one another in a particular kind of assembly when they are supposed to do the same job (nails vs screws, for instance). But these are just phenomena that we can use as part of an assembly, not prescriptions for successful learning. The question of whether any given type of technology affects learning is meaningless. Of course it does, in the specific, because we are using it to help enable learning. But it only does so in an orchestrated assembly with countless others, and that orchestration is and must always be substantially different from any other. So, please, let's all stop pretending that educational technologies (including pedagogical methods) can be researched in the same reductive ways as natural phenomena, as generalizable laws of entailment. They cannot.

References

Arthur, W. B. (2009). The Nature of Technology: what it is and how it evolves (Kindle ed.). New York, USA: Free Press. (Arthur's definition of technology as the orchestration of phenomena for some purpose, and his insights into how technologies evolve through assembly, underpins the above)

Kauffman, S. A. (2019). A World Beyond Physics: The Emergence and Evolution of Life. Oxford University Press.

Longo, G., Montévil, M., & Kauffman, S. (2012). No entailing laws, but enablement in the evolution of the biosphere. Proceedings from 14th annual conference companion on Genetic and evolutionary computation, Philadelphia, Pennsylvania, USA. Full text available at https://dl.acm.org/doi/pdf/10.1145/2330784.2330946