Landing : Athabascau University

Beyond learning outcomes

What we teach, what a student learns, what we assess This is a slide deck for a talk I'm giving today, at a faculty workshop, on the subject of learning outcomes.

I think that well-considered learning outcomes can be really helpful when planning and designing learning activities, especially where there is a need to assess learning. They can help keep a learning designer focused, and to remember to ensure that assessment activities actually make a positive contribution to learning. They can also be helpful to teachers while teaching, as a framework to keep them on track (if they wish to remain on track).  However, that's about it. Learning outcomes are not useful when applied to bureaucratic ends, they are very poor descriptors of what learning actually happens, as a rule, and they are of very little (if any) use to students under most circumstances (there are exceptions - it's a design issue, not a logical flaw).

The big point of my talk, though, is that we should be measuring what students have actually learned, not whether they have learned what we think we have taught, and that the purpose of everything we do should be to support learning, not to support bureaucracy.

I frame this in terms of the relationships between:

  • what we teach (what we actually teach, not just what we think we are teaching, including stuff like attitudes, beliefs, methods of teaching, etc),
  • what a student learns in the process (an individual student, not students as a whole), and
  • what we assess (formally and summatively, not necessarily as part of the learning process).

There are many things that we teach that any given student will not learn, albeit that (arguably) we wouldn't be teaching at all if learning were not happening for someone. Most students get a small subset of that. There are also many things that we teach without intentionally teaching, not all of them good or useful.

There are also very many things that students learn that we do not teach, intentionally or otherwise. In fact, it is normal for us to mandate this as part of a learning design: any mildly creative or problem-solving/inquiry-oriented activity will lead to different learning outcomes for every learner. Even in the most horribly regimented teaching contexts, students are the ones that connect everything together, and that's always going to include a lot more than what their teachers teach.

Similarly, there are lots of things that we assess that we do not teach, even with great constructive alignment. For example, the students' ability to string a sentence together tends to be not just a prerequisite but something that is actively graded in typical assessments.

My main points are that, though it is good to have a teaching plan (albeit that it should be flexible,  reponsive to student needs, and should accommodate serendipity)learning :

  • students should be participants in planning outcomes and
  • we should assess what students actually learn, not what we think we are teaching.

From a learning perspective, there's less than no point in summatively judging what learners have not learned. However, that's exactly what most institutions actually do. Assessment should be about how learners have positively changed, not whether they have met our demands.

This also implies that students should be participants in the planning and use of learning outcomes: they should be able to personalize their learning, and we should recognize their needs and interests. I use andragogy to frame this, because it is relatively uncontroversial, is easily understood, and doesn't require people to change everything in their world view to become better teachers, but I could have equally used quite a large number of other models. Connectivism, Communities of Practice, and most constructivist theories, for instance, force us to similar conclusions.

I suggest that appreciative inquiry may be useful as an approach to assessment, inasmuch as the research methodology is purpose-built to bring about positive change, and its focus on success rather than failure makes sense in a learning context.

I also suggest the use of outcome mapping (and its close cousin, outcome harvesting) as a means of capturing unplanned as well as planned outcomes. I like these methods because they only look at changes, and then try to find out what led to those changes. Again, it's about evaluation rather than judgment.

Comments

  • Gerald Ardito November 2, 2018 - 11:59am

    Jon,

    I really enjoyed reviewing the slide deck. If you have any audio, it would be great to be able to hear the presentation itself.

    I was unaware of Outcome Mapping or Outcome Harvesting, so I look forward to learning more about them.

    Particularly intriguing to me is how much tunnel vision gets added by not looking at all learning outcomes. So, thanks for that.

  • Jon Dron November 2, 2018 - 2:32pm

    Thanks Gerald!

    Yes, I had an 'aha' moment when I drew the Venn diagram and mapped it to intended outcomes. I've said as much in so many words many times before but, seeing it laid out in front of me, it just seemed so obvious that the only thing we should ever assess is (in positive terms) what a student has learned. It's good news if that happens to overlap with what we are trying to teach and what we are trying to assess, but that's not the main point. Another nice thing about thinking this way is that, if a student achieves outcomes that we did not intend, but that happen to overlap with something else we assess (e.g. a different course) it can be used as evidence towards that, too. This does mean that we need to have a pretty fair idea of our intended outcomes across the whole institution (or at least a whole program), and thus will run into the many problems of misusing learning outcomes as currency/bureaucratic measurement tools, but it might be a way to sell the idea to those in charge.

    Alas, no audio or recording of any kind.

    I came across the concept of outcome mapping/harvesting at the International Federation of National Teaching Fellows conference earlier this year and it resonated with many of the things I have been grappling with over recent years. The details are fairly mundane and obvious - it's a project management tool for dealing with complex projects and emergent/unanticipated/fuzzy outcomes, but the general principle of identifying change (any change) as an outcome, and working back from that to discover what led to it makes an awful lot of sense to me in a learning context. The act of mapping itself is a highly reflective and potentially very effective pedagogical process in its own right, so it's a good idea even if bureaucracy prevents you from actually using those outcomes in summative assessment. If the powers that be prevent you from adding new outcomes (or from removing those that are pre-ordained) you can always add an outcome along the lines of 'be a reflective practitioner' or 'demonstrate the ability to be a lifelong learner' or 'critically evaluate their learning in the field' to the specified outcomes, but I think it is way more useful if we are allowed to be flexible in specifying criteria for success on a per-student basis.  

    Jon

  • Gerald Ardito November 7, 2018 - 7:10am

    Jon,

    In a chunk of time full of synchronicity, I have been wondering about the same things. 

    I have been working with my teacher candidates in designing curriculum, specifically units of study and lesson plans for elementary school students. They are good students and very committed to being good teachers, but all of there work amounted to an addition problem = fact 1 + fact 2 + fact 3 = learning. You can imagine that the assessments they designed were very traditional and linear as well.

    I have been working with them pretty intensely around seeing that their job is to create learners and people in love iwth learning, not pushing content. I want to investigate outcome mapping and outcome harvesting further as tools that they could be possibly using to change their thinking and practice.