Athabasca University | AU Student/Staff Login | Invited Guest Login
The Limitation of Portolio
Using portforlio as a tool of authentic assessment for student performance in higher education has been adopted by many institutions lately. Richard J. Shavelson, a professor of education at Stanford University, and his colleaque, Stephen Klein and Roger Benjamin, argued that portfolios do not and cannot produce the trustworthy information that is needed for making a sound assessment of student learning. The inherent limitations of portfolio assessment include lack of standardization, not feasible for large scale learning assessment, and bias. In their conclusion, the authors stated: "Gathering valid data about student performance levels and performance improvement requires making comparisons relative to fixed benchmarks and that can only be done when the assessments are standardized. Consequently, we urge the higher education community to embrace authentic, standardized performance-assessment approaches so as to gather valid data that can be used to improve teaching and learning as well as meet its obligations to external audiences to account for its actions and outcomes regarding student learning."
- Portfolio Final Submission - COMP266
May 29, 2023 - 6:46pm
Alexander Gorgi - At the end of this post a successful reader will be able to make better use of learning outcomes
May 9, 2024 - 1:08pm
Jon Dron
SuTuan Lulee
Comments
I wonder if their PhDs were assessed using standardized testing?
Standardized testing proves that you know how to pass the standardized test under conditions that resemble almost nothing in the real world where the knowledge being assessed will be applied. Authenticity? Not on this planet. Their own CLA assessment is designed to show analytic and problem-solving skills, which is commendable, but the criteria they apply in those assessments, if they really do show what they set out to show, would work better in a portfolio, without the inauthentic, artificial, unnaturally stressful and meaning-free context of an exam setting. The majority of standardized tests to fixed benchmarks (including their own work) give little opportunity by design to show creativity, ability to work with others, ability to synthesise and connect ideas, to analyse in the light of complex interconnected knowledge.Their own criteria would make that unfair. Yet these are arguably the most important competencies needed for almost anything apart from factory work.
I wonder how they feel about projects? Presumably these are tarred with the same brush.
It is simply wrong to suggest that portfolio assessment criteria must lack rigour. I have sat on countless exam boards as an external assessor and with external assessors for programs I have led or been involved with, using portfolios, projects, open essays, research papers, vive voces, constrained exam questions as well as almost every form of standardized test in use today. When criteria for competence are well-specified the agreement between independent examiners of portfolios is at least as high as between those marking standardized tests and, for that matter, any other form of assessment. It does raise the important issue that, for fairness, consistency and rigour there does have to be quality control as well as quality assurance at all stages from design of criteria to evaluation of their effectiveness and back again, but that is true across the board and is a matter of management process, not of the format of the assessment.
Because they have no connection with learning, standardized tests are always more expensive than e-portfolios. A portfolio is both a process and a product so has innate value as a learning experience and contributes greatly to the learning process. Time not spent on a separate testing process is time gained for actually doing something useful like learning something, say. Standardized tests are always separate from the learning experience so they always cost more as the are always required in addition to the learning interventions themselves.
Portfolios encourage intrinsic motivation by enabling control, competence and (for most e-portfolio methods at least) connection. A few people who have not read any research on the subject since 1973 and who think behaviourism is the last word in pedagogic theory believe standardized tests also motivate learners. This is wrong. Research-based evidence shows unequivocally that standardized tests are extremely poor motivators for learning and almost always have a demotivating effect. This is not only expensive and wasteful. The least of the consequences is poor, shallow, strategic and inefficient learning driven by extrinsic motivation at the expense of intrinsic motivation - that, charitably, is a waste of time and money. At worst it causes drop-outs or, worst of all, causes an aversion to subjects covered and even to formal education itself. That is too high a cost to bear. If they are only interested in measuring marking load rather than the cost-effectiveness of the whole learning transaction then it is putting the cart before the horse and then, to extend the metaphor slightly, wasting time only looking at the horse's dung.
If we assume that the authors are products of such a system themselves it begins to make sense that they wrote this. An inability to understand make meaningful connections between evidence and theory is a symptom of the effects of this weak and short-sighted approach to education.
Jon
SuTuan thanks for posting this provocative paper. It pushes my buttons as well! I agree with Jon's post and would like to add that it is exactly this type of thinking (which exists within school administrations and boards) that constrains educational growth. Educational systems have to answer to tax-payers and assessment is the traditional criteria used. We have to learn how to assess differently, yet make the results palatable to the 'powers to be'. Standardized testing is the easy way out.
Absolutely! And indeed, thanks SuTuan for highlighting this one.
Standardized testing is the lazy way out but, like most lazy solutions, is much more effort in the end.
We do need to get away from being simultaneously producers and arbiters of education though, and one good thing to be said about such otherwise harmful practices is that they can make this easier to accomplish. Education is one of those rare trades where, when *we* do our job badly we get to tell the person who is paying us that it is their fault and, astonishingly, they tend to accept this on the whole. To add injury to insult, while they may often be the ones paying for it, they are not even really the customer (the primary customer is society as a whole). It is, however, still far better to use portfolios to achieve this separation, as long as there are good and accountable criteria for measuring competence, such as may be found in Athabasca University's PLAR process.
Thanks for the feedback, Dr. Dron and Susan!
Shavelson is one of the key scholars whose publications have been traced by me in order to pace up with current studies on formative assessment. The immediate response when I first read his comments on portfolio assessment was: “This is interesting!” I don’t often hear voices that do not support “portfolio as a tool for student assessment”. Thus I bookmarked here at Landing for further reference and to share with Landing-ers.
I have developed more than one e-portfolios for different courses and programs I took from academic institutions so that my instructors could assess my learning results. I learned a lot from developing e-portfolios. I agree with Dr. Dron that portfolio creation “has innate value as a learning experience and contributes greatly to the learning process” and restricted causal perspectives might be too narrow to be used for complex-learning such as PHD research. However, I also recognize the value of empirical evidences produced by accuracy-based approaches to assessment. Because, educational assessment wants to draw inferences about what students know, can do, or have accomplished more broadly basing on what they know and do observed (Mislevy & Riconscente, 2005). Inferences are hypotheses, and the validation of inferences is hypothesis testing that embraces all of the experimental, statistical, and philosophical means (Messick, 1989).
Over and above, I consider the validity of educational assessment a continuum with portfolio approach on the one end and criterion-based approach on the other. It’s a degree rather than an absolute value. Assessment is an integrated evaluative judgment. It refers to the degree to which empirical evidence and theoretical rationale support the adequacy and appropriateness of interpretations (Messick, 1989). Students performances need to be assessed via multiple ways (informal, formal) by multiple participants (instructor, peer and self) (Perkins, 1994). Therefore using any single way of assessment to build a claim about student performances could be insufficient and is an easy way out.
Assessment has being “a frustrating professional problem for the people involved.” (Patton, 1986, p. 11). I think none of portfolio or criterion-based standard assessment is an easy job. Differences in the teacher psychological belief (trait, behavioral, information processing, sociocultural …) and the purpose of assessment (to publish in league tables, to provide certificates, to support learning …) will lead to different practices of assessment (Black, Harrison, Lee, Marshall, & Wiliam, 2003).
Thanks for sharing your time and attention with me! Does my lengthy post make any sense to you?
Su-Tuan Lulee
References
Black, P., Harrison, C., Lee, C., Marshall, B., Wiliam, D., & Press, O. U. (2003). Assessment for Learning: Putting it into Practice (1st ed.). New York: Open University Press.
Messick, S. (1989). Validity. Educational measurement. In R. L. Linn (Ed.), Educational Measurement, The American Council on Education, Macmilan series on higher education (3rd ed., pp. 13-103). New York, NY: Macmilan Publishing Co., Inc.
Mislevy, R. J., & Richardson, M. M. (2005). Evidence-Centered Assessment Design: Layers, Structures, and Terminology ( No. 9). PADI Technical Report (p. 46). CA: SRI International. Retrieved from http://padi.sri.com/downloads/TR9_ECD.pdf
Patton, M. Q. (1986). Utilization-Focused Evaluation (2nd ed.). Sage Publications, Inc.
Perkins, D., & Blythe, T. (1994). Putting understanding up front. (cover story). Educational Leadership, 51(5), 4. doi:Article