I attended a recent nyc edtech meet up at Knewton in NYC. While looking at their promotional materials on their platform it occurred to me that this system has a stronger basis in validity over high stakes standardized testing (HSST). I know it’s a (big) data driven approach, likely similar to what I was familiar with at Sylvain, except that digitalization allows you to address many more dimensions in the data, to cross-reference different domain skills and to better represent intellectual development over time. This post is about the validity of big data adaptive learning systems as compared to high stakes standardized testing (HSST).
- The easiest distinction to be made is to contrast the “snapshot in time” nature of HSST and the developmental histories of adaptive learning. Development is the way students and teachers understand school-based learning especially when it’s not linear, but proceeds in fits and starts. Neither does a snapshot relate to the purposes of assessment. In adaptive learning error is not judgement, but an excuse for more learning.
- This point may seem esoteric but I think important. HSST must represent an ambitious construct interpretation, that is, a single HSST question must represent the same learning that is represented in hundreds if not thousands of questions in an adaptive learning system. And while the assessments in the adaptive system are part of the learning process, HSST constructs often stand outside of any pedagogical process. (See #1 below)
- There are negative consequences associated with HSST. Because of the lag time between testing and reporting, there is less instructional relevance to HSSTs. Assessments in adaptive learning provide immediate feedback and are instrumental to the learning process. There are also many unintended consequences, like instructional time that is wasted on test prep or the disassociation of error from an opportunity to learning.
- Assessments are consequential for students. In adaptive learning assessments determines the instructional pathway the student will pursue. If done well, the student will perceive this assessment to have been appropriate and helpful. In many HSST (e.g. the SAT) assessments may be perceived as a threat and associated with a lack of opportunity. See #2 below
It seems to me that as Adaptive Learning becomes more common and its validity become recognized, HSST will no longer be needed.
#1. “If the IUA does not claim much (e.g., that students with high scores on the test can generally perform the kinds of tasks included in the test), it does not require much empirical support beyond data supporting the generalizability of the scores. A more-ambitious interpretation (e.g., one involving inferences about some theoretical construct) would require more evidence (e.g., evidence evaluating the theory and the consistency of the test scores with the theory) to support the additional claims being made”. Kane (2013) p.3
#2. “The SAT is a mind-numbing, stress-inducing ritual of torture. The College Board can change the test all it likes, but no single exam, given on a single day, should determine anyone’s fate. The fact that we have been using this test to perform exactly this function for generations now is a national scandal”. NYTimes