What to Do about Testing: A Response to Audrey Waters

Audrey Waters posted about John Oliver’s takedown of testing and Pearson Ed.  She asks:

How do we seize the opportunity of all this media attention to the problems with standardized testing to do more than talk about testing?  . . . Can we articulate (a better alternative) now so that Pearson and other testing companies don’t replace the old model with simply a re-branded, repackaged one?

Samuel Messick was a Vice President and Distinguished Research Scientist at Educational Testing Services (ETS).  His was an authoritative voice on test validity advocating for restraint in the use of test scores, better and more in-depth interpretations of test score. the collection of multiple sources of information for making important decisions and for consideration of the consequences of test use.  I believe that much of his legacy has been ignored, co-opted, or argued away (even at ETS I suspect).  I’ll speculate on what would he advocate;

  • using more than one or two sources of information when making complex important decisions,
  • understanding the information in the context of a decision and considering the consequences of your testing practices.
  • I also suspect that I could argue with him for the consideration of the validity of testing practices with how it fit within an overall set of district practices.  (i.e. If a student fails, how do you respond?)

Technically Pearson may not be at fault for it is the district use of tests that is most problematic, but Pearson is at least implicit in not providing better guidance and for developing ways for districts  to collect other sources of information.  Eg. The value added model of teacher assessment needs many more sources of information and in fact does not really provide an assessable model of pedagogy, only largely discredited positivist assertions. The first step is to expose those who advocate positivist models of empiricism for which even analytic philosophers would no longer advocate.

Finally it necessary to look at the overall model of education which is still primarily built of a mechanistic metaphor with the student as a vessel to be filled.  The metaphor should be a biological organism adapting in an environment that is primarily social, networked and interactive.  When Pearson speaks of their “potential game-changer: performance tasks”, they are talking in this direction, but their really co-opting performance tasks within the old metaphor.  They have a long way to go.  We should expunge the mechanistic metaphor from educational leadership and assessment models.

The bottom line for Pearson

You may not be technically wrong in your assessments, but when your the brunt of a comedic takedown, you should really look at the consequences of your products use and attempt to deal with it.

Education Needs Clarity

Beginnings: My Graduate Experience  (The 90s and the oughts)

My PhD was not motivated by a career path, but by my love of learning. Temple U’s Associate Professor Helmut Bartel (a proclaimed social constructionist) was an intellectual guide who helped me to recognize the relevance of social theories to my professional experiences; that is, I was by nature a pragmatist. Helmut left Temple before I could develop a dissertation topic and it was fortuitous because I needed to challenge myself to align my thoughts with new mentors. While trying to form a dissertation topic a professor said offhand, “It sounds to me that your talking about validity.” I read Messick’s chapter titled Validity in Linn’s (ed) Handbook of Educational Measurement. The references and the lineage of his ideas were all different, but the conversations where much the same and they centered around a pragmatic approach. The patrons of validity, Messick, Cronback and Meehl, were very clearly analytic in their thinking, but the logic of pragmatism was already deeply embedded in their thought.

Why Philosophy

My studies were in educational psychology, and I do find many discussion in philosophy to be tedious and boring, so why discuss philosophy.  Because, for everything we say, there are many things that are left unsaid and for everything we do, much of the reasoning is left unsaid and unquestioned.  The philosophy I discuss is about shining a light on practices to see what we are taking for granted and to understand what has been left unsaid. What we need is clarity, and that is precisely the purpose of philosophy in its analytic, neoanalytic and pragmatic forms.

Where is Validity in Educational Practice

How do you address validity questions that appear paradigmatically opposed to traditional empirical scientific practice? I begin with an adaptation of a thought who linage I trace Helmut. A successful paradigm change must account for the current paradigm in both its successes and failures in order to forge a true new order. The dominate and implicit practice paradigms today are still mostly based in a dualist objectivist analytic philosophy. Post-modern / post-structural and Marxis based critiques all excel at accounting for the ideological failures of an analytic approach, but not its successes. They fail to point to a way to move practice forward and seem to be losing steam, even as their critiques of analytic approaches remain valid. I think a better way is to consider pragmatism.
Pragmatism and Analytic Philosophy share a commitment to logic and the science method. What Pragmatism brings is a unity of science, practice and ethics (Boncompagni, 2001). Scientific practices are always situated in the midst of ethical horizons best understood as historicized ideological practices. This also matches my earlier experiences where I was working in disability services. The field was moving on from the least restrictive environment to minority rights and people first language. I thoroughly believe in the practicality of science, but science based practices were slow to adapt and often seemed to be standing in the way of ethically empowering practices. Obsessed with an unsustainable conception of objectivity, many scientists could not see how a lack of ethics impoverished science and made it weaker, not stronger.

Pragmatism to the Analytic and Back

I see the history of Pragmatism beginning with Peirce, James, Dewey and Mead, but it became overshadowed by the analytic approaches of European trained academics, especially those associated with the Vienna Circle. As problems were recognized in Analytic Philosophy there began a slow and constant evolution towards pragmatism. In Analytic Philosophy this included people and their ideas such as Quinn, Kuhn, and Wittgenstein. In educational psychology this included Cronbach, Meehl and Messick. This may not be exactly James’ or Dewey’s Pragmatism, but it’s much closer than the direction sought by the Vienna Circle or BF Skinner and I believe that a movement towards pragmatism continues today.

To understand pragmatic social science, let’s begin with Joseph Margolis’ claim: “language and what language uniquely makes possible in the way of the evolving powers of the human mind are emergent, artifactual, hybrid precipitates of the joint processes of biological and cultural evolution;” I see this as something like taking up the naturalism and social behaviorism of Dewey and Mead.  This approach may no longer provide  a foundation for infallible truths, but there is still room for an ethical, objective and empirically warranted practice. This social behavioral and empirical science should be distinguished from Skinner’s radical behaviorism in the same way logical positivism is distinguished from current analytic / pragmatic  approaches.   The knowledge radical behaviorism engenders, fails to adequately recognize the full nature of language and the social world it makes possible.  As a result radical behaviorism leaves knowledge as flat and shallow and more often results in situations (as Wittgenstein noted) where the educational problem and the method pass one another by without interacting. To be valid, empirical methods must reflect the contextualized, artifactual and ethical demands of the problems within a philosophically Darwinian framework of an organism’s adaptation to the social and physical environment. Adaptation is very personal and includes concepts like social poetics.  That is, I accept analytic tools and methods, but recognize them only within social ethical fields that are interpretive as above.  Just as analytic philosophy has moved back toward Pierce, James, Dewey and Mead, radical behaviorism can only be relevant by moving toward Vygotsky, Dewey, Wittgenstein and social poetics.


Margolis, Joseph (2012-10-17). Pragmatism Ascendent: A Yard of Narrative, a Touch of Prophecy (p. 133). Stanford University Press. Kindle Edition.

Boncompagni, A (2011). Book Review on New Perspectives on Pragmatism and Analytic Philosophy, EUROPEAN JOURNAL OF PRAGMATISM AND AMERICAN PHILOSOPHY, III, 2, 290-299. http://lnx.journalofpragmatism.eu/wp-content/uploads/2012/01/calcaterra-new-perspective.pdf

Garrison, J (1995). Deweyan Pragmatism and the Epistemology of Contemporary Social Constructivism, American Educational Research Journal, 32, 716-740.