A Framework for Action with Reflection: Measurement with Validity

My recent reading reinforce a need for validation thinking.

First, David Jones’ blog in a recent post considers a quote about the problems stemming from theory without action (idealism) and action without “philosophical reflection” (mindlessness).

Second, a reassertion of a limited (but still robust) neo-positivism by Philip Tetlock in Expert Political Judgment.

My Response: Let’s begin by refine Jones quote to read as follows:

Theory without the measurement of empirical correlates to justify action will lead to actions that are based only on biased judgment; and action without broad reflection (even if that action is supported by logical empirical correlates) just as surely leads to action that is based on unrecognized biased judgments.

This is also the basic argument that stands between logical positivism and radical social constructivism (at least in its straw men forms).  I believe that it argues for a dialectic type of response that is `implied by Jones quote.  One of the problems in the positivism / constructivism argument is that both sides spin complex arguments that fail at parsimony, that is, they become needlessly complex in the attempt to justify their radical stances.  This is where validity thinking can serve as Occam’s Razor.

First, consider a unified idea of construct validity: measurements (at least in the real world almost always) measure constructs, not real objects.  (eg. Even if we measure real objects [like pencils] we must define pencils [as not pens or not markers] in a way that indicates that what is indeed being measured is a construct.  This is not quite idealism.  Although it is possible to distinguish between constructs (like IQ) and real objects (like pencils); we cannot operationalize the measurement of real objects without referring to a construct in some way.  So, operationalism, the logical positivist’s banning of constructs by fiat, will not hold.)

Next: The purpose of measurement is to overcome the biases of Jones’ idealism.  We don’t normally fall into idealism because we rely on scientific methodology (based in measurement) to counter bias.  But, we can’t totally escape idealism with measurement because constructs provide a place for ideas and idea bias to creep back in.  Hence the dialectic, we measure with reflection.

In Messick’s validation framework reflection looks like this.

  1. Content validity: does the way we are measuring make sense both logically and through the experience of ourselves and others.
  2. Substantive: is there a theory (empirically supported) that gives meaning to the measures taken.
  3. Structural: do the measures faithfully reproduce the tasks or processes theorized to exist in the contexts or natural settings to which you want to extrapolate.
  4. Generalizability:  Is there evidence that what you are measuring applies to other places and other times.  (Even if you’re not measuring to generalize your findings, evidence against generalizability should cause you to reflect on why the place and time matter)
  5. External: This is mostly (convergent or discriminate) criterion evidence.
  6. Consequential: Show me evidence that measuring is helping. (eg. Are “No Child Left Behind” measures helping students become better prepared for life after school; or are they at least helping to further the right wing agenda to undermine the power of the NEA.  (Opps. . . sorry . . . cynicism slipping in there)

Therefore, a framework for action with reflection is measurement with validity.