Validated Methodological Operationism: Improve Analytics by Validating Your Operations

Many measured processes can be improved by validating your process operations.  This is true whether your are talking about business, experimental, or educational processes.

A New View Of Operationism

Interesting read on operationism by Uljana Feest – (2005)  Operationalism in Psychology: What the Debate is About, What the Debate Should Be About [Journal of the History of Behavioral Sciences, 41(2) 131-149].

The basic gist: Psychologist’s historical use of operationalism was methodological rather the positivist (even though they may have referenced positivism for philosophical cover).  So criticizing operationism using positivist arguments is somewhat misguided, but operations can be criticized through validation arguments.

What does Feest mean by a methodological reading of operationism?

. . . I mean that psychologists did not intend to say, generally, what constitutes the meaning of a scientific term.  . . . in offering operational definitions, scientists were partially and temporarily specifying their usage of certain concepts by saying what kind of empirical indicators they took to be indicative of the referents of the concepts (p. 133).

She concludes by saying:

. . . the debate should then be about what are adequate concepts and how (not whether) to operationalize them, and how (not whether) to validate them (p.146).

So any debate about operationism is really about constructs and their validation.  Within this framework, I will list 4 specific types of operationism.

Positivists, Empirist Operationism

This idea can be represented by Percy Bridgman’s original conception of operationsim

in general, we mean by a concept nothing more than a set of operations; the concept is synomonous with the corresponding set of operations (Bridgeman, P.[1927]. The logic of Modern Physics, Macmillan:NY. p.5).

The biggest problem with this approach is that any set of operations can never be said to exhaust the entirety of meaning in any construct, a position that is also supported by cognitive psychology’s understanding of cognitive processes in the meaning and use of concepts (Andersen, H., Barker, P & Chen, X. (2006). The Cognitive Structure of Scientific Revolutions, Cambridge University Press).

Methodological Operationism

The idea that operations are the empirical indicators of the construct (Feest).

Naive Pragmatic Operationism

Regardless with how you conceive of a construct, within any measured process, no matter if that process is an experimental, business or any other process that is controlled by measures, those measurement operations are methodologically defining that construct in the function of that process.  If you throw any measure in place without determining how and why you are using that measure, you are operating in the same fashion as any operationists in the positivist empiricist mode and you are subject to the same kinds of problems.  Garbage in = garbage out; this is the real potential problem with this approach.  There are many business process that do not meet their expectations and those problems can be traced back to poor quality measurements whose construct are not appropriately operationalized.

Validated Methodological Operationism

This represents measured processes whose operations are clear and whose quality and validity has been adequately evaluated.

Note

Feest references the gap between qualitative and quantitative research as being about operationism.  I believe this is incorrect.  Operationism is about construct validity (unified theory).  Criticism of qualitative research is usually about research validity (a different validity) and the value of different research purposes.

What Can You Do with Validity?

A Follow-up on my last post.

Are your measures valid across a range of concerns?  Improving validity will lead to improved actions, better frameworks for acting and ultimately improved performance. In example:

The turn of the century saw an increase in the expectations tied to measurement through such phenomena as “No Child Left Behind” or SAT test prep classes.  This has begun to change as colleges put less emphasis on SAT scores and I believe we’ll soon see similar changes in high stakes graduation tests.  Two observations:

  • While high expectations pose difficult challenges for assessment, most of the problems that resulted in less use of assessment are in the expectations placed in specific tests not in the capabilities of assessment in general.  It’s a hermeneutic problem.  The meaning of test scores was much narrower than were the expectations for assessment; a mismatch between the meaning that was required and the meaning that the test could supply.    From a narrow psychometric perspective involving external validity, these test were valid, but from other perspective (structural or consequential validity – see the previous post) they are found wanting.
  • People will still act and those actions will still require assessment and those assessments will still be made.  They will just be more casual, less observable, and even less valid that those made by high stakes tests.

Most actionable situations require a range of assessments that are valid across a range of validity concepts.  Just because some are less empirical or more qualitative does not mean they should not be considered in an appropriate mix.


Why Be Concerned with Validity: My Personal Experience

My PhD was in education psychology and most of my classes occurred in the mid to late 90s.  The paradigm wars were winding down, but there was still a noticeable split between hermeneutic social constructionists* and the psychometrician.  My nature is to want to synthesize, often leading one to walk in two worlds.  Too what would I be drawn; a hermeneutic account of psychometrics of course.

I was investigating dissertation topics around disability.  The split here was conveyed as between old psychometric ways of conceiving of disability and new socially constructed accounts.  An advisor made a casual comment that my concerns seemed to be about validity and it seemed insightful.  Yes!  The problem was that existing measures were validated by psychometric models that did not account for the hermeneutics of identity construction or for the consequences of resulting identities.

I started my investigation by reading Samuel Messick’s chapter on Validity in Educational Measurement (3rd ed.).  What I read was Messick’s attempt to address hermeneutic aspects of measurement from a psychometric perspective.  What was important in measuring, is the meaning you derive from the data and the associated implication for action.  First, there are only 2 ways to think about invalidity:

  • Construct Under-representation; The construct you are interested in is larger than what your assessment is able to measure.
  • Construct Irrelevance; you are measuring things that are irrelevant to the information you need to take action and lead to either false positives or false negatives.

Messick would later write about six categories of validity concerns.  I take these categories to be a framework for how to think about or find meaning in measurement.  They are 6 different way of looking for under-representation or irrelevance:

  1. Content – Is there evidence that the scope of the content appropriate and representative of the construct.
  2. Substantive – Is there a theory for the processes and tasks being performed and is there empirical support for the theory.
  3. Structural – Is there evidence that the assessment faithfully reproduces the tasks or processes in contexts or in the natural settings to which you are trying to extrapolate.
  4. Generalization – Has the assessment been shown to apply to many different groups, contexts and over time.  While this may not reduce validity in specific situations, it would indicate to look much closer at the situation your in.
  5. External – convergent or divergent criterion evidence.
  6. Consequential – Is there evidence that your actions are improved by the assessment and that it is fair and free of bias.

* Note – I have no interest in most philosophical discussions of the beliefs of social constructionist or realists.  For me, SC is mostly about the ways that things and people are thoroughly effected and affected by the pervasiveness of language and its accompanying hermeneutics.  Not only is there no denial of reality, the current trend is to highlight the embodied nature of our living even as it is totally inhabited by hermeneutics.  I fall back on pragmatics, not because it is defensible, but because it is a way to go on.  Most other discussions are about drawing boundaries that are just too fluid to nail down in a convincing manner.