Naturalistic Decision-making or Algorithmic Practice: Which is Appropriate and When

Interesting Article in APA’s American Psychologist.

Kahneman, D. & Klein, G. (2009). Conditions for Intuitive Expertise: A Failure to Disagree, American Psychologist, 64, #6, 515-524.

The question, what works best, the intuition of expert decision-makers (Naturalistic Decision Making) or a statistical prediction algorithmic approach (Heuristics and Biases).

The answer of course, it depends on the context.

Intuition (which is presented as a form of pattern recognition) works well when the context include clear and consistent patterns and the experts has ample opportunities to practice recognition.

Where simple and valid clues exist, humans will find them given sufficient experience and enough rapid feedback. (p. 523)

This expert pattern recognition type of decision-making is especially relevant when time is a factor like in nursing or firefighting.  In situations where there are contra-indications, an algorithmic would be warranted, but the authors note there may be a potential for push back from practitioners.

An important point here is that an evidence-based approach is portraited not a simplistic application of science, but rather the development of a specific practice oriented algorithm – an scientific extenuation of the practice.

Contra-indications for a naturalistic decision-making process would include:

  • weak or difficult to detect patterns (e.g. high ceiling effects),
  • the lack of feedback,
  • feedback over long time periods or situations involving wicked problems where the feedback is misleading.

Contra-indications for a hubristic algorithmic approach include:

  • a lack of adequate knowledge about relevant variables,
  • reliable criterion,
  • a body of similar cases,
  • a cost benefit ratio that allow for algorithm development,
  • a low likelihood of changing conditions that would render the algorithm obsolete

The authors also note that algorithmic approaches should be closely monitored for changing conditions.

My take: Kahneman and Klein set up their discussion as a debate between themselves and discuss different approaches primarily as an either or choice.  I value their clarifications, but I would like to think of the many other situations where algorithms would be appropriate to supplementing not replacing naturalistic decision-making.  For instance, they use nursing diagnoses as an example of a reliable intuition space.  In some situations it is appropriate to use it, however diagnosis is a complex tasks that can include a large amount of data that can be combined in different ways.  I’ll have to look at the literature to see if there is a contra-example for naturalistic decision-making.  I’m not saying that naturalistic decision-making is inappropriate in many situations, only that they seem to be short changing algorithmic approaches.  There are also indications that these to authors are not sharing a philosophical heuristic framework.  My bet is that the positivist side is overstating naturalistic bias (which mean failing to see their own) and the naturalistic side is ignoring sources of bias when is suits them (throwing our the scientific baby with the bath water).  Again this is pointing to a need for a framework that can being people with different perspective into true communication and exchange.

Howe’s Critique of a Positivist Evidence-based Movement with a Potentially Valid Way Forward

A summary of Kenneth Howe’s article criticizing positivism and the new orthodoxy in educational science (evidence-based education).

(Howe, K.R., (2009). Epistemology, Methodology, and Education Sciences: Positivist Dogma, Rhetoric, and the Education Science Question, Education Researcher, 38 (#6) pp. 428-440.

Keywords: Philosophy; politics; research methodology

“Although explicitly articulated versions (of positivism) were cast off quite some time ago in philosophy, positivism continues to thrive in tacit form on the broader scene . . . now resurgent in the new scientific orthodoxy.” (p.428)

(A positivist stance on science) has sought to “construct a priestly ethos – by suggesting that it is the singular mediator of knowledge, or at least of whatever knowledge has real value . . . and should therefore enjoy a commensurate authority” (Howe quoting Lessl, from Science and Rhetoric).

Howe traces the outline of this tacit form of positivism through the National Research Council’s 2002 report titled Scientific Research in Education and relates this report to three dogmas of positivism:

  1. The quantitative – qualitative dichotomy – A reductionism dogma that had the consequence of limiting the acceptable range of what could be considered valid in research studies.
  2. The fact value distinction – An attempts to portray science as a value free process with the effect of obscuring the underlying values in operation.
  3. The division between the sciences and the humanities. Another distinction of positivism designed to limit any discussions to a narrow view of science.

Howe’s article does a good job of summarizing these general critiques of positivist methodology, which include: (1)its overall claims could not stand up to philosophical scrutiny, (2) it tended to not recognize many of its own limitations including applying adequate standards to itself and (3) it also was inhabited by a political agenda that sought to stifle and block many important directions that inquiry otherwise might have taken.

The crux of the political matter: While the goal of positivism may have been to positively establish a objective verifiable method of conducting social science modeled on the physical sciences, the primary result was an attempt to politically limit the scope of what could be considered meaningful scientific statements to include only statements that were verifiable in a narrow positivist sense. Howe is among the cohort who believe that the evidence-based movement is being used by some as a context to advance a tacit return to a form of positivism.

The crux of the scientific matter: Howe’s primary interest appear to be political, the politics of how research is received and funded, but there is also an effectiveness issue.  Positivism’s primarily scientific problems are in the tendency to ignore or to down play many of the limitations of positivist methods, (overstating the meaning of positivist research) and in the way it oversimplifies and fails to problematizes the rather complex relationship between research and practice.

Messick’s Six Part Validity Framework as a Response

There are four responses to Howe in this journal issue. To me, none of the responses address the primary issue at play: to bring some sense of unity to varying ideas and communication with people using different scientific methodological frameworks.  There are suggestions to allow for multiple methods, but they are more of a juxtaposition of methods rather than a framework that serves to guide and support communication and understanding among scientists use differing methods.  This is why I support Messick’s validity framework as a response to just this type of concern.  Although Messick spoke specifically of test validity, there is nothing that would preclude this framework from being applicable to practice validity and to the development of post-positivist evidence to support the validity of practices.  What is the evidence-based movement really concerned with, if it is not the validity of the practices being pursued by practitioners.  This is not primarily about the validity of individual research studies, but is about the validity of practices and developing evidence to support the validity of specific practices.  It is also a mature framework that considers the full range of inquiry when developing evidence.

Messick’s six areas for developing validity are six different types of validity evidence and I develop here an initial set of ideas about how they might relate to evidence-based practice as follows:

  • Content – Content defines the scope of the practice domain with evidence (including rationales and philosophical debates) for the relevance of a particular practice, how the practice represents ideas within the general domain and the technical quality as compared to other examples of the practice.
  • Substantive – Evidence that the design of actual processes involved are consistent with the knowledge of design from relevant domains (i.e. psychology, sociology, engineering, etc. . ..)
  • Structural – The consistence between the processes involved and the theories that underly and support rationales for the structure of the actual process.
  • External – Empirical data to support criterion evidence (Random controlled trials (RCT) would be one example).  For many practices this may include both convergent and discriminant evidence.  (My thinking is still in development here, but I am think that empirical evidence from the research base would function more like criterion evidence.  Direct empirical evidence from the actual practice being validated would be considered in most situations under consequential evidence.  See below.)
  • Generalization – Evidence for the practice to be relevant and effective across different populations, contexts and time periods.
  • Consequential – Evidence that the practice is actually achieving the purpose it was originally intended to achieve.

I consider this list to be an early formation with more development needed.  Critiques are most welcome.

Messick’s original formulation for test validity is available here.

Design, Hermeneutics, Wittgenstein and Our Ethical Commitment to the World

I. A New Understanding of Design

The Harvard Business Ideacast #160 is an interview with Roberto Verganti, the author of Design Driven Innovation – Changing the Rules of Competition by Radically Innovating what Things Mean.  Verganti’s ideas about design point to the etymology of design, derived from the latin designer – to designate, or as Verganti presents it; to ascribe meaning.  For Verganti, designing seems to be an act of hermeneutics; finding interpretations that change the meaning of things or services.  One of examples that Verganti gives is Sony, who invented the idea the personal and portable music player with the Walkman; later digitalized as the CD playing Diskman.  Later, it was Apple that put music on portable hard disks and subsequently changed the meaning of a personal portable music device, leading to the success of the ipod.  The marketing success of these products was not due to any kind of technical advantage.  Many companies could have put music on portable hard disks including Sony.  Success came in how Apple was able to changed the meaning and the place of music in people’s lives.  In other words, it was a hermeneutic act.  Understanding design as interpretation helps to clarify how design thinking is relevant to human activity across many functions.  First, a short diversion that will hopefully lead to a deeper understanding of hermeneutics.

II. Hermeneutics in Contemporary Philosophy

I’m not a professional philosopher, but hermeneutics was a buzz word from my graduate days in the 90s and here is my take on it.  Whether you trace the philosophical line of thought from Schleiermacher to Gadamer or from Nietzsche to Derrida, hermeneutics and meaning has been playing a central role in contemporary philosophy.  For me, hermeneutics culminates in the later ideas of Wittgenstein because of the scope of his thought that includes important spaces for science, and ethics.

Humans are meaning making organisms that are realized as they participate within situated forms of life.  Just like the story of Adam and Eve naming the animals, we actively experience the world around us, we ascribe meaning to the world and to those experiences and there seems not else we can do.  Words and experiences are better understood if they are not thought of as abstract representations of the mind, but rather, the means of hermeneutic action in the context of a lived life. Deed before word.  This draws from the idea that we do not directly experience the world, but whether we are perceiving objects or our experiences, they are understood by meaning creating mediators like scientific methodology.  Changing the meaning changes our basic understanding of what the object or experience will be.

According to Janik (2002), Wittgenstein took many fundamental ideas from Heinrich Hertz who believed that “rhetorical adequacy is as important as architecture” (see p. 8 ) when talking about scientific models.  Hertz gave three criteria for scientific models

  • They must be logically permissible, (i.e., internally consistent, empirically correct)
  • They must be communicatively appropriate or effective.
  • They must have usefulness in a given situation.

You can see in Hertz, the origin of the thought that even scientific meaning is derived from action; again, the idea of deed before word or action that leads to meaning.  This is also the space where ethics enters into the conversation.  A focus on action also leads to discussions of usefulness and of the need to evaluate the consequences of action.

Contemporary hermeneutics is not trivial.  It is a profound view of the world and what human are thought to be.  It was well expressed in a quoted passage of Slavoj Žižek’s Parallax View, recently discussed and posted by Jeff Meyerhoff in Philosophy Autobiography.

. . . from Kierkegaard and Nietzsche to the late work of Wittgenstein, the most radical authentic core of being human is perceived as a concrete practico-ethical engagement and/or choice which precedes (and grounds) every “theory,” every theoretical account of itself, and is, in this radical sense of the term, contingent . . . (Quoting Fichte) “What philosophy one chooses depends on what kind of man one is.” . . . in the last resort there is not theory, just a fundamental practice-ethical decision about what kind of life one wants to commit oneself to.

III. As a final task, I will look at hermeneutics as design from the different ideas that I’ve been discussing recently:

  • Validity – As I’ve said before, Messick’s idea of validity can be thought of as a hermeneutics of measurement and that in turn also serves for me as another path that hermeneutics enters into an account of science.  Science (through measurement and theory) is not a raw empirical experience of the world, but is a hermeneutic application of experience.  It should not be divorced from art as it so often is, but both are still contingent to one’s commitments and practice-ethical decisions.  The Messick account of validity is consistent with Wittgenstein when he emphasizes that validity is found in the interpretation of test use, not in the abstract qualities of a test and when he says that validity should include an evaluation of the consequences of assessment.  An obvious statement of ethical import.
  • In response to an earlier query from Ann Burdick, who wonders why non-designers are so active in design conversations, design, as it is in Verganti’s hermeneutic act, is a form of life that is understood at some level by all humans.  Artists, directors, writers and the like acquire specialized skills and expertise at interpretation within the mediums of their specialization, but at some level; we are all practico-ethical designers and artists of our lives and of those around us that we touch.  Specialization allows them the ability to speak for the broader society, but all people have a need to act designerly.
  • In terms of Fred Collopy’s Management by Design, this is an acknowledgement of the central role of managers as practical ethical interpreter heros, leading society through the chaotic world of business.  Designing managers must master the range of hermeneutic  tools that allows managers and the organizations they lead to re-interpret and to change the meanings of their historical circumstances in route to envisioning and acting on a new and changed future.
  • In terms of the evidence-based movement, evidence and science are profound tools of interpretation, just as they were for Wittgenstein.  You might say, they are the sword and shield of our business hero.  But our manager heros are also like King Arthur in that his true strength is not in his weapons, but in his commitment to those around him, they to him, and in the practical and ethical choices they all make, or fail to make.

Evidence-based approaches, science, art, design, theories, words; these are the tools of our choices and they are not trivial tools.  They are the best and most productive tools we have to create our lived world, but they do not release us from the need to make the ethical choice.  And as Bob Dylan said; “You’re gonna have to serve somebody”.

Where are we going?

The following is part of a response to a post by Richard Puyt and follows a sort of challenge by George Siemens.

A business person should really the ultimate as an educated person.  The social sciences are important, scientific methodology and statistics, mathematics, engineering, design, technology, I could go on and on, but mixing all that stuff together and keeping things in balance is also difficult  (See this article that is also relevant to EBMgmt)

Maybe it’s time not to specialize and professionalize in business education, but to re-define what a liberally educated person should be.  The liberal arts studies are rooted in the 19th century.  What would be a liberal education for the 21st Century.  We might be at a beginning.  Blogs have become the education commons or agora connected with open source education.  What would you consider as the most important items to include in a common body of business management?  Where are we going?  A quote from George Siemens is relevant:

In my youth, I went on a silent spiritual retreat. Days without speaking – except for ~1 hour each day with a spiritual adviser. On day 3, he made a statement that has guided much of my thinking since: never move away from something – you never know where you’ll end up…always walk toward something – this ensures you end up where you want to be. If we desire to do away with universities because we think they are obsolete (and in many ways, they are), we really don’t know what the future will look like. Change is about moving toward what we desire. But many reform advocates are not really clear on this yet. For that matter, I’ll direct the question to you: What type of higher education system are you moving toward? What are you working to achieve?

That really is the question isn’t it!  What future do we want to create?  Times a-wasting!

Evidence-Based Management as a Research/Practice Gap Problem

This is a response I made to a post on the Evidence Soup Blog about the potential demise of EBMmgt
I’ve been think about the health of the movement in response to (Tracy’s) post and I’m still surprised by the lack of EBMgmt discussions and how the movement does not seem to be gaining much traction. I re-looked at the Rousseau – Learmonth and the Van De Van, Johnson – McKelvey discussions for potential reasons why. (both are in Academy of Management vol31 #4, 2006). Here’s my take after reading them:
(1) Cognitive, Translation and Synthesis Problems: One, just like the example Rousseau gave in her Presidential Address, there are too many different concerns and issues floating about. We need the field to be more organized so people can get a better cognitive handle on what’s important. Also, I’m not sure peer review is the best strategy. When I did my dissertation, doing something exciting took a back seat to doing something bounded and do-able. I can’t imagine someone whose publishing for tenure doing anything more than incremental and that does not translate well for cognitive translation reasons. We need a synthesis strategy.
Possible response – A EBMgmt wiki See my 7-31 post on scientific publishing at howardjohnson.edublogs.org
(2) Belief problems – Henry Mintzberg believes that managers are trained by experience and MBA programs should be shut down. (3-26-09 Harvard Business Ideacast) He says that universities are good for that scientific management stuff, but implies that science is only a small part (management’s mostly tacit stuff). All my previously mentioned discussions noted that managers and consultant do not read the scientific literature. Part of the problem is communication (see #1), but part is current management paradigms that include little science.
Possible response – Far be it from me to suggest how to deal with paradigm change.
(3) Philosophical Problems – If EBMmgt is to succeed, it must be presented as a post-positivist formulation. Taken at face value, it seems positivist; and positivism has been so thoroughly critiqued that I could see where many people would dismiss it out of hand. Part of my thing is trying to be post-positivist, without throwing out the baby with the bath water. Rousseau tries to mollify Learmonth’s concern that touches on this area, she sees some issue, but I don’t see understanding. A positivist outlook will only lead you in circles.
Possible response – It’s much like your previous post, you need “both and” thinking, not “either or” thinking. EBMgmt must be an art and a science. This is how I understand the validity issue that I’ve mentioned to you before. I use Messick’s validity as a model for post-positivist science. It’s also important because measurement is the heart of science.
I would love your thoughts