Two Different Ways of Implementing Evidence-based Practice and their Different Requirements for Evidence

It intuitively seems to me that there are two way of applying evidence in Evidence-based Management.

  1. One I’ll call evidence-based decision-making (EBDM), bringing evidence into decision processes.
  2. The other I’ll refer to as evidence-supported interventions (ESI), specific practices that are empirically supported.

I suspect that EBDM will be a tougher nut to crack in practice.  This is because decision-making is often context dependent, involves ill structured problems, and can be cognitively complex.  (See March, 1991; for one take on this complexity.)  Decision processes require a higher level of interpretation regarding the evidence and can easily fall prey to logical errors.  Most thinking on decision-making has stressed that research should begin by analyzing of how people make decisions in real time, not as some sort of abstract logical process.  As Daniel Kahneman (2003) puts it;

psychological theories of intuitive thinking cannot match the elegance and precision of formal normative models of belief and choice, but this is just another way of saying that rational models are psychologically unrealistic ( p. 1449).

Nonetheless, evidence should inform decision processes and I believe that evidence supported protocols, as one example, can prepare the decision space for better decision-making outcomes.  However, this type of process also begins to bring me closer to the second way of applying evidence; through evidence-supported interventions.

Mullen, Bledsoe, & Bellamy (2008) define Evidence-supported Interventions (ESI) as

specific interventions (e.g., assessment instruments, treatment and prevention protocols, etc.) determined to have a reasonable degree of empirical support.

(Other names might include evidence-based practices, empirically supported treatments, or empirically informed interventions.)  In implementation settings, ESIs function as standardized practices; practices where all or a portion of the operational, tactical, logistical, administrative or training aspects of a practice are able to conform to a specific and unified set of criteria.  In other words, the contexts of implementation will allow practice to be replicated exactly as they were defined and constructed in supporting research.   In being evidence-based, it is important that critical issues flow both ways.  If the contexts do not allow replication, or present confounding variables and complexity not addressed in research, it will necessarily reduce the level of support that can be claimed for any research supported practice.

There are many differences between EBDM and ESIs.  I would like to focus here on the different role that theory plays in each.  There are no data or practices that are completely theory free.  All are theory and value laden to some extent.  All datum, hypothesis, or knowledge depend on assumptions and implications that are based in someway on theory.  But, all do not depend in the same way or to the same extent.  I will borrow on Otto & Ziegler (2008) to explain how some of these differences can be ascribed to either causal descriptions or causal explanations.  First, I agree with Otto and Ziegler who say that

Probably, it is fair to say that the quest for causal explanation is theory driven, whereas causal description is not necessarily grounded in theory.

ESIs, because they focus on replication, do not need to be as concerned with the fact that they are relying on causal descriptions.  EBDM, however, are using evidence in a more theoretical way than in the replication of a standard practice.  Because they are dealing with complex and context independent reasoning, they need evidence that is valid in a causal explanative manner.

Two observations – From a strict positivist perspective, this creates a problem for EBDM because of the difficulty in achieving a necessary level of causal explanation.  Positivism can live better through a ESI approach because it can depend on causal description.  Instead an EBDM approach must adopt an argumentative type role in validating evidence.  This is the approach that validity theory has taken.  Validity theory began with a positivist framework that was centered on a criterion approach to validity.  As it became more and more apparent that constructs were the central concern (theoretical concerns) it adopted a unified construct validity perspective that needed an argumentative approach.  This is an approach where validity is never an either or proposition, but rather a concern for the level of validity achieved.  While this is not necessarily the most clear way, it is very pragmatic and practical and able to be implemented across a wide variety of practice locations.

Two Conclusions:

  1. EBDM is concerned with supporting naturalistic decision processes with evidence that is empirically and theoretically supported and can be a easily included in that decision process.
  2. ESIs are concerned with practices, protocols and processes that can function in a standardized manner through the replication of empirically supported research interventions.

References

Kahneman, D., (Dec., 2003). Maps of Bounded Rationality: Psychology for Behavioral Economics, The American Economic Review, Vol. 93, No. 5 , pp. 1449-1475.

March, J.G., (1991).  How Decisions Happen in Organizations, Human-Computer Interaction, 6, 95-117. accessed 02-15-2010 at http://choo.fis.utoronto.ca/fis/courses/lis2176/Readings/march.pdf

Mullen, E.J., Bledsoe, S.E. & Bellamy, J.L., (2008). Implementing Evidence-Based Social Work Practice, Research on Social Work Practice, Vol. 18 No. 4, July 2008 325-338.

Otto, H., & Ziegler, H., (2008). The Notion of Causal Impact in Evidence-Based Social Work: An Introduction to the Special Issue on What Works? Research on Social Work Practice, Vol. 18 No. 4, July 2008 273-277.