What to Do about Testing: A Response to Audrey Waters

Audrey Waters posted about John Oliver’s takedown of testing and Pearson Ed.  She asks:

How do we seize the opportunity of all this media attention to the problems with standardized testing to do more than talk about testing?  . . . Can we articulate (a better alternative) now so that Pearson and other testing companies don’t replace the old model with simply a re-branded, repackaged one?

Samuel Messick was a Vice President and Distinguished Research Scientist at Educational Testing Services (ETS).  His was an authoritative voice on test validity advocating for restraint in the use of test scores, better and more in-depth interpretations of test score. the collection of multiple sources of information for making important decisions and for consideration of the consequences of test use.  I believe that much of his legacy has been ignored, co-opted, or argued away (even at ETS I suspect).  I’ll speculate on what would he advocate;

  • using more than one or two sources of information when making complex important decisions,
  • understanding the information in the context of a decision and considering the consequences of your testing practices.
  • I also suspect that I could argue with him for the consideration of the validity of testing practices with how it fit within an overall set of district practices.  (i.e. If a student fails, how do you respond?)

Technically Pearson may not be at fault for it is the district use of tests that is most problematic, but Pearson is at least implicit in not providing better guidance and for developing ways for districts  to collect other sources of information.  Eg. The value added model of teacher assessment needs many more sources of information and in fact does not really provide an assessable model of pedagogy, only largely discredited positivist assertions. The first step is to expose those who advocate positivist models of empiricism for which even analytic philosophers would no longer advocate.

Finally it necessary to look at the overall model of education which is still primarily built of a mechanistic metaphor with the student as a vessel to be filled.  The metaphor should be a biological organism adapting in an environment that is primarily social, networked and interactive.  When Pearson speaks of their “potential game-changer: performance tasks”, they are talking in this direction, but their really co-opting performance tasks within the old metaphor.  They have a long way to go.  We should expunge the mechanistic metaphor from educational leadership and assessment models.

The bottom line for Pearson

You may not be technically wrong in your assessments, but when your the brunt of a comedic takedown, you should really look at the consequences of your products use and attempt to deal with it.

Education Needs Clarity

Beginnings: My Graduate Experience  (The 90s and the oughts)

My PhD was not motivated by a career path, but by my love of learning. Temple U’s Associate Professor Helmut Bartel (a proclaimed social constructionist) was an intellectual guide who helped me to recognize the relevance of social theories to my professional experiences; that is, I was by nature a pragmatist. Helmut left Temple before I could develop a dissertation topic and it was fortuitous because I needed to challenge myself to align my thoughts with new mentors. While trying to form a dissertation topic a professor said offhand, “It sounds to me that your talking about validity.” I read Messick’s chapter titled Validity in Linn’s (ed) Handbook of Educational Measurement. The references and the lineage of his ideas were all different, but the conversations where much the same and they centered around a pragmatic approach. The patrons of validity, Messick, Cronback and Meehl, were very clearly analytic in their thinking, but the logic of pragmatism was already deeply embedded in their thought.

Why Philosophy

My studies were in educational psychology, and I do find many discussion in philosophy to be tedious and boring, so why discuss philosophy.  Because, for everything we say, there are many things that are left unsaid and for everything we do, much of the reasoning is left unsaid and unquestioned.  The philosophy I discuss is about shining a light on practices to see what we are taking for granted and to understand what has been left unsaid. What we need is clarity, and that is precisely the purpose of philosophy in its analytic, neoanalytic and pragmatic forms.

Where is Validity in Educational Practice

How do you address validity questions that appear paradigmatically opposed to traditional empirical scientific practice? I begin with an adaptation of a thought who linage I trace Helmut. A successful paradigm change must account for the current paradigm in both its successes and failures in order to forge a true new order. The dominate and implicit practice paradigms today are still mostly based in a dualist objectivist analytic philosophy. Post-modern / post-structural and Marxis based critiques all excel at accounting for the ideological failures of an analytic approach, but not its successes. They fail to point to a way to move practice forward and seem to be losing steam, even as their critiques of analytic approaches remain valid. I think a better way is to consider pragmatism.
Pragmatism and Analytic Philosophy share a commitment to logic and the science method. What Pragmatism brings is a unity of science, practice and ethics (Boncompagni, 2001). Scientific practices are always situated in the midst of ethical horizons best understood as historicized ideological practices. This also matches my earlier experiences where I was working in disability services. The field was moving on from the least restrictive environment to minority rights and people first language. I thoroughly believe in the practicality of science, but science based practices were slow to adapt and often seemed to be standing in the way of ethically empowering practices. Obsessed with an unsustainable conception of objectivity, many scientists could not see how a lack of ethics impoverished science and made it weaker, not stronger.

Pragmatism to the Analytic and Back

I see the history of Pragmatism beginning with Peirce, James, Dewey and Mead, but it became overshadowed by the analytic approaches of European trained academics, especially those associated with the Vienna Circle. As problems were recognized in Analytic Philosophy there began a slow and constant evolution towards pragmatism. In Analytic Philosophy this included people and their ideas such as Quinn, Kuhn, and Wittgenstein. In educational psychology this included Cronbach, Meehl and Messick. This may not be exactly James’ or Dewey’s Pragmatism, but it’s much closer than the direction sought by the Vienna Circle or BF Skinner and I believe that a movement towards pragmatism continues today.

To understand pragmatic social science, let’s begin with Joseph Margolis’ claim: “language and what language uniquely makes possible in the way of the evolving powers of the human mind are emergent, artifactual, hybrid precipitates of the joint processes of biological and cultural evolution;” I see this as something like taking up the naturalism and social behaviorism of Dewey and Mead.  This approach may no longer provide  a foundation for infallible truths, but there is still room for an ethical, objective and empirically warranted practice. This social behavioral and empirical science should be distinguished from Skinner’s radical behaviorism in the same way logical positivism is distinguished from current analytic / pragmatic  approaches.   The knowledge radical behaviorism engenders, fails to adequately recognize the full nature of language and the social world it makes possible.  As a result radical behaviorism leaves knowledge as flat and shallow and more often results in situations (as Wittgenstein noted) where the educational problem and the method pass one another by without interacting. To be valid, empirical methods must reflect the contextualized, artifactual and ethical demands of the problems within a philosophically Darwinian framework of an organism’s adaptation to the social and physical environment. Adaptation is very personal and includes concepts like social poetics.  That is, I accept analytic tools and methods, but recognize them only within social ethical fields that are interpretive as above.  Just as analytic philosophy has moved back toward Pierce, James, Dewey and Mead, radical behaviorism can only be relevant by moving toward Vygotsky, Dewey, Wittgenstein and social poetics.


Margolis, Joseph (2012-10-17). Pragmatism Ascendent: A Yard of Narrative, a Touch of Prophecy (p. 133). Stanford University Press. Kindle Edition.

Boncompagni, A (2011). Book Review on New Perspectives on Pragmatism and Analytic Philosophy, EUROPEAN JOURNAL OF PRAGMATISM AND AMERICAN PHILOSOPHY, III, 2, 290-299. http://lnx.journalofpragmatism.eu/wp-content/uploads/2012/01/calcaterra-new-perspective.pdf

Garrison, J (1995). Deweyan Pragmatism and the Epistemology of Contemporary Social Constructivism, American Educational Research Journal, 32, 716-740.

Locating Performance in Pedagogy

Philosophy has a radical way of approaching and dealing with knowledge – for instance, it tries to overcome doctrines which do not question themselves and to compensate for the progressive drift of using and expanding knowledge only technically. Philosophy tries to understand the world . . ..  From: Lucian Ionel

As Lucian Ionel notes, this is an important part the philosophical method of Gregory Loewen’s Hermeneutic Pedagogy.  It’s yet another way of looking at the educational process and noticing what normally flys under the radar.  Loewen’s method seems to be categorizing pedagogy into three classes: Hexis, Praxis and Phronesis.  These 3, along with Episteme and Techne, form the intellectual foundation of Greek philosophical thought.  Episteme is concerned with aspects of knowledge and Techne is about craft or skills in production, both important, but Hexis, Praxis and Phronesis seem to make up the the core ideas of Loewen’s educational processes. I’m studying his approach and think that it might fit the direction of my recent thoughts about performance assessment.  This post is preliminary, about how my previous thought might map onto Loewen’s basic framework.

The specific analysis that Loewen pursues is decidedly Marxist and I do not share this approach.  For instance in Helix (introduced below) Loewen focuses on the reproduction of capitalist repression.  It’s true that current problems with inequality are an supported by the reproduction of a political economy, (see the Piketty discussion everywhere on the web these days), but I want to focus on the need for reproduction if we are to have any kind of culture.  We can discuss what should not be reproduced, but to stop reproduction would mean stopping culture itself.  Praxis also has a Marxist interpretation in Loewen and it has been a term with a substantial history in Critical Theory, but again, extension can be more than just a method for resistance.  Extension (as praxis) and phronesis (as wisdom) can be seen as the way in which culture remains a living and growing entity, able to adapt to current and future challenges.  Thus, I like Loewen’s analytic framework, I just disagree with it narrow NeoMarxist interpretation.  Indeed, it is possible that by extending this framework to approach all aspects of a complex and multifaceted culture based reality, it may be able to reflect back and re-approach it’s original intent from a more productive direction; though it is not my intention to pursue this.


I will key Helix as repetition and re-production.  It focuses on the passing of cultural knowledge.  In current educational practice, think of Helix as represent the standardized curriculums associated with No Child Left Behind and the Common Core.  These curriculum represent the basic knowledge that is expected by all citizens (re-production) and is (at least partially) achieved through memorization and direct instruction; pedagogy that is high in repetition.  Many current educational practices can be represented by Helix.


Praxis, generally understood as practice,  here is keyed as extension.   Think of representing applied knowledge that expands and changes according to the contexts and needs of practice; the learning necessary for practical performance.  This is often considered learning transfer, but in the wake of social cultural learning theory I think of this as extending by adding new learning.  This is not emphasized in current educational practice.  You can see it in activities such as creative writing, service learning or project-based learning, but it is often conceived as an after thought, not as a core educational component.

There are 2 things that should be  included in praxis education to make it more of a core goal of educational practice.  First, at this level you still want to provide lots of structure to these activities and to link them to existing curriculum.  Educational scaffolding can be used as the glue that links the curriculum to the activity structure.  Secondly, bring measurement into these performance activities. Measurement is a core component to education practice.  The inability to satisfactorily measure performance-based practice hurts its standing.  This means development not only in educational practice, but also development in educational measurement.  Note – This does not mean standardized assessment as currently practiced.  See this post on Ontologically Responsible Assessment for more info.


Phronesis is often translated as practical wisdom and it is the second part of my take on performance-based learning.  This is what I consider to involved higher levels of cognitive learning as well as what is often considered character education.  This certainly includes the higher levels of Bloom’s taxonomy, but broken down into more socially relevant skills that are more practice oriented and more socially oriented.  Bloom’s categories are overly individualistic and do not include socially interactive and practice relevant abilities that are becoming increasingly important for today’s workforce.  This is even more true of Bloom’s Affective and Psychomotor Domains which are more closely based on outdated behavioral theory.

Some of the qualities and cognitions to include are: problem identification and solving, creative thinking, situated strategic thinking, self-motivation, persistence, resilience, metacognition and self-directed learning, collaboration, effective situated communication and the ability to form strategic relationships.  For me this is similar to the Praxis level, but it is more open ended and with less structure and less dependence on specific curriculum.  At the praxis level, scaffolding was more knowledge based and emanated from standard curriculum.  At the Phronesis level, we’re moving toward a more skill and abilities foci.  Scaffolding at this level are more socially oriented and come from teachers or peers.

This Phronesis level asks a student to explore self-knowledge; not to just use knowledge in a technical sense, but also in a consciously creative and moral fashion.  This is Lucian Ionel quoting Loewen:

What is gained through this process is what we call self-knowledge: “Phronesis sees through the practicality of repetition and extension by seeing them as rationalizations for the world as it has been. In its subtle but forceful presence, the wisdom of reflective practice asks us to stand outside of the dominion of discourse, the caveat of custom, and move ourselves into the brightest human light of self-understanding anew.”

Where helix and praxis can be scripted (at least to a certain sense in praxis) phronesis is open-ended and reflexive.  It leads to process questions such as: Why is it this way; how have we arrived at this point?  What does or does not make sense here?  Can things be different?  How would you scale a new approach?

These skills and abilities are some of the most important personal qualities in personal success, but fall mostly outside of current educational practice.  They are not only the most difficult to measure, but measures tend to serve different purposes in the educational process.  The overall process is more relational and less mechanistic than at either the Helix or Praxis levels.  These measures must be concieved in more of a joint dialogical nature and less of an automated and behavioral fashion.  This does not mean that we give up on scientific objectivity or become less empirical in measurement.  But it does mean that we do not allow narrow definitions of empirical objectivity to constrict the construct we want to measure.  Narrow (and more traditional) measures represent the “doctrines which do not question themselves” and are the ones who fail “to compensate for the progressive drift of using and expanding knowledge only technically” which Ionel mentioned in the leading quote.

A Practice Perspective on the Quants and the Humanists

Lee Drutman responded to Timothy Egan’s New York Times Article about creativity and Big Data.

First TE says that companies like Amazon who are based on quantitative methods are not creative because they “marginalized messiness”.  LD responds that “(d)ata analysis and everything that goes into it can be highly creative”, meaning (I guess) that Quants can get down in the mess too.  Both are good points but miss another aspect that unites the arts / humanities and the sciences, and this is the heart of my argument.  They are both creating practices that effect our live in important ways.   The point is that we all create.  It’s not whether we are or are not creative.  It’s a question of what we are creating.  From John Shotter’s Cultural Politics of Everyday Life:

But now, many take seriously Foucault’s (1972: 49) claim that our task consists of not – of no longer – treating discourses as groups of signs . . . but as practices that systematically form the objects of which they speak.

In other words, it’s not wether the Quants are creative, but do their analyses treat me as an object to be controlled, or do they treat me as a human being where the analysis respects my being.  That’s called ontologically responsible assessment.  Again, from Shotter:

I want to argue not for a radical change in our practices, but for a self-conscious noticing of their actual nature.

We should offer people clear and understandable analysis where they can make new connections, but also respects and is responsible to their rights as a person.  Yes, as Lee claims, the sciences and the humanities can work together.  But beyond that, they are both human based social practices.  If we see them as practices a la Foucault, there is much more in common than is different.  They are both not only creative, but they are creating.

#PLENK2010 – Theory, Validity and Relativism

Thanks George; a good question I’ve been pondering for a couple days.

First issue; I’ll try to get to a better understanding of how I use of the term belief.  I’m thinking pragmatically here about how I act in most everyday situations, not in an idealized logical theoretical way.  In this context, I use the term belief not as an ill-formed or unsubstantiated theory, but more as a gestalt of everything I know that relates to the context in which I’m about to act.  Some of that may be informed by knowing that we’re not even conscious of drawing upon at the time we’re deciding and acting.  (Thinking of JG’s reference to intuition.)  I think that well founded theories are important, and hopefully, we get them into our gestalt beliefs in a way that will influence our actions to advance whatever practice we are engaged in.

Second issue, I’ll try to get at the relativism issue implied by your question.  Even though I love discussing theory, my primary concern has always been with practice.  (You might say, Phronesis informed by Pragmatism.)  Where I think Boghossian is arguing from first principles to achieve a version of a valid philosophical theory, I’m attempting to achieve the best practice possible by combining the best of all concepts and theories, as I understand them, that can move me towards that best practice.  I need well founded theories, but I’m interested in them in instrumental ways.  Now, that does lead me toward a relativist’s path, but I don’t argue for relativist first principles.  Not all viewpoints are equal and our arguments can be substantiated, but there are limitations to our thinking and knowing that should also be acknowledged.  I favored Joseph Margolis‘ explanation in grad school and I’ll go back to that now.

(According to Wikipedia) Margolis lists 5 themes in philosophy that have been gathering momentum since the time of Kant. (Emphasis added)

  1. Reality is cognitively intransparent. That is, everything we say about the world must pass through our conceptual schemes and the limits of our language, hence there is no way of knowing whether what we say “corresponds” to what there is; what the world is like independent of our investigating it;
  2. The structure of reality and the structure of thought are symbiotized. That is, there is no way of knowing how much of the apparent intelligibility of the world is a contribution of the mind and how much the world itself contributes to that seeming intelligibility;
  3. Thinking has a history. That is, all we take to be universal, rational, logical, necessary, right behaviour, laws of nature, and so on, are changing artifacts of the historical existence of different societies and societal groups. All are open to change and all are the sites of hegemonic struggle;
  4. The structure of thinking is preformed. That is, our thinking is formed by the enculturing process by which human babies become adults. The infant begins in a holistic space which is immediately parsed according to the norms and conduct and language she is brought up in. By taking part in the process, we alter it, alter ourselves, and alter the conditions for the next generation;
  5. Human culture, including human beings, are socially constructed or socially constituted. That is, they have no natures, but are (referentially) or have (predicatively) histories, narratized careers.

I don’t see this as a strong version of relativism that offers no possibility of making arguable judgements (I don’t know where he is now, but back in the 90s Margolis was willing to acknowledge that some of these, especially the social construction parts, could be open to argument).  It’s just that theories, judgements, scientific findings and the like are definitely limited by our cognitive abilities and situatedness.  As an example, one of my favorite topics is validity, which I conceive of as the degree to which evidence and theory support specific practices (I draw this from test validity, not philosophical validity).  You can make a judgement about the objectivity and correctness of a conclusion or test, but no matter how strong the evidence, validity never reaches 100%.  I must posses an openness to look at things in other ways, which also can be stated as, I expect science to progress by giving us better ways of understanding what we once thought differently about.

P.S. I generally thing of theoretical validity as how a theory is substantiated in a general sense.  Usually this will include the requirements of being predictive, descriptive, and testable, but I’ll usually judge that according to the context of my judgement, not in a prescriptive sense.

Richard Florida needs John Hagel

Richard Florida needs John Hagel.  Florida and Hagel have been referencing each other recently (here and here) and that’s a good thing; at least I think it is for Richard’s ideas.  Florida’s work has been very popular, but has also received a fair amount of criticism.  The best founded of the critiques (in my opinion) refer to the generality of his data.  It is correlational and very general, you might say it paints a picture with too broad of a brush.

The issue is that we know two things:

  1. We know a bit about how individuals can be creative and
  2. We know how creativity and innovations correlate with clusters of people, but

We don’t know other things like:

  1. We don’t really know much about the network ecologies and interactions that drive this increased creativity and
  2. We don’t have specific experiments to demonstrate and validate any kind of intervention and
  3. We don’t know if there is a critical mass for clusters to spur innovations.

Enter John Hagel who’s concept of “pull” begins to specify some of the ways and mechanism that may be behind the effects described by Florida. I think the next step is to describe in detail how individual environments work to spur innovation.  Many of the interventions inspired by Florida have semed a little like shooting in the dark.  What is needed is a little more specificity in how things work.  Maybe not to a prescriptive level, but just so we understand what can be successful and in what ways.  There have been a number of projects inspired by Richard, and I believe that his ideas are valid, we just need more causal analysis and measures of success to drive things forward; and in the proper direction.

Critical Thinking, Scientific Reasoning, and the Incorporation of Evidence into Everyday Practice: A Conceptual SymbiosisI

It seems to me that there is a natural affinity between evidence-based practice, scientific reasoning and critical thinking.  I think Kuhn (quoted in Dawson, 2000) captures the essence of this symbiosis:

I have undertaken here to show that these two abilities–the ability to recognize the possible falsehood of a theory and the identification of evidence capable of disconfirming it–are the foundational abilities that lie at the heart of both informal and scientific reasoning. These abilities lie at the heart of critical thinking, which similarly can be regarded, at the most global level, as the ability to justify what one claims to be true (Kuhn, 1993).

Some background considerations and directions for future thoughts and research.

  1. I’m taking the perspective that what cognitive control we have over our decisions and actions, is mediated by our beliefs, theories, schemas and prior knowledge.  Without this mediation everyday actions would represent an unbearable cognitive load.
  2. Although there are good strategies for enabling critical thinking, at it’s core, critical thinking is the ability and disposition to seek disconfirming evidence and use it to change our minds (beefs schemas, theories, etc. . . ).
  3. Although we often equate scientific thinking with the scientific method (hypothesis testing), the core of it’s reasoning is also the disposition to seek and make use of disconfirming evidence.
  4. Evidence-based organizations must actively support critical thinking through their culture and in the organization of their internal processes and practices.
  5. Practice validity (seeking evidence for the validity of organizational practices) is the ability to justify the efficacy of our actions, just as Kuhn considers critical thinking to be a way to justify our claims to truth.

A shout-out to Harold Jarche who’s post Critical thinking in the organization led me down this primrose path.


Dawson, R. (2000). Critical Thinking, Scientific Thinking, and Everyday Thinking: Metacognition about Cognition, Academic Exchange Quarterly, accessed 4-8–10 at http://www.thefreelibrary.com/Critical+Thinking,+Scientific+Thinking,+and+Everyday+Thinking:…-a067872702

Kuhn, D. (1993). Connecting scientific and informal reasoning. Merrill-Palmer Quarterly, 39(1), 74-103.

Two Different Ways of Implementing Evidence-based Practice and their Different Requirements for Evidence

It intuitively seems to me that there are two way of applying evidence in Evidence-based Management.

  1. One I’ll call evidence-based decision-making (EBDM), bringing evidence into decision processes.
  2. The other I’ll refer to as evidence-supported interventions (ESI), specific practices that are empirically supported.

I suspect that EBDM will be a tougher nut to crack in practice.  This is because decision-making is often context dependent, involves ill structured problems, and can be cognitively complex.  (See March, 1991; for one take on this complexity.)  Decision processes require a higher level of interpretation regarding the evidence and can easily fall prey to logical errors.  Most thinking on decision-making has stressed that research should begin by analyzing of how people make decisions in real time, not as some sort of abstract logical process.  As Daniel Kahneman (2003) puts it;

psychological theories of intuitive thinking cannot match the elegance and precision of formal normative models of belief and choice, but this is just another way of saying that rational models are psychologically unrealistic ( p. 1449).

Nonetheless, evidence should inform decision processes and I believe that evidence supported protocols, as one example, can prepare the decision space for better decision-making outcomes.  However, this type of process also begins to bring me closer to the second way of applying evidence; through evidence-supported interventions.

Mullen, Bledsoe, & Bellamy (2008) define Evidence-supported Interventions (ESI) as

specific interventions (e.g., assessment instruments, treatment and prevention protocols, etc.) determined to have a reasonable degree of empirical support.

(Other names might include evidence-based practices, empirically supported treatments, or empirically informed interventions.)  In implementation settings, ESIs function as standardized practices; practices where all or a portion of the operational, tactical, logistical, administrative or training aspects of a practice are able to conform to a specific and unified set of criteria.  In other words, the contexts of implementation will allow practice to be replicated exactly as they were defined and constructed in supporting research.   In being evidence-based, it is important that critical issues flow both ways.  If the contexts do not allow replication, or present confounding variables and complexity not addressed in research, it will necessarily reduce the level of support that can be claimed for any research supported practice.

There are many differences between EBDM and ESIs.  I would like to focus here on the different role that theory plays in each.  There are no data or practices that are completely theory free.  All are theory and value laden to some extent.  All datum, hypothesis, or knowledge depend on assumptions and implications that are based in someway on theory.  But, all do not depend in the same way or to the same extent.  I will borrow on Otto & Ziegler (2008) to explain how some of these differences can be ascribed to either causal descriptions or causal explanations.  First, I agree with Otto and Ziegler who say that

Probably, it is fair to say that the quest for causal explanation is theory driven, whereas causal description is not necessarily grounded in theory.

ESIs, because they focus on replication, do not need to be as concerned with the fact that they are relying on causal descriptions.  EBDM, however, are using evidence in a more theoretical way than in the replication of a standard practice.  Because they are dealing with complex and context independent reasoning, they need evidence that is valid in a causal explanative manner.

Two observations – From a strict positivist perspective, this creates a problem for EBDM because of the difficulty in achieving a necessary level of causal explanation.  Positivism can live better through a ESI approach because it can depend on causal description.  Instead an EBDM approach must adopt an argumentative type role in validating evidence.  This is the approach that validity theory has taken.  Validity theory began with a positivist framework that was centered on a criterion approach to validity.  As it became more and more apparent that constructs were the central concern (theoretical concerns) it adopted a unified construct validity perspective that needed an argumentative approach.  This is an approach where validity is never an either or proposition, but rather a concern for the level of validity achieved.  While this is not necessarily the most clear way, it is very pragmatic and practical and able to be implemented across a wide variety of practice locations.

Two Conclusions:

  1. EBDM is concerned with supporting naturalistic decision processes with evidence that is empirically and theoretically supported and can be a easily included in that decision process.
  2. ESIs are concerned with practices, protocols and processes that can function in a standardized manner through the replication of empirically supported research interventions.


Kahneman, D., (Dec., 2003). Maps of Bounded Rationality: Psychology for Behavioral Economics, The American Economic Review, Vol. 93, No. 5 , pp. 1449-1475.

March, J.G., (1991).  How Decisions Happen in Organizations, Human-Computer Interaction, 6, 95-117. accessed 02-15-2010 at http://choo.fis.utoronto.ca/fis/courses/lis2176/Readings/march.pdf

Mullen, E.J., Bledsoe, S.E. & Bellamy, J.L., (2008). Implementing Evidence-Based Social Work Practice, Research on Social Work Practice, Vol. 18 No. 4, July 2008 325-338.

Otto, H., & Ziegler, H., (2008). The Notion of Causal Impact in Evidence-Based Social Work: An Introduction to the Special Issue on What Works? Research on Social Work Practice, Vol. 18 No. 4, July 2008 273-277.

4 Types of Evidence-based Practitioner Information Needs

This is a thought in development, not a finished product.  I currently can think of 4 different types of evidence-based information that would be of interest practitioners: the structure of practice, the scope of practice, the applicability  (the level of confidence that the evidence is applicable to your specific context), and the measured consequences of practice (intended or unintended).
1. Form – How should my practice be structured according to the evidence from best practice models and all forms of evidence.  What do we know about how the practice or protocol should be structured.  Is there evidence for a correspondence between the theoretical proscribed structure and the actual practice I’m reviewing.
2. Scope – What different aspects should be included in my practice.  What different types of actions are important for goal achievement.  Does my local process include all aspects demonstrated to be important in a successful practice.
3. Applicability – Do the models generalize well to my specific situation.  Just because research was valid for college sophomores does not necessarily mean I should have confidence that the evidence generalizes to my situation.
4. Consequential – Are my local measures consistent with and confirm what the evidence predicts should happen. Include intended and unintended consequences.  In addition to external research information, local measures should  also be an important source for generating evidence.

Evidence-based Practice Defined

Evidence-based practice (whether in management, education, medicine or other) describes a process designed to facilitate an integrative and evaluative judgement, based on empirical evidence and theoretical rationale, as to the appropriateness and adequacy of actions or proposed actions, that are based on the processes or protocols of a practice or program.
Anyone familiar with the validity theory of Samuel Messick will recognize that this is drawn from his language.  That is because of my belief that the evidence-based movement should focus on practice in a way that can best be understood as practice validity (or process validity, or program validity depending on your perspective and task)
Evidence-based practice represents a multifaceted judgment. This is not a cookbook or a cherry picking approach.  It is a multifaceted judgement integrating all relevant information to evaluate actions as to their appropriateness for specific contexts and goals.  These actions are based on empirically supported theoretical rationales that are also backed up by locally collected data.
Evidence-based practice is scientific inquiry. This approach does not just rely on the results of scientific inquiry, it represents a form of scientific inquiry, but instead of being directed toward some aspect of theory (an a-contextual methodology), it is directed toward a specific practice (a contextualized methodology).  Though much of the literature equates an evidence-based approach with a decision-making process, the application of evidence is often focused on processes (or standing practices, or programs) and whether these processes are adequate and appropriate toward intended goals.
Evidence-based Practice should be associated with valid clinical or practice judgment.  A common criticism of the cookbook approach to EBP is that it impinges upon the ability of clinicians to make clinical judgements, but that is only true if one does not recognize that EBP is itself a judgement.  It is a form of clinical judgement and should occupy an important space within clinical practice.  This is an important point because research information seldom exactly fits the contexts of practice.  Applying this type of information is not direct, but requires making an informed judgment.  This is also an area that needs additional study.  I don’t believe that we really fully understand all aspects (especially cognitive aspects) of clinical judgment.