Understanding Paul Meehl’s Metascience: Pragmatism for Empirical Social Science

Some recent involvement in LinkedIn conversations has led me to delve more into Paul Meehl’s work in Philosophy of Science or what he referred to as scientific metatheory.  As the book A Paul Meehl Reader notes, Paul’s essays were painstakingly written and most readers do not read his work so much as they mine his work for insights over many years; so I suspects this will be a long term project.

Here is the first nugget: progress in the soft sciences is difficult and painstaking and much of the existing research work mat be flawed and found wanting. Here are some reasons:

  1. Theory testing often involves derived auxiliary theories which, if not highly supported themselves, will add unknown noise into the data.  Often these theories are also not spelled out or understood.
  2. Experimenter error, experimenter bias, or editorial bias is present more often than is generally acknowledged or even known or considered.
  3. Inadequate statistical power.  In general, much more power is needed.  Meehl thinks that we should often seek statistical power in the .9 range in order to overcome unknown noise (error) in the data.
  4. Seriously accounting for the crud factor (the possible effect of ambient correlational noise in the data).
  5. Unconsidered validity concerns.  The foundation of science is measurement, but often the validity of measurement tools are not considered seriously.  Experiments are often measuring things is new ways even if they are using well studied instrument and this requires analysis for validity.

What this means is that more methodological care is needed such as:

  1. Seeking predicted point values that are stronger in terms of falsification and lend more verisimilitude than the often weak corroboration that come from non-null significance testing.
  2. More power (i.e. .9) in hypothesis testing to protect against weak auxiliaries, unknown bias and general crud.
  3. Understanding the difference between statical significance and evidentiary support.  Observations are evaluated in terms of statistical hypotheses and are a statistician’s concerns about the probability of the observations.  But theories are evaluated by the accumulation of logical facts.  These are not evaluated in terms of probabilities, but in terms of verisimilitude.
  4. Science should seek more complete conceptual understanding of the phenomena under study.

I believe this last point is similar to Wittgenstin’s concerns that in science problem and method often pass one another by without interacting.  I think this concern is also similar to verisimilitude in theory. Verisimilitude maybe considered a fussy interpretive concept, but the problems uncovered by the Reproducibility Project show that hard sciences are not as interpretive free as is often supposed. I’m also coming to the conclusion that it is in Meehl (and the like minded Messick) that traditional empirical science and pragmatism can be brought together.  It is the idea that a social constructivist approach must account for both the successes and the failures of empirical science if it is to move forward productively.  Meehl and Messick were not pragmatists, but I am saying that in dealing with the problems thay saw in empirical science, a critical pragmatic approach can be envisioned.  As Meehl along with Wittgenstein, Popper and maybe Lakotos are some of the best critics within the empirical sciences and building from their critiques seems like an interesting place to explore.



A Psychological Framework for Studying Social Networks

This is a short review of an article I found interesting.

Westaby, J.D., Pfaff, D.L. & Redding, N., (2014). Psychology and Social Networks: A Dynamic Network Theory Perspective, American Psychologist, 69, 269-284.

The authors note that a psychological perspective on social networks is rarely taken and advocate for more research.  They define 8 (psychological) roles that are thought to be played in these networks regarding goal achievement that they present as a framework to encourage more research.  The roles are:

  1. Goal Striving; directly attempting to achieve a specific goal.
  2. System Supporting; supporting those in goal pursuit.
  3. Goal Preventing; actively working to prevent goal achievement.
  4. Supportive Resisting; supporting goal preventeurs.
  5. System negating; responding with negative affect such as making fun of a person who is goal striving.
  6. System reacting; responding with negative affect toward those resisting goal pursuits.
  7. Interacting; People who can affect goals even though they do not intend to support or resist.
  8. Observing; People who only observe network activity, but nonetheless ca be involved in unintended effects.

This framework could make for an interesting analysis of networks and may have practical relevance for a wide variety of practices.  It may prove to be hard to disentangle the effects wrought by multiple or even conflicting goals in complex environments, or with fluid and changing alliances and more study is needed, however it may be interesting to follow.

Validated Methodological Operationism: Improve Analytics by Validating Your Operations

Many measured processes can be improved by validating your process operations.  This is true whether your are talking about business, experimental, or educational processes.

A New View Of Operationism

Interesting read on operationism by Uljana Feest – (2005)  Operationalism in Psychology: What the Debate is About, What the Debate Should Be About [Journal of the History of Behavioral Sciences, 41(2) 131-149].

The basic gist: Psychologist’s historical use of operationalism was methodological rather the positivist (even though they may have referenced positivism for philosophical cover).  So criticizing operationism using positivist arguments is somewhat misguided, but operations can be criticized through validation arguments.

What does Feest mean by a methodological reading of operationism?

. . . I mean that psychologists did not intend to say, generally, what constitutes the meaning of a scientific term.  . . . in offering operational definitions, scientists were partially and temporarily specifying their usage of certain concepts by saying what kind of empirical indicators they took to be indicative of the referents of the concepts (p. 133).

She concludes by saying:

. . . the debate should then be about what are adequate concepts and how (not whether) to operationalize them, and how (not whether) to validate them (p.146).

So any debate about operationism is really about constructs and their validation.  Within this framework, I will list 4 specific types of operationism.

Positivists, Empirist Operationism

This idea can be represented by Percy Bridgman’s original conception of operationsim

in general, we mean by a concept nothing more than a set of operations; the concept is synomonous with the corresponding set of operations (Bridgeman, P.[1927]. The logic of Modern Physics, Macmillan:NY. p.5).

The biggest problem with this approach is that any set of operations can never be said to exhaust the entirety of meaning in any construct, a position that is also supported by cognitive psychology’s understanding of cognitive processes in the meaning and use of concepts (Andersen, H., Barker, P & Chen, X. (2006). The Cognitive Structure of Scientific Revolutions, Cambridge University Press).

Methodological Operationism

The idea that operations are the empirical indicators of the construct (Feest).

Naive Pragmatic Operationism

Regardless with how you conceive of a construct, within any measured process, no matter if that process is an experimental, business or any other process that is controlled by measures, those measurement operations are methodologically defining that construct in the function of that process.  If you throw any measure in place without determining how and why you are using that measure, you are operating in the same fashion as any operationists in the positivist empiricist mode and you are subject to the same kinds of problems.  Garbage in = garbage out; this is the real potential problem with this approach.  There are many business process that do not meet their expectations and those problems can be traced back to poor quality measurements whose construct are not appropriately operationalized.

Validated Methodological Operationism

This represents measured processes whose operations are clear and whose quality and validity has been adequately evaluated.


Feest references the gap between qualitative and quantitative research as being about operationism.  I believe this is incorrect.  Operationism is about construct validity (unified theory).  Criticism of qualitative research is usually about research validity (a different validity) and the value of different research purposes.

Collaborative Inquiry: Collaboration as a Method to Increase Research Capabilities

Catherine Lombardozzi has blogged about Collaborative Inquiry, a topic I’ve been skirting for  a while with terms like: learning networks, un-conference, collective intelligence, distributed decision making, distributed cognition or connectivism.  I think this basic idea is a natural part of web 2.0 thinking that is collaborative in its core nature. I also think it is a way to tap into the knowledge flows associated with Hagel’s Power of Pull thinking and is also just a good way to address knowledge areas that are in flux as opposed to stable and well established.  Less and less of our practices are stable today and this is a natural way to gain knowledge and direction.

To extend this idea I’ve been thinking about why you would want to choose a collaborative research structure over a more traditional set up, and I’ve put my ideas into this concept map.  It’s not a finished product; just beginning thoughts.

A Comparison of Individual and Collaborative Research

Individual Collaborative Research Comparision - Why would you choose collaborative research methodology

New Forms for Pedagogy: Another Take

Developing Creativity through Lifewide Education by Norman Jackson considers the inadequacies of the structure of higher education and claims that;

(E)duration that is dominated by the mastery of content and cognitive performance in abstract situations, (it) is not enough. . . . (Quoting Douglas Thomas and John Seeley Brown)  “What is required to succeed in education is a theory that is responsive to the context of constant flux, while at the same time is grounded in a theory of learning”.

And it’s not just educational practice.   The problems also extend to research based knowledge generation.

Paradoxically, the core enterprise of research – the production of new knowledge – is generally seen as an objective systematic activity rather than a creative activity that combines, in imaginative ways, objective and more intuitive forms of thinking.

This critique of knowledge generation also fits with the ides of my last post inspired by Jay Cross.   If your context is rather stable, than knowledge generation that emphasizes objectivity and systematicity will work relatively well.  But if one’s situation trends towards a contextual flux in a complex multi-demensional variable field, then systematic objectivity may be useful for verification of experimental data, but not for generation hypotheses and theories, the things that lead and guide inquiry.  Current methodological thought treats the creative generation of hypotheses and theories rather cavalierly considering their central place in inquiry.

Norman list 8 propositions for a new curricular approach.

In order to facilitate students’ creative development for the real world we must create a curriculum that –

  • Proposition 1 : gives them the freedom and empowers them to make choices so that they can find deeply satisfying and personally challenging situations that inspire and require their creativity. A curriculum should nurture their spirit: their will to be and become a better more developed person and create new value in the world around them
  • Proposition 2: enables them to experience and appreciate knowledge and knowing in all its forms. And enables them to experience and appreciate themselves as knower, maker, player, narrator and enquirer
  • Proposition 3 : enables them to appreciate the significance of being able to deal with situations and to see situations as the fundamental opportunity for being creative. They need to be empowered to create new situations individually and with others by connecting people and transferring, adapting and integrating ideas, resources and opportunities, in an imaginative, willful and productive way, to solve problems and create new value.
  • Proposition 4: prepares them for and gives them experiences of adventuring in uncertain and unfamiliar situations, through which they encounter and learn to deal with situations that do not always result in success but which do not penalize ‘mistakes’ or failure to reach a successful outcome
  • Proposition 5 : enables them to develop and practice the repertoire of communication and literacy skills they need to be effective in a modern world
  • Proposition 6: encourages participants to behave ethically and with social responsibility promoting creativity as means of making a difference to people or adding value to the world
  • Proposition 7: engenders a commitment to personal and cooperative learning and the continuing development of capability for the demands of any situation and the more strategic development of capability for future learning
  • Proposition 8: helps them develop and explain their understandings of what creativity means in the situations in which they participate or create, and values and recognizes their awareness and application

More broadly, how do we do this?  I’ll fall back on Hagel, Brown and Divison’s Power of Pull framework.

  1. Tap into knowledge flows, especially through Web 2.o technologies such as Personal Learning Environments or community wide collaborative research projects.
  2. Find a trustful, creative and knowledge flow filled environments (both virtual and physical).  Places where serendipity is more likely to strike.
  3. Rather than scalable efficiency,  strategize for scalable connectivity, scalable learning, and new possibilities for performance.
  4. Tap into people’s passion.  You could say, manage by helping people find inspiration.

People might say; “this does not represent the real world”.  I would counter that their real world was the 20th Century.  That world is now fading, and as they say, the new world is here, it’s just not evenly distributed,

One Description of Science and the Basis for an Argumentative Approach to Validity Issues

I came across an interesting metaphor for science (and structural ways of understanding in general) in the Partially Examined Podcast Episode #8.   Here is my take on the metaphor.

Imagine the world as a white canvas with black spots on it.  Over that, lay a mesh made of squares and describe what shows through the mesh.  We are describing the world, but as it shows through the mesh.  Change the mesh in size or in shape and we have a new description of the world.

Now, these descriptions are useful and allow us to do things, but they are not truth, they are description.  They may be highly accurate in their descriptions of an actual world, but they are still descriptions.  It’s how science functions and is how science progresses and changes.  It also is why I advocate an argumentative approach to validity in the use of scientific structures like assessment or the use of evidence.  Old forms of validity (dependent on criterion validity) and much of the current discussion of evidence-based approaches is about the accuracy in certain forms of description.  But we must also allow for discussions of the mesh (to return to the metaphor).  As in construct validity, any discussion of how the world is must also include a discussion of how the mesh interact with the world to create the description.

In addition to methods like random controlled trials (RCTs), there is also a need for research into how we understand and rethink the assumptions and things that are sometimes unexamined in research.  RCTs are very good at helping us do things with very accurate descriptions (like describe linear causal processes).  We also need research that uses other meshes that will allow us to understand in new ways and facilitating our ability to do new and different things; to make progress.

Channeling Pandora: Ideas from a 2007 Interview with Management Professor Bill Starbuck

Reading through documents from the Stanford Evidence-based Management Blog Site, I came across an interesting article (Un)Learning and (Mis)Education Through the Eyes of Bill Starbuck: An Interview with Pandora’s Playmate (Michael Barnett,(2007) Academy of Management Learning and Education, 6, 1, 114-127).

Starbuck seems to be concerned with two things: (1) methodological problems in research and (2) un-learning or fossilized behavior in organizations.
On methodology:  You can’t just apply standard statistical protocols in research and expect to get good answers.  You must painstakingly build a reasonable methodology fitting your methods to contexts and tasks, much like you are fitting together the pieces of a puzzle.  In a personal example: I consulted practically every text book I had when developing methods for my dissertation, but the most common were introductory statistical texts.  I kept asking myself: what am I doing, what are the core statistical concepts I need to do this, how can I shape my methods to fit my tasks to these core concepts.  Almost all advanced statistical techniques are an extrapolation of the concepts found in introductory statistics and your can’t really understand how to use these advanced procedures until you understand their core and how they fit your methodological circumstances.  As Starbuck points out, the concept of statistical significance is the beginning of results reporting, not the end.  You must go on to build a case for substantive importance.  He points out that mistakes are common in reporting effect sizes.  I believe that this often happens because people simply apply a statistical protocol instead of understanding what their statistic are doing.

A favorite issue of mine (that Starbuck implies, but does not address directly) is the lack of a theoretical framework.  Without theory, you are flying empirically blind.  Think of the four blind men empirically describing an elephant by holding a trunk, leg, body and tail.  Vision (or collaboration) would have allowed the men to “see” how their individual observations fit together as a hole.  You must begin with the empirical, but reality is always larger than your empirical study and you need the “vision” of a theoretical framework to understand the larger picture and how things fit together.  Theory is thus an important part of your overall methodological tact.

On (Un)Learning: Starbuck discusses the need to unlearn or to change organizational processes in response to the changing environment.  It is a problem where past success cloud your vision obscuring the fact the what worked before is no longer working.  The problem is not that people can’t think of what to do to be successful, it’s that they already know what to do and their belief keeps them from seeing that there even is a problem or seeing the correct problem.  Starbuck talks about problem solving in courses he taught.  He found that people often found that the problem they needed to solve was not the problem they initially had in mind.  Their biggest need was to change beliefs and perspectives.

The psychologist Vygotsky spoke of something very similar as fossillized behavior.  As someone is presented with a unique problem, they must work out the process solutions to the problem externally.  Later the process is internalized to some extent and become somewhat automated, requiring much less cognitive load.  After more time this can become fossilized, that is, behavior that is no longer tied to a process or reason, but continues as a sort of tradition or habit.  This would apply at the organizational level as well as the individual posychological level.  I would like to investigate the concept of organizational resilience as a possible response to fossilized organizational behavior as well as a way of responding to extreem events.  This would  emphasize an ability to change in response to environmental demands.  Starbuck thinks that constant change is too disruptive to organizations, but I believe that there may be a combination of processes, capabilities and diversity that enable organizations to sense and respond, not necessarily constantly, but reasonably on an ongoing basis as well as when the enevitable  black swan happens.

Bridging Science and Practice via a Science of the Artificial

Interesting article,

Bridging Practice and Theory: A Design Science Approach by Jan Holmstrom, Mikko Ketokivi, & Ari-Pekka Hameri (Decision Sciences, 40 (1), 65-87.)

It’s really less about bridging theory and practice than it is about developing a science of the artificial a la Herb Simon.  My take on the topic – focus less on discovering the world (predicting) and more or creating the world by making artifacts the central focus or unit of analysis of research.  Make problem framing leading to artifact fabrication a part of research.  You can think of this as a science of practice.  Think less of theories about how the mind works than of artifacts to improve practice.  Theories are still important, for instance, you can’t judge practice validity without theory; but think of artifacts as a bridge between theory and practice.  Hmm, maybe it is about bridging.

But it’s also about engineering.  Think about the prevalence of artifacts (broadly speaking) in our world, and the way that artifact can shape cognition and behavior, other disciplines beging to share concerns ussually associated with engineering.  Education likely has more in common with engineering than it does with traditional psychology.

A side bar here about validity.  Research, including a Science of the Artificial, is about answering questions.  The methodology you choose must match the research question.  In this conception of research you will likely ask a series of different questions, some of which will only become apparent as you go through a process.  You still have to change your methodology as your questions require.  This make bias more possible, indeed bias and objectivity plays slightly different roles in this artificial science when you’re purposely creating rather than discovering.

The Integration of Design and World: More on Design Thinking.

This post responses to Anne Burdick’s invitation concerning the presentation: Design without Designers (found in the comments of my last post).  I will address the question, why would educational theory build on design concepts or how do I see the relation between education and design? I will look at three areas:

  • Erasing the distinction between art and science
  • Artifactual cultural aspects of psychology
  • The trans-disciplinary nature of ideas

Erasing the distinction between art and science

I see general changes in the practice of science along the following lines:

  • The critique of positivism (for promising more than methodology could ever deliver)
  • The critique of postmodernism (for fetishizing certainty; i.e. If positivism fails than scientific judgement cannot be trusted at all.) and
  • More acceptance for addressing real world problems (where problems tend to be interdisciplinary and often involve mixed methods).

The result is that many of the walls and silos of science have been reduced including the distinction between art and science.  In example, I often refer to judgements based on validity.  Although validity uses rational and empirical tools, building a body of evidence and achieving a combined judgement is more like telling a story.

Artifactual cultural aspects of psychology

The work of (or derived from) Vygotsky is popular in psychology and education.  It has also proved consistent with, and complimentary to the recent findings of the “brain sciences”.  While there are genetic and hardwired aspects of psychology, the structure of our minds can be said to reflect, to a great extent, the structure of the social and artifactual world that we live in.  The design of the world is more than just a decorative environment to an autonomous mind, it has an impact on who we are in both development and in how we interact with it in our ordinary lives.

Our delineation of the subject matter of psychology has to take account of discourses, significations, subjectivities, and positionings, for it is in these that psychological phenomena actually exist. (Harré and Gillet, 1994, The Discursive Mind and the Virtual Faculty)

The trans-disciplinary nature of ideas

Ideas never seem to respect the traditional academic disciplinary structure the way that methods and theories did during most of the 20th Century.  In the mid-90s a graduate school mentor pointed out that you could read many books at that time and have no clue to the discipline of the author without reading the back cover.  Psychologist, educators, literary critics, philosophers, sociologists and yes, designers, they all often seem to be speaking in the same language about the same type of things.

In Conclusion

  • The distinction between art and science is dissolving.  Method is important, but it does not rule.  Achieving a scientific break-through is analogous to creating a work of art (even though it still uses rational and empirical tools).
  • The design of our world is not just decoration, it reflects who we are and who we are reflects the design of the world.
  • Tools (artifacts, concepts, theories, etc. . .) are needed to act on the world.  Where these tools come from is less important than our ability to make use of them.

So in the above ways, design and design thinking is everywhere.  I do think designers should be more present in my own thinking as both a technical adjunct and as a foundation of both my thought and of the academic curriculum?  Yes, I do!  What do you think from a designer’s perspective?  How does the thinking of designers and current design curriculum fit into the above ideas?

Future EBMgmt Research Ideas

  1. I will need to think more on an evidence evaluation framework and how Rousseau’s model might be enhanced by Messick’s model of validity as discusses in my last post.
  2. As Messick said that validity is about test use not tests in themselves, so evidence-based practice is about how the evidence is used in practice, not about the evidence itself.  This needs to be spelled out.
  3. The practice research gap – Research validity generally become greater the more context can be controlled and parsed out of studies.  In evidence-based practice evidence must be related to context to be valid.  The more confident you are of research results, the less confidence you are that it will relate to the constellation of factors seen in contexts.  I don’t know how you can get beyond this without some applied research that puts research syntheses to the test.
  4. Practice is most often cross or interdisciplinary.  This impacts the last point, but it also means that each practice relates to many potential disciplines.  Accumulating the vast amounts of data will be next to impossible in a practical manor.  We need a technological solution through some sort of Web 3.0 or metadata solution as well as a technological way to compile data.