Some recent involvement in LinkedIn conversations has led me to delve more into Paul Meehl’s work in Philosophy of Science or what he referred to as scientific metatheory. As the book A Paul Meehl Reader notes, Paul’s essays were painstakingly written and most readers do not read his work so much as they mine his work for insights over many years; so I suspects this will be a long term project.
Here is the first nugget: progress in the soft sciences is difficult and painstaking and much of the existing research work mat be flawed and found wanting. Here are some reasons:
- Theory testing often involves derived auxiliary theories which, if not highly supported themselves, will add unknown noise into the data. Often these theories are also not spelled out or understood.
- Experimenter error, experimenter bias, or editorial bias is present more often than is generally acknowledged or even known or considered.
- Inadequate statistical power. In general, much more power is needed. Meehl thinks that we should often seek statistical power in the .9 range in order to overcome unknown noise (error) in the data.
- Seriously accounting for the crud factor (the possible effect of ambient correlational noise in the data).
- Unconsidered validity concerns. The foundation of science is measurement, but often the validity of measurement tools are not considered seriously. Experiments are often measuring things is new ways even if they are using well studied instrument and this requires analysis for validity.
What this means is that more methodological care is needed such as:
- Seeking predicted point values that are stronger in terms of falsification and lend more verisimilitude than the often weak corroboration that come from non-null significance testing.
- More power (i.e. .9) in hypothesis testing to protect against weak auxiliaries, unknown bias and general crud.
- Understanding the difference between statical significance and evidentiary support. Observations are evaluated in terms of statistical hypotheses and are a statistician’s concerns about the probability of the observations. But theories are evaluated by the accumulation of logical facts. These are not evaluated in terms of probabilities, but in terms of verisimilitude.
- Science should seek more complete conceptual understanding of the phenomena under study.
I believe this last point is similar to Wittgenstin’s concerns that in science problem and method often pass one another by without interacting. I think this concern is also similar to verisimilitude in theory. Verisimilitude maybe considered a fussy interpretive concept, but the problems uncovered by the Reproducibility Project show that hard sciences are not as interpretive free as is often supposed. I’m also coming to the conclusion that it is in Meehl (and the like minded Messick) that traditional empirical science and pragmatism can be brought together. It is the idea that a social constructivist approach must account for both the successes and the failures of empirical science if it is to move forward productively. Meehl and Messick were not pragmatists, but I am saying that in dealing with the problems thay saw in empirical science, a critical pragmatic approach can be envisioned. As Meehl along with Wittgenstein, Popper and maybe Lakotos are some of the best critics within the empirical sciences and building from their critiques seems like an interesting place to explore.