Achieving Clarity Through Assessment

(P)ragmatism is that method of reflexion which is guided by constantly holding in view its purpose and the purpose of the ideas it analyzes . . . a method of reflexion having for its purpose to render ideas clear. Wikipedia the Pragmatic Maxim CS Peirce CS Peirce

Some measurement specialists argue that adding value implications and social consequences to the validity framework unduly burdens the concept. However, it is simply not the case that values are being added to validity in this unified view. Rather, values are intrinsic to the meaning and outcomes of the testing and have always been . . . This makes explicit what has been latent all along, namely, that validity judgements are value judgments. (emphasis in original) (Messick, 1995) Samuel Messick

I’ve found Agile interesting because it puts purpose and values front and center in practice; a natural example of a paradigm change toward pragmatism in action. As Quine argued in Two Dogmas of Empiricism, the empirical world only impinges on our experiences along the edges, but to accommodate this edge experience, we often must re-evaluate our belief systems down to the core. It is easy to ignore these edge experiences and one reason why a paradigm change of core beliefs can be hard. I think Agile comes from this core reevaluation, but I don’t think it would be well served if the measurement paradign didn’t match the nature of the Agile Paradigm.

Assessment can be a method of reflection that constantly holds its purpose and values in view and I believe Pragmatic Assessment can clarify Agile for practice and achieve the purpose of expectations management. One way it does this is by creating an operational definition of Agile that looks not only at the score, but observes the process of how the score and how Agile is achieved. Since Agile can be implemented over time, a Maturity style assessment could help, especially if it spells out how the underlying paradigm changes through the Maturity Process. This is how assessment can clarify Agility, achieve the expectations management that you intend and spell out much of the metaphysics of Agile.

Assessments should not be written in stone, but should change as guided by a scientific inquiry process with a focus on consequences in the use of the assessment. But the purpose of the assessment remains central. François Chollet’s recent piece, The impossibility of intelligence explosion, is an example of what can happen if purpose is ignored. Alfred Benit first developed his IQ test to identify French students for an early version of special education. Later, in the rapid mobilization before WWI, the US Army adapted his ideas to help with their forced choice to identify new recruits for leadership positions. But it is something completely different to use such a test to define a construct of intelligence as we naturally understand it. It leads not only to inapproariate uses by people like Charles Murray, but also confounds our understanding of concepts like Artificial Intelligence. An operational definition of intelligence as scored by an IQ test will expose how different this psychometric definition is from our everyday understanding of being smart. Purpose, sometimes, is everything and appropriately operationalizing a fuzzy concept can really help to clarify our ideas.

Note: The above  post was part of a conversation on medium with Stefan Wolpers about assessment in Agile frameworks, but I believe it represents assessment as a part of a process of pragmatic reflection that applies to most if not all educational measurement. Just as we wrongly think of assessment as measuring chunks of knowledge rather than aspects of performance, educational measurement measures cognitive processes.

Steven Pinker vs Leon Wieseltier: Beyond the Polemic in Science vs Humanities

Let me tell a story. The 18th and 19th Century was a time of the flowering of the enlightenment and intellectual achievement. But it was also a flowering of the idealism that lie (even if dormant) in the work of philosophers like Rene Descartes and Immanuel Kant. Idealism was an impediment to the rising desire to exert scientific control in social areas such as management and education. By the 20th Century, people like George Edward Moore (1873–1958), Bertrand Russell (1872–1970) and Rudolf Carnap (1891–1970) tired of what they viewed as rampant intellectual muddle-headedness and struck out as empirical positivists to move beyond this state of affairs. As noted by Guyer and Horstmann, Moore and Russell’s critique was so devastating that it is not possible to serious entertain an idealist position in the English speaking world today. And that is inspite of the fact that their banishment of idealism largely was unsuccessful.

In a very similar fashion Zat Rana recently cautioned us sgainst thinking too deeply in a post that echos the idealist concerns of Moore and Russell.

The trap that most of us fall into is one in which we impose too much of the invisible world onto the real, tangible world. The result of this is that we add needless context and depth that detracts from the clarity provided by the senses we use to paint the part of reality in which we commonly live.(Emphasis added)

But relying exclusively on our senses has a subtle way of re-introducing idealism back into the equation, but an idealism that can easily be ignored. Science and data are our best tools, but they always operate as a form of simplification and must be situated in a larger intellectual field to find its natural validity. WVO Quine famously critiqued this form of reductionism:

The dogma of reductionism survives in the supposition that each statement, taken in isolation from its fellows, can admit of confirmation or infirmation at all. My countersuggestion, . . . statements about the external world face the tribunal of sense experience not individually but only as a corporate body. . . . (T)otal science is like a field of force whose boundary conditions are experience. A conflict with experience at the periphery occasions readjustments in the interior of the field. . . . Re-evaluation of some statements entails re-evaluation of others, because of their logical interconnections — the logical laws being in turn simply certain further statements of the system, . . . But the total field is so undetermined by its boundary conditions, experience, that there is much latitude of choice as to what statements to re-evaluate in the light of any single contrary experience . . . Each man is given a scientific heritage plus a continuing barrage of sensory stimulation; and the considerations which guide him in warping his scientific heritage to fit his continuing sensory promptings are, where rational, pragmatic. Quine’s Two Dogma of Empiricism

In fact, we might suggest in closing, the main alternative to what is essentially the epistemological idealism of a great deal of twentieth-century philosophy has not been any straightforward form of realism, but rather what might be called the “life philosophy” . . . the lived experience of “being-in-the-world”, from which both the “subjective” such as sense-data and the “objective” such as objects theorized by science are abstractions or constructions made for specific purposes. Guyer and Horstmann

What these authors address is more than reductionism but also an impulse for an anti-metaphysical approach and, as noted by CS Peirce, show me someone who claims to be beyond metaphysics, to rely simply on sense data, and I’ll show you someone whose thoughts are overflowing with metaphysical assumptions.

But we are left with the question, how do we respond to a confusing world in spite of Zat’s concerns, which are legitimate? That, I believe is the role of education; providing people with broad tools of thinking. To be able to use analytic frameworks and understand their historical development in both the sciences and the humanities and how to combine these frameworks in practices that reflect good thinking.

This brings me back to Pinker and Wieseltier. When an educated person brings good thinking to a problem, there is no bright line that separates the humanities from the sciences. To say otherwise is to return to the failed approach of the early 20th Century empiricists. The holistic approach is the essence of the whole life philosophy of pragmatism. Metaphysics, the laying out of the assumptions behind an analysis, requires both humanistic and scientific frameworks. I mostly must disagree with Pinker, not because I don’t favor science to understand the world, but because Pinker’s position implies an underlying reductionism, a devaluing of the metaphysical. In the same way that 20th Century empiricism failed to rid themselves of idealism, Pinker is not advocating for being scientific, but simply being under-educated and unthoughtful in how he analyses these science / humanity wars. In short, science needs more than methodological rigor, it also needs intellectual rigor.