Measurement Literacy: Without Meaning, Measures Indeed Can Get Out of Hand

We say that someone is literate if they can read for meaning or if they can calculate with numbers.  There is also a need for measurement literacy which is when someone can say what numbers mean when they are obtained in measures.  Although it is a bit wonkish, it’s still important to remember that measures measure constructs not the thing being measured itself.  Constructs are concepts that are thought (theoretically) to be a property of things, but they are not the things.  To understand the meaning of number obtained from measurement, it is necessary to understand the construct.  Harold Jarche recently posted 2 quotes that expressed negative opinions on measurement processes.  I believe these critiques are ill founded for two reasons:

  1. Poorly designed measures should not be used to condemn measurement practice and
  2. Eliminating measures often lead to politics, gut instincts and other poorly founded basis for decision-making.

In this post I would like to go deeper on this subject and show how the problem can be explained as a problem with measurement literacy.

First, Jarche quotes from Charles Green at The Trusted Advisor:

If you can measure it, you can manage it; if you can’t measure it, you can’t manage it; if you can’t manage it, it’s because you can’t measure it; and if you managed it, it’s because you measured it.

Every one of those statements is wrong. But business eats it up. And it’s easy to see why. …  The ubiquity of measurement inexorably leads people to mistake the measures themselves for the things they were intended to measure (Emphasis added).

The second quote is from Dave Snowden:

We face the challenge of meeting increasing legitimate demands for social services with decreasing real time resources. That brings with it questions of rationing, control and measurement which, however well intentioned, conspire to make the problem worse rather than better. For me this all comes back to one fundamental error, namely we are treating all the processes of government as if they were tasks for engineers rather than a complex problem of co-evolution at multiple levels (individuals, the community, the environment etc.).

I posted this response on Harold Jarche’s blog:

I agree that there are many instances of problems resulting from measures that are based on little more than common sense or tradition, but it is not helpful to base decisions on gut instincts or politics. I believe the need is to increase people’s understanding of good measurement practices and how to develop a deeper understanding of what their measurements really mean. Everyone should know if their measures are valid. In turn, that means being able to say what your measures mean, how they are relevant to practice, and how they are helping to improve practice. It’s not just for big wigs either. Front line employees need to understand how to use measurement to guide practice.

Going further, Charles Green, also said this in his post;

There’s nothing inherently wrong with measuring. Or transactions. Or markets. They’re fine things.  But undiluted and without moderating influences, they become not just a bad deal; they can be a prime cause of ruining the whole deal.

Green is not clear here, to the extent that he doesn’t explain moderating influence.  For measurement, I believe this moderation influence could be meaning or construct supported meaning.  First, Measurement can easily get out of hand because numbers can do two things.  Through constructs they can have meaning, like words, but they can also be calculated.  Being able to calculate with numbers is not the same as being able to say what they mean.  Though people often conflate the two, they are not the same.  Calculation can result in the potential for meaning, like when we calculate a Pearson’s correlation.  But, understanding meaning requires a deeper understanding of how measures were obtained, what is the theoretical construct basis for the measures as well as consequential and other basis for the validity of the measures.

Many people have a good grasp of statistics and how to calculate, but they have less knowledge about measures, validity, designing measures and measurement meaning.  Mistaking measures for the things they represent is a problem of meaning.  Having measures confound complex evolutionary problems is rooted in miss-understanding measurement meaning.  I believe many people would like to give up measurement, but that would not ultimately result in better consequences.  What is needed is better education directed toward literacy in measurement meaning.

Agile or CMMI: the Differential is Knowledge.

How you frame a topic (question, problem, etc. . . ) is critical in how you come to understand it.  It’s at the heart of why perspective is so important.  A different theory, metaphor or emotional tact can make a vast difference in meaning.

I have been reading about agile methods and how it is an unplanned methods that is contrasted with planned methods, especially with CMMI (Capability Maturity Model Integration).  Although planning is the most salient difference in features, knowledge is the key difference here for analysis; what you know or are capable of knowing as opposed to what you don’t know and are incapable of knowing. CMMI is a framework for managing what you know and can include standardized processes, evidence-based practices or processes and practices that can be standardized.  Many processes or practices are repeatable and can be standardized and managed through a CMMI framework.  But, we also know that science (the basis for standardization) makes knowledge claims in narrow and very specific ways.  This means that there are many aspects to practice that are unique, context bound, not repeatable and not standardizable.  These aspects of practice are best approached through Agile methodology.  The decision to use agile or CMMI methodology should be base on what you are able to know.

I believe the time is approaching that will require more agility, but this will not preclude an expansion of CMMI methods.  Agile methodology needs to incorporate CMMI methodology according to what we are able to know, while maintaining agility for what we are unable to know.  Some project may call primarily for agile methods, some for CMMI methods and some will call for mixed methods.