Tag Archives: Measurement Validity

A New Form for Validity

Posted on by
Reply

Thinking about new projects.  Here are the general contures of a new way of looking at validity.

  1. There have been criticism of Samuel Messick unified view of construct validity and Kanes Argument based approached.  I have yet to accept any logical argument made against either framework, yet I am sympathetic when it is said that these frameworks are not practical administratively.  
  2. Consider an argument made by the philosopher Karl Popper.  Popper makes a distinction between justification and criticism on the way to his famous idea of fallisficationism.  Just like one cannot claim that one’s theory is true through experimentation (you can only be sure of your results if they are false), so too it is precarious to justify one’s beliefs, but easy to demonstrate if their false.  Justification can be seen as a next to impossible task, but criticism is more likely to be seen as true.  If we respond to criticism with a desire to improve and adjust our beliefs than our beliefs will approach a closer version of what you might call truth.  So, the best way to justify assessment validity is by being open to criticism; always seeking to improve through critical reasoning.
  3. This does not nullify Messick’s framework (Messick, 1995), but it shifts it from justification to a framework for critique and critical thinking.  Messick’s framework moves from a hopelessly difficult attempt at justification and becomes a critical framework for knowledge transparency.  Recent developments in philosophy  have demonstrated the contingent nature and how the shape of knowledge is shaped by the form of its production.  Messick’s transparent critical framework for the production of assessment knowledge is the best way to see the underlying contingencies
  4. Kane’s framing of  validity as an argument is more suited to a critical approach than a justificationist approach.   The very nature of argument sets up a 2 sided dialogue.  Every argument presupposes a dialogic counter argument.  If you enter into an argument you must be willing to entertain and engage with critical position.  Kane’s framework is more suited to respond to critical than to depend on justification.

Managing Relationships, Supporting Performance

Posted on by
Reply
Interesting NY Times opinion piece today that extends my recent posts on validity and performance management.  It is entitled: Why Your Boss Is Wrong About You, By Samuel Culbert.  In that article he states:
In my years studying (performance) reviews, I’ve learned that they are subjective evaluations that measure how “comfortable” a boss is with an employee, not how much an employee contributes to overall results.
Samuel Culbert in this statement leads us to believe the problem with performance reviews is their subjective nature.  From a measurement perspective I believe this is incorrectly stated.  The problem is one of validity, that performance measurements typically measure the bosses level of  comfort with an employee, not their performance.  Greater objectivity will not help us if comfort is still the construct being measured.  Instead we must look at validity.
Culbert proposes a performance preview process as an alternative.
It’s something I call the performance preview. Instead of top-down reviews, both boss and subordinate are held responsible for setting goals and achieving results. . . . bosses are taught how to truly manage, and learn that it’s in their interest to listen to their subordinates to get the results . . . “Tell me your problems as they happen; we’re in it together and it’s my job to ensure results.” . . . . It encouraged supervisors to act as coaches and mentors.  . . . But understand that the performance review makes it nearly impossible to have the kind of trusting relationships in the workplace that make improvement possible.
This preview process may be a good idea in and of itself, but it does not logically get at the root problem.  Measurement within this new process can have just as many validity problems as the old process.  This is why validity is important.
Two additional things I’m thinking:
Culbert doesn’t quite get there, but I sense he is looking at something like Action Analytics, measurement tied to real-time feedback that can support performance while performance is still in formation.  Instead of measurement in the service of performance review it is measurement in the service of performance support.
The second thought is this.  The central point in Culbert’s process is trust between an employee and a boss, that is, a management relationship.  This is the central construct in successful management and it may be important to measure.
This leads me to an interesting final conclusion and maybe a management axiom:
In managing people, we management relationships, but support their performance.