Evidence-based Testing (Assessment)

I confess; I love the 35,000 foot view.  An article by an old pro, who gives us their overview of the future of their field.  This is Howard Wainer’s, 14 Conversations About Three Things.  His intended audience are researchers of the 21st Century.  His three things are what skills will they need (see #1 below), what problems are worth investigating (See #2) and what topics are not (See #3).

What jumped out at me was the topic, Evidence-based Testing (EBTD) and the premiss behind his recommendation.  (I have more study to do, but EBTD seems to be testing designed with validity in mind.)  His premiss is that statistical analysis has been very well researched and we can get more bang for the buck by focusing on improvements in test design.  We have done a better job improving data analysis than we have in data collection.  I think this premiss holds true across society (education, business, science etc. . . ).  We are generally better at analysis than we are at data collection.  In many cases it is garbage in – garbage out.  It’s not that analysis is unimportant, it’s just that the easiest way to improve analysis is in improving the data / information that forms the basis of analysis.  How do we do this?  By designing measures with greater validity.

  1. Six skills needed by 21st Century Researchers: Bayesian Methods, (Modeling) Causal Inference, (Dealing with) Missing Data, (Graphic representation for) Picturing Data, Writing Clear Prose, A Deep Understanding of Type I and Type II Errors.
  2. Important topic for 21st Century investigation: Evidence-based Test Design, Value Added (statistical) Models, New Kinds of Data (mostly made possible by computer networks)The integration of Computerized Adaptive Testing, Diagnostic Testing and Individualized Instruction.
  3. Topic that can be given a rest: Differential Item Functioning, The Rasch Model, Factor Analysis / Path Models, New Measures of Reliability.


      Wainer, H, (2010). 14 Conversations About Three Things, Journal of Educational and Behavioral Statistics, 35 (#1) 5-25.