Making Inferences about the Use of Artifacts in Practices

This post is to think through what I brought up last post, applying the concept of validity to practice.

I remember hearing in school validity asked the question: “does the test measure what it’s intended to measure”?  The problem with this type of approach is that it leads you in circles, both practically and epistemologically.  Messick changed this to a question that was quite literally more consequential.  Is there evidence that the use of the test brings or contributes to the results you intended?

If you view a test or assessment as an artifact imbedded in a practice, you could apply the same type of logic to any artifact that play an active role in that practice.  In artifact creation like in Holmstrom’s article, the logic of validity could be applied as a guide.  Although there would still be an artistic element it would not be random or unsupported.

There is an evidentiary aspect to this way of considering artifacts.  Validity is all about evidence. Theoretical evidence, process evidence, empirical evidence, consequential evidence, generalizability evidence; this is all about validity.  In fact, validity theory can be a way of accounting for evidence from all type of methodology.  Random Controlled Trails are still the best way of judging validity for certain types of research questions, but any type of method can contribute to evidence.

Summary: Validity is a well-developed body of though that can be applied to making inferential judgements about evidence supporting the use of assessment or other active artifacts in the context of a specific practice.

Problem-Based Artifact Creation Process

This post extends my thinking about the last post. The concept of validity is as applicable to practices as it is to tests. When I develop an assessments and protocol, I think about validity from the get go, not just after the fact. Similarly, when designing a process or practice, it is good to add a validity perspective. That is what I’ve done with Holmstrom’s problem-based artifact creation process; add where content, theory, and data should be considered. Here is a prototype process map:Design Sci Map