What to Do about Testing: A Response to Audrey Waters

Audrey Waters posted about John Oliver’s takedown of testing and Pearson Ed.  She asks:

How do we seize the opportunity of all this media attention to the problems with standardized testing to do more than talk about testing?  . . . Can we articulate (a better alternative) now so that Pearson and other testing companies don’t replace the old model with simply a re-branded, repackaged one?

Samuel Messick was a Vice President and Distinguished Research Scientist at Educational Testing Services (ETS).  His was an authoritative voice on test validity advocating for restraint in the use of test scores, better and more in-depth interpretations of test score. the collection of multiple sources of information for making important decisions and for consideration of the consequences of test use.  I believe that much of his legacy has been ignored, co-opted, or argued away (even at ETS I suspect).  I’ll speculate on what would he advocate;

  • using more than one or two sources of information when making complex important decisions,
  • understanding the information in the context of a decision and considering the consequences of your testing practices.
  • I also suspect that I could argue with him for the consideration of the validity of testing practices with how it fit within an overall set of district practices.  (i.e. If a student fails, how do you respond?)

Technically Pearson may not be at fault for it is the district use of tests that is most problematic, but Pearson is at least implicit in not providing better guidance and for developing ways for districts  to collect other sources of information.  Eg. The value added model of teacher assessment needs many more sources of information and in fact does not really provide an assessable model of pedagogy, only largely discredited positivist assertions. The first step is to expose those who advocate positivist models of empiricism for which even analytic philosophers would no longer advocate.

Finally it necessary to look at the overall model of education which is still primarily built of a mechanistic metaphor with the student as a vessel to be filled.  The metaphor should be a biological organism adapting in an environment that is primarily social, networked and interactive.  When Pearson speaks of their “potential game-changer: performance tasks”, they are talking in this direction, but their really co-opting performance tasks within the old metaphor.  They have a long way to go.  We should expunge the mechanistic metaphor from educational leadership and assessment models.

The bottom line for Pearson

You may not be technically wrong in your assessments, but when your the brunt of a comedic takedown, you should really look at the consequences of your products use and attempt to deal with it.

Education Needs Clarity

Beginnings: My Graduate Experience  (The 90s and the oughts)

My PhD was not motivated by a career path, but by my love of learning. Temple U’s Associate Professor Helmut Bartel (a proclaimed social constructionist) was an intellectual guide who helped me to recognize the relevance of social theories to my professional experiences; that is, I was by nature a pragmatist. Helmut left Temple before I could develop a dissertation topic and it was fortuitous because I needed to challenge myself to align my thoughts with new mentors. While trying to form a dissertation topic a professor said offhand, “It sounds to me that your talking about validity.” I read Messick’s chapter titled Validity in Linn’s (ed) Handbook of Educational Measurement. The references and the lineage of his ideas were all different, but the conversations where much the same and they centered around a pragmatic approach. The patrons of validity, Messick, Cronback and Meehl, were very clearly analytic in their thinking, but the logic of pragmatism was already deeply embedded in their thought.

Why Philosophy

My studies were in educational psychology, and I do find many discussion in philosophy to be tedious and boring, so why discuss philosophy.  Because, for everything we say, there are many things that are left unsaid and for everything we do, much of the reasoning is left unsaid and unquestioned.  The philosophy I discuss is about shining a light on practices to see what we are taking for granted and to understand what has been left unsaid. What we need is clarity, and that is precisely the purpose of philosophy in its analytic, neoanalytic and pragmatic forms.

Where is Validity in Educational Practice

How do you address validity questions that appear paradigmatically opposed to traditional empirical scientific practice? I begin with an adaptation of a thought who linage I trace Helmut. A successful paradigm change must account for the current paradigm in both its successes and failures in order to forge a true new order. The dominate and implicit practice paradigms today are still mostly based in a dualist objectivist analytic philosophy. Post-modern / post-structural and Marxis based critiques all excel at accounting for the ideological failures of an analytic approach, but not its successes. They fail to point to a way to move practice forward and seem to be losing steam, even as their critiques of analytic approaches remain valid. I think a better way is to consider pragmatism.
Pragmatism and Analytic Philosophy share a commitment to logic and the science method. What Pragmatism brings is a unity of science, practice and ethics (Boncompagni, 2001). Scientific practices are always situated in the midst of ethical horizons best understood as historicized ideological practices. This also matches my earlier experiences where I was working in disability services. The field was moving on from the least restrictive environment to minority rights and people first language. I thoroughly believe in the practicality of science, but science based practices were slow to adapt and often seemed to be standing in the way of ethically empowering practices. Obsessed with an unsustainable conception of objectivity, many scientists could not see how a lack of ethics impoverished science and made it weaker, not stronger.

Pragmatism to the Analytic and Back

I see the history of Pragmatism beginning with Peirce, James, Dewey and Mead, but it became overshadowed by the analytic approaches of European trained academics, especially those associated with the Vienna Circle. As problems were recognized in Analytic Philosophy there began a slow and constant evolution towards pragmatism. In Analytic Philosophy this included people and their ideas such as Quinn, Kuhn, and Wittgenstein. In educational psychology this included Cronbach, Meehl and Messick. This may not be exactly James’ or Dewey’s Pragmatism, but it’s much closer than the direction sought by the Vienna Circle or BF Skinner and I believe that a movement towards pragmatism continues today.

To understand pragmatic social science, let’s begin with Joseph Margolis’ claim: “language and what language uniquely makes possible in the way of the evolving powers of the human mind are emergent, artifactual, hybrid precipitates of the joint processes of biological and cultural evolution;” I see this as something like taking up the naturalism and social behaviorism of Dewey and Mead.  This approach may no longer provide  a foundation for infallible truths, but there is still room for an ethical, objective and empirically warranted practice. This social behavioral and empirical science should be distinguished from Skinner’s radical behaviorism in the same way logical positivism is distinguished from current analytic / pragmatic  approaches.   The knowledge radical behaviorism engenders, fails to adequately recognize the full nature of language and the social world it makes possible.  As a result radical behaviorism leaves knowledge as flat and shallow and more often results in situations (as Wittgenstein noted) where the educational problem and the method pass one another by without interacting. To be valid, empirical methods must reflect the contextualized, artifactual and ethical demands of the problems within a philosophically Darwinian framework of an organism’s adaptation to the social and physical environment. Adaptation is very personal and includes concepts like social poetics.  That is, I accept analytic tools and methods, but recognize them only within social ethical fields that are interpretive as above.  Just as analytic philosophy has moved back toward Pierce, James, Dewey and Mead, radical behaviorism can only be relevant by moving toward Vygotsky, Dewey, Wittgenstein and social poetics.

References

Margolis, Joseph (2012-10-17). Pragmatism Ascendent: A Yard of Narrative, a Touch of Prophecy (p. 133). Stanford University Press. Kindle Edition.

Boncompagni, A (2011). Book Review on New Perspectives on Pragmatism and Analytic Philosophy, EUROPEAN JOURNAL OF PRAGMATISM AND AMERICAN PHILOSOPHY, III, 2, 290-299. http://lnx.journalofpragmatism.eu/wp-content/uploads/2012/01/calcaterra-new-perspective.pdf

Garrison, J (1995). Deweyan Pragmatism and the Epistemology of Contemporary Social Constructivism, American Educational Research Journal, 32, 716-740.

Understanding Paul Meehl’s Metascience: Pragmatism for Empirical Social Science

Some recent involvement in LinkedIn conversations has led me to delve more into Paul Meehl’s work in Philosophy of Science or what he referred to as scientific metatheory.  As the book A Paul Meehl Reader notes, Paul’s essays were painstakingly written and most readers do not read his work so much as they mine his work for insights over many years; so I suspects this will be a long term project.

Here is the first nugget: progress in the soft sciences is difficult and painstaking and much of the existing research work mat be flawed and found wanting. Here are some reasons:

  1. Theory testing often involves derived auxiliary theories which, if not highly supported themselves, will add unknown noise into the data.  Often these theories are also not spelled out or understood.
  2. Experimenter error, experimenter bias, or editorial bias is present more often than is generally acknowledged or even known or considered.
  3. Inadequate statistical power.  In general, much more power is needed.  Meehl thinks that we should often seek statistical power in the .9 range in order to overcome unknown noise (error) in the data.
  4. Seriously accounting for the crud factor (the possible effect of ambient correlational noise in the data).
  5. Unconsidered validity concerns.  The foundation of science is measurement, but often the validity of measurement tools are not considered seriously.  Experiments are often measuring things is new ways even if they are using well studied instrument and this requires analysis for validity.

What this means is that more methodological care is needed such as:

  1. Seeking predicted point values that are stronger in terms of falsification and lend more verisimilitude than the often weak corroboration that come from non-null significance testing.
  2. More power (i.e. .9) in hypothesis testing to protect against weak auxiliaries, unknown bias and general crud.
  3. Understanding the difference between statical significance and evidentiary support.  Observations are evaluated in terms of statistical hypotheses and are a statistician’s concerns about the probability of the observations.  But theories are evaluated by the accumulation of logical facts.  These are not evaluated in terms of probabilities, but in terms of verisimilitude.
  4. Science should seek more complete conceptual understanding of the phenomena under study.

I believe this last point is similar to Wittgenstin’s concerns that in science problem and method often pass one another by without interacting.  I think this concern is also similar to verisimilitude in theory. Verisimilitude maybe considered a fussy interpretive concept, but the problems uncovered by the Reproducibility Project show that hard sciences are not as interpretive free as is often supposed. I’m also coming to the conclusion that it is in Meehl (and the like minded Messick) that traditional empirical science and pragmatism can be brought together.  It is the idea that a social constructivist approach must account for both the successes and the failures of empirical science if it is to move forward productively.  Meehl and Messick were not pragmatists, but I am saying that in dealing with the problems thay saw in empirical science, a critical pragmatic approach can be envisioned.  As Meehl along with Wittgenstein, Popper and maybe Lakotos are some of the best critics within the empirical sciences and building from their critiques seems like an interesting place to explore.

 

 

A Psychological Framework for Studying Social Networks

This is a short review of an article I found interesting.

Westaby, J.D., Pfaff, D.L. & Redding, N., (2014). Psychology and Social Networks: A Dynamic Network Theory Perspective, American Psychologist, 69, 269-284.

The authors note that a psychological perspective on social networks is rarely taken and advocate for more research.  They define 8 (psychological) roles that are thought to be played in these networks regarding goal achievement that they present as a framework to encourage more research.  The roles are:

  1. Goal Striving; directly attempting to achieve a specific goal.
  2. System Supporting; supporting those in goal pursuit.
  3. Goal Preventing; actively working to prevent goal achievement.
  4. Supportive Resisting; supporting goal preventeurs.
  5. System negating; responding with negative affect such as making fun of a person who is goal striving.
  6. System reacting; responding with negative affect toward those resisting goal pursuits.
  7. Interacting; People who can affect goals even though they do not intend to support or resist.
  8. Observing; People who only observe network activity, but nonetheless ca be involved in unintended effects.

This framework could make for an interesting analysis of networks and may have practical relevance for a wide variety of practices.  It may prove to be hard to disentangle the effects wrought by multiple or even conflicting goals in complex environments, or with fluid and changing alliances and more study is needed, however it may be interesting to follow.

The Goal of Edtech: Transparent, Tangible and Trustworthy

This post Lies at the intersection of 3 recent events:

  1. My intuition (and experience) on the need to make tech easy for teachers to adapt into practice
  2. A recent post by Jose Ferreira on Big Data and the Mathematics of Effectiveness and
  3. Comments of charter school educators I heard at the recent NYEdTech Meet up on 4-15

First, I believe that design should be an important factor in the coming ed tech revolution in educational practice.  Tech must be designed in 1 of 2 ways.  Either design it in a way that it can easily be adapted to existing practice (one comment at nyedtech was; “I don’t have 2 professional development days to learn a new computer program”.) or we should see a redesign of practice that is both relatively easy to implement and worth the effort.  I believe that real progress will require some type of redesign, but it has to fit the larger picture of what is needed in education as it evolves into a data intensive practice and it must make teacher’s work more productive.  Anything that increases the workload will not cut it.  My own take is in some version of the flipped classroom that involves adapted learning.  Lower level knowledge tasks are handled by technology and are linked to higher level skills that are more teacher intensive.

Data intensive technology is certainly the future of education, but as InBloom has highlighted, people are very sensitive about students data.  InBlooms CEO Iwan Streichenberger and Jose Ferreira both characterize this sensitivity as a misunderstanding, however this mischaracterizes and trivializes valid concerns.  For data to have meaning, it must be embedded in practice.  What critics of InBloom were mostly worried about were potential problem in practice.  The Reuters Article K-12 student database jazzes tech startups, spooks parents, Quotes Frank Catalano:

“The hype in the tech press is that education is an engineering problem that can be fixed by technology,” said Frank Catalano of Intrinsic Strategy, a consulting firm focused on education and technology. “To my mind, that’s a very naive and destructive view.”

Frank also recommends:

We need to pull back and think small, not big.  . . .  By precisely packaging and identifying what data is gathered, how it will be analyzed (or “mined”), and what result is anticipated, you remove the vague what-ifs. Everyone is then judging discrete products that can be understood, poked, prodded and dissected.  . . .  Transparent. Tangible. Aiming for trust. It’s not a perfect plan. But it sure as hell has got to be better than what’s happening now.

Finally there was a comment by Dr. Eric Tucker of the Brooklyn Lab School on the schools role in identity formulation.  This wasn’t highlighted in the wrap-up, but I think it deserves recognition that the impact of data should be conceived as a educational outcome, not the solution of an engineering problem.  Students are not widgets.  Nore are they data points.  We must not loose sight that we are building educated people and the core of that process is found in identity formulation.

Clarifying Concepts in Education and Pedagogy

This post is a preface to my post on Gergory Loewen’s hermeneutic pedagogy.

Much of the empirically based research in education seems fadish.  We must consider that the problem might originate in what Wittgenstein referred to as a conceptual confusion, based on a miss-understanding of how concepts relate to methodology.

‘The existence of the experimental method makes us think that we have the means of solving the problems which trouble us; though problem and method pass one another by.’   In the same way, using the techniques of  mathematical proof cannot solve the fundamental problems of mathematics.  In both cases we must turn back to a deep and sustained examination of the conceptual basis of each discipline. (Wittgenstein and Psychology)

In education today, what is it that we want students to learn, who do we want them to become and what pedagogy do we employ to those ends?  This question is at the conceptual core of education.  To this I have a 3 fold answer.

  1. We want to pass on to the next generation what it means to be a functional person in todays society.  To know the beauty possible in music, the complexity and competing claims in the development of democracy, the depth of self-understanding in Shakespeare, or the ability to evaluate scientific claims.  The beginning of these aims, the first steps, is to be found in the knowledge of facts, theories and disciplinary concepts and languages.  Skills in reading, in numeracy, and in using various discourses.  Histories, common narratives, and cultural traditions.  This is the goal of cultural transmission.  It is not the end, but understanding the culture into which we are immersed is the beginning of any educational journey.
  2. Second, we must do more than “parrot” this knowledge.  We must read not just for comprehension, but to interpret language in multiple way and to understand how it can be directed to different people.  We must have more that the ability to calculate, we must understand how numbers fit a purpose, whether it is making a budget, devising a mathematical proof or evaluating a statistical claim.  We must know more than historical narratives, we must know how they relate to ourselves and to others.  This is extending basic learning and making it function as practical knowledge.
  3. Finally, we must use this knowledge to carve out our own path.  To become the self-reflexive practitioners that are the creative innovators, collaborators, communicators and strategiers; able to solve the problems of  both today and tomorrow.

Once we are conceptually clear on the ontology our students, who we want them to be and to become, then it will be time to address the pedagogy.  How will we make it happen.  This is my corresponding pedagogy.

  1. Direct Instruction monitored for recall of basic facts and knowledge as well as the schema that allow us to efficiently categorize this knowledge base and retrieve it when needed.  (Including both the schematic conceptualizing and the technological scaffolding to enable us to access and find information when it is needed; i.e. Artificial Intelligence)
  2. Performance abilities and project methods that give us the opportunity to engage in practical activity using our knowledge and to be able to participate and be literate in disciplinary discourses.
  3. Opened Ended Projects involving complex problem identification and problem-solving.  The opportunity to demonstrate character and persistence.

 

Why Performance-based Education is Needed

We (have arrived) at a most surprising conclusion: . . . the things supposedly contained “in” (our inner lives) are not to be found “inside” us as individuals at all, but “in” the continuously unfolding relations occurring between ourselves and others (or an otherness), in our surroundings.  (We cannot) hide the contents of our inner lives wholly inside ourselves, for, Like it or not, we “display” them in the unfolding movement of our living out our lives, responsively, amongst others.  . . . we cannot but be immersed in it. (Quoting Wittgenstein) “Only in the Stream of thought and life do words [and our other activities] have meaning”. John Shotter, 1998, Social Construction as Social Poetics

Compare the activities of 2 students.

  • One studies a book, hears a lecture, and memorizes facts and theories of lead-base paint as an environmental hazard, before taking a test of recall.  This is educating the latent mind of a student.  But realistically, how long will this information be available?  How well prepared is that student to be a productive part of society?
  • A second student also studies this book, but is not concerned with recall, confident that the content exists in digital resources that act as a scaffold to their understanding and can be located whenever needed.  This students then participates in a peer discussion locating potential lead problems in their community and strategizing how this problem might be solved including additional research for resources through governmental and environmental organizations.  The students defends their activities and strategies orally and they include a record of the resources they used in devising and supporting their strategies.  They also documents their actions in a digitalized portfolio.  How well prepared is this student to participate in society, to understand this topic in depth and over time, and to be responsible to their peers and their teacher for their engagement and their actions?

This is the educational relevance of Wittgenstein’s preference for finding meaning through practice.  We have an idealized view of cognition, that our knowledge can be contextualized without contextualing cognitive skills.  Knowing something is a cognitive skill.  Being able to apply that knowledge within practice is also a cognitive skill, abet at a much higher functional level of cognition.  This higher functional level represents the difference between project-based learning with performance assessment and lower level pedagogy with recall-based standardized assessment.  Certainly the second student has emerged from this activity as a more capable, confident and engaged person.  This doe not mean that facts and theories are not important.  These types of things make up a significant portion of the discourse that students must have in order to engage each other, as well as the experts in this topic.  But until they have engage responsively with others in authentic situations, this higher level of cognition will not be fully developed and even the lower level knowledge will not be significantly understood.

Locating Performance in Pedagogy

Philosophy has a radical way of approaching and dealing with knowledge – for instance, it tries to overcome doctrines which do not question themselves and to compensate for the progressive drift of using and expanding knowledge only technically. Philosophy tries to understand the world . . ..  From: Lucian Ionel

As Lucian Ionel notes, this is an important part the philosophical method of Gregory Loewen’s Hermeneutic Pedagogy.  It’s yet another way of looking at the educational process and noticing what normally flys under the radar.  Loewen’s method seems to be categorizing pedagogy into three classes: Hexis, Praxis and Phronesis.  These 3, along with Episteme and Techne, form the intellectual foundation of Greek philosophical thought.  Episteme is concerned with aspects of knowledge and Techne is about craft or skills in production, both important, but Hexis, Praxis and Phronesis seem to make up the the core ideas of Loewen’s educational processes. I’m studying his approach and think that it might fit the direction of my recent thoughts about performance assessment.  This post is preliminary, about how my previous thought might map onto Loewen’s basic framework.

The specific analysis that Loewen pursues is decidedly Marxist and I do not share this approach.  For instance in Helix (introduced below) Loewen focuses on the reproduction of capitalist repression.  It’s true that current problems with inequality are an supported by the reproduction of a political economy, (see the Piketty discussion everywhere on the web these days), but I want to focus on the need for reproduction if we are to have any kind of culture.  We can discuss what should not be reproduced, but to stop reproduction would mean stopping culture itself.  Praxis also has a Marxist interpretation in Loewen and it has been a term with a substantial history in Critical Theory, but again, extension can be more than just a method for resistance.  Extension (as praxis) and phronesis (as wisdom) can be seen as the way in which culture remains a living and growing entity, able to adapt to current and future challenges.  Thus, I like Loewen’s analytic framework, I just disagree with it narrow NeoMarxist interpretation.  Indeed, it is possible that by extending this framework to approach all aspects of a complex and multifaceted culture based reality, it may be able to reflect back and re-approach it’s original intent from a more productive direction; though it is not my intention to pursue this.

Helix

I will key Helix as repetition and re-production.  It focuses on the passing of cultural knowledge.  In current educational practice, think of Helix as represent the standardized curriculums associated with No Child Left Behind and the Common Core.  These curriculum represent the basic knowledge that is expected by all citizens (re-production) and is (at least partially) achieved through memorization and direct instruction; pedagogy that is high in repetition.  Many current educational practices can be represented by Helix.

Praxis

Praxis, generally understood as practice,  here is keyed as extension.   Think of representing applied knowledge that expands and changes according to the contexts and needs of practice; the learning necessary for practical performance.  This is often considered learning transfer, but in the wake of social cultural learning theory I think of this as extending by adding new learning.  This is not emphasized in current educational practice.  You can see it in activities such as creative writing, service learning or project-based learning, but it is often conceived as an after thought, not as a core educational component.

There are 2 things that should be  included in praxis education to make it more of a core goal of educational practice.  First, at this level you still want to provide lots of structure to these activities and to link them to existing curriculum.  Educational scaffolding can be used as the glue that links the curriculum to the activity structure.  Secondly, bring measurement into these performance activities. Measurement is a core component to education practice.  The inability to satisfactorily measure performance-based practice hurts its standing.  This means development not only in educational practice, but also development in educational measurement.  Note – This does not mean standardized assessment as currently practiced.  See this post on Ontologically Responsible Assessment for more info.

Phronesis

Phronesis is often translated as practical wisdom and it is the second part of my take on performance-based learning.  This is what I consider to involved higher levels of cognitive learning as well as what is often considered character education.  This certainly includes the higher levels of Bloom’s taxonomy, but broken down into more socially relevant skills that are more practice oriented and more socially oriented.  Bloom’s categories are overly individualistic and do not include socially interactive and practice relevant abilities that are becoming increasingly important for today’s workforce.  This is even more true of Bloom’s Affective and Psychomotor Domains which are more closely based on outdated behavioral theory.

Some of the qualities and cognitions to include are: problem identification and solving, creative thinking, situated strategic thinking, self-motivation, persistence, resilience, metacognition and self-directed learning, collaboration, effective situated communication and the ability to form strategic relationships.  For me this is similar to the Praxis level, but it is more open ended and with less structure and less dependence on specific curriculum.  At the praxis level, scaffolding was more knowledge based and emanated from standard curriculum.  At the Phronesis level, we’re moving toward a more skill and abilities foci.  Scaffolding at this level are more socially oriented and come from teachers or peers.

This Phronesis level asks a student to explore self-knowledge; not to just use knowledge in a technical sense, but also in a consciously creative and moral fashion.  This is Lucian Ionel quoting Loewen:

What is gained through this process is what we call self-knowledge: “Phronesis sees through the practicality of repetition and extension by seeing them as rationalizations for the world as it has been. In its subtle but forceful presence, the wisdom of reflective practice asks us to stand outside of the dominion of discourse, the caveat of custom, and move ourselves into the brightest human light of self-understanding anew.”

Where helix and praxis can be scripted (at least to a certain sense in praxis) phronesis is open-ended and reflexive.  It leads to process questions such as: Why is it this way; how have we arrived at this point?  What does or does not make sense here?  Can things be different?  How would you scale a new approach?

These skills and abilities are some of the most important personal qualities in personal success, but fall mostly outside of current educational practice.  They are not only the most difficult to measure, but measures tend to serve different purposes in the educational process.  The overall process is more relational and less mechanistic than at either the Helix or Praxis levels.  These measures must be concieved in more of a joint dialogical nature and less of an automated and behavioral fashion.  This does not mean that we give up on scientific objectivity or become less empirical in measurement.  But it does mean that we do not allow narrow definitions of empirical objectivity to constrict the construct we want to measure.  Narrow (and more traditional) measures represent the “doctrines which do not question themselves” and are the ones who fail “to compensate for the progressive drift of using and expanding knowledge only technically” which Ionel mentioned in the leading quote.

Let’s Bring a Level of Artistry to Building Forms of Digital Life

Matthias Melcher’s post on the digital humanities has got me thinking about extending the ideas from my post here that  referenced Lee Drutman’s ideas on the creativity of quants.

Let’s start with this John Shotter quote about Foucault’s Archeology of Knowledge; ideas about how the world got to be the way it is now.

But now, many take seriously Foucault’s (1972: 49) claim that our task consists of not – of no longer – treating discourses as groups of signs . . . but as practices that systematically form the objects of which they speak.

In many discussions the humanities and the sciences are structurally defined by how they differ from each other.  But step back; distance oneself to see the practice and form of life that normally escapes notice.  People engaged with educational discourses are shaping educational practices (forms of life) and students (their object) much as a painter shapes the forms on his or her canvas.  This is not to critique these practices, but to bring to our attention the artistry that is possible in creating all forms of life: not just painting and literature, but no less in educational practice, data science, or social science.  Also, as participants jointly engaging in these forms of life, let us also bring artistry to the objects of which they speak; us.

Here’s my main point: Data science is about to transform education.  It can take many different forms.  Will we take the notice and expend the effort to add a level of artistry in what we create, or will we blindly stumble through.  Can data be an architectural tool through which we create a more beautiful world.

Unpacking Ontologically Responsible Assessment

In this post I want to unpack the term Ontologically Responsible Assessment mentioned in this post

Why Develop an Ontology:

An ontology defines a common vocabulary for researchers who need to share information in a domain. It includes machine-interpretable definitions of basic concepts in the domain and relations among them.  . . .   There is no one correct way to model a domain— there are always viable alternatives. The best solution almost always depends on the application that you have in mind .  Source

When people say that students need 21st Century skills, what they really mean is that they want to change their ontological commitments as to what students are, and to what they will become.  When we move from a mechanistic factory model of education to a dialogic networked model; we are really changing our ontological commitments from components in a machine to actors in a network.  Ontologies try to clarify questions about the nature of being and becoming a student in the context of educational practice.  I would add (to the typical information systems objectivist account) that an ontology in educational practice also involves recognizing that students are constituted by networked relationships and the various domain discourses within which they interact.  The main difference in this ontology is that (in contradistinction to most information systems ontologies) its organization is not hierarchal and behavioral, but rather contexted, networked and dialogic.  This doesn’t mean there is no place for hierarchal behavioral objectives, just that they no longer form the core of our educational goals.

Why Responsibility:

Depending on whether one believes that reality is objectively given or subjectively / collectively constituted, the understanding of responsibility will differ. This, in turn, has a serious impact on how individuals and collectives can or should use IS (Information Systems).  . . . Reality is thus not given and open to objective discovery but outcome of the intentional activity of perception and interpersonal communication. This means that the dimensions of responsibility must be discovered through communication. (Stahl, 2007 Available from Research Gate or Ontologies, Integrated Series in Information Systems Volume 14, 2007, pp 143-169)

Education can’t be conceived through objective behavioral description, rather, it is conceived in the context of conversational realities.  Students are not cogs in a machine, but are people and the conversational realities where we meet them involves commitments, requirements, privileges, and various other high level latent traits that defy easy objectification.  To be responsible is to jointly actualize an educational program.

What Do I mean by Assessment:

What is the purpose of educational assessment?  Wikipedia speaks about documenting knowledge, skills, attitudes, and beliefs.  Merriam Webster talks about making judgements.  Edutopia talks about assessment as a mechanism for instruction.  I want to focus on another aspects that may seem technical, but I believe gets to the heart of the matter.

What is it that we measure are latent constructs for the most part.  As Michael Kane (2013) frames it:

Test scores are of interest because they are used to support claims that go beyond (often far beyond) the observed performances. We generally do not employ test scores simply to report how a test taker performed on certain tasks on a certain occasion and under certain conditions. Rather, the scores are used to support claims that a test taker has, for example, some level of achievement in some domain, some standing on a trait, or some probability of succeeding in an educational program or other activity. These claims are not generally self-evident and merit evaluation.  Validating the Interpretations and Uses of Test Scores

More than anything else, assessment, at its core, is the process of estimating a latent trait and making it visible.  It is the first step in the analytic process of drawing connections, but we can’t connect the dots until they are visible to us.  There are 2 people for whom this is of primary importance: the teacher and the student.  If we observe the educational practices that involve testing, these are often the last 2 stakeholders that are given consideration, but they should be the first.

Conclusion

This 3 fold understanding of educational assessment includes developing an ontology where assessment practices recognize a full account of the being and becoming of students.  It does not restrict our view to what is easily measured, but essentially meaningless in the bigger picture or final analysis.  Secondly, it is responsible in that assessment is linked to an expectation for engagement that goes beyond behavioral description to fully recognize the full complexity of that student engagement as a dialogic and networked individual.  And finally, it does not use data in a mechanistic fashion, but uses construct measurement to make their joint responsibilities and ontologies visible to teachers and students in everyday educational practice.