Naturalistic Decision-making or Algorithmic Practice: Which is Appropriate and When

Interesting Article in APA’s American Psychologist.

Kahneman, D. & Klein, G. (2009). Conditions for Intuitive Expertise: A Failure to Disagree, American Psychologist, 64, #6, 515-524.

The question, what works best, the intuition of expert decision-makers (Naturalistic Decision Making) or a statistical prediction algorithmic approach (Heuristics and Biases).

The answer of course, it depends on the context.

Intuition (which is presented as a form of pattern recognition) works well when the context include clear and consistent patterns and the experts has ample opportunities to practice recognition.

Where simple and valid clues exist, humans will find them given sufficient experience and enough rapid feedback. (p. 523)

This expert pattern recognition type of decision-making is especially relevant when time is a factor like in nursing or firefighting.  In situations where there are contra-indications, an algorithmic would be warranted, but the authors note there may be a potential for push back from practitioners.

An important point here is that an evidence-based approach is portraited not a simplistic application of science, but rather the development of a specific practice oriented algorithm – an scientific extenuation of the practice.

Contra-indications for a naturalistic decision-making process would include:

  • weak or difficult to detect patterns (e.g. high ceiling effects),
  • the lack of feedback,
  • feedback over long time periods or situations involving wicked problems where the feedback is misleading.

Contra-indications for a hubristic algorithmic approach include:

  • a lack of adequate knowledge about relevant variables,
  • reliable criterion,
  • a body of similar cases,
  • a cost benefit ratio that allow for algorithm development,
  • a low likelihood of changing conditions that would render the algorithm obsolete

The authors also note that algorithmic approaches should be closely monitored for changing conditions.

My take: Kahneman and Klein set up their discussion as a debate between themselves and discuss different approaches primarily as an either or choice.  I value their clarifications, but I would like to think of the many other situations where algorithms would be appropriate to supplementing not replacing naturalistic decision-making.  For instance, they use nursing diagnoses as an example of a reliable intuition space.  In some situations it is appropriate to use it, however diagnosis is a complex tasks that can include a large amount of data that can be combined in different ways.  I’ll have to look at the literature to see if there is a contra-example for naturalistic decision-making.  I’m not saying that naturalistic decision-making is inappropriate in many situations, only that they seem to be short changing algorithmic approaches.  There are also indications that these to authors are not sharing a philosophical heuristic framework.  My bet is that the positivist side is overstating naturalistic bias (which mean failing to see their own) and the naturalistic side is ignoring sources of bias when is suits them (throwing our the scientific baby with the bath water).  Again this is pointing to a need for a framework that can being people with different perspective into true communication and exchange.

I. The Issue, where should we go with higher education?

In the future of higher education conversation I take the perspective that the future is now, or rather, about following the trends that are discernible now.  It less about what to expect in the future than it is about what should be happening now.  These future questions address 2 potential areas: technology enabled new possibilities and the weaknesses of current pedagogy and curriculum.

1 New possibilities enabled by technology include things like: distance-education, open source content and programing, and increased connectivity through social some.

2 Weaknesses of  current pedagogy and curriculum, (allowing potential university competitors or collaborators) is more interesting because it is about change, a much more difficult process.  It also includes critiques of social science and educational paradigms (such as education, sociology or business MBA programs) and who can potentially tap into areas of discontent, especially discontent over the value of education and continued rising costs.

This will take more than one post and could be a potential research project.  I’ll begin by looking at the critiques of pedagogy and curriculum.

II. The Critiques of the current system

(a) The New York Times has issued debate on teacher preparation.  The debate is wide ranging, but there is significant current around the idea that education degrees do little to prepare you for classroom skills you will need in real schools.

(b) Seth Godin has a similar take on business schools (another arm of the social sciences).  First, Seth say b schools are good for 3 things:

  1. screen for future employers
  2. to build a network
  3. and third

“(and least important) reason to go to business school is actually to learn something. And this is where traditional business schools really fail. The core curriculum at business schools is as close to irrelevant as you can imagine.”

Seth goes on to indicate that the following list is what people really need and that b schools do not supply:

  1. Finding, hiring, and managing supergreat people
  2. Embracing change and moving quickly
  3. Understanding and excelling at business development and at making deals with other companies
  4. Prioritizing tasks in a job that changes every day
  5. Selling — to people, to companies, and to markets

This is an interesting critique, although it’s bait oversimplified.  Seth has completed an alternative MBA program and I’m guessing that his program is more about changing pedagogy with a more open curriculum.  When you think that open source education may open a very large universe of potential curriculum and the new pedagogy is how you navigate through this universe.  It highlights the increasing importance in trust: in yourself, in student, in teachers, mentors, and in the educational aspects of the collaboration process.

(c) Henry Mintzberg (of MiGill University) suggest closing down MBA programs in a Harvard Business Ideacast podcast; (also available through itunes) that management can not be studied out of context and leads to too many false positive decisions (to use a research based metaphor).

III My Take –

(a) People need resources, now more than ever, and university resources are a potentially large source, but we need a longer time view.  If we are forced to leave the university for resources, it will be a hugh loss of potential resources.

(b) How can we devise a pedagogy that is future oriented?  You might learn something today, but you’ll need a reminder 10 years from now, or you’ll need to revisit it in 10 years, or you’ll need that support community in 10 years and all the years in between?  How can we pedagogically organize content (curriculum) so that it can be accessed, synthesized, and further developed in a just in time fashion?  (That last one sounds like a pedagogy and technological question.)  Organizing things must be part of it.  Social media must be part of it.  Teaching process (not content) must be part of it, but an organized layered approach.  Following a rote process will not do (see the post on Bill Starbuck), you need to understand what the process is doing so that it can be shaped appropriately to the context.

(c) We need to re-think what it means to be an educated person in many different fields.  A problem with many programs is a common problem, they think that they already know what to do, and maybe 30 years ago there was an illusion that they did indeed.  What is an educated person; what can they do; who can they become?

I’ll chew on that for a while.

The Big Shift: Moving to a social-cultural-constructivist Educational Framework for Organizational Learning

While reading Jay Cross’s comments on John Hagel’s definition of the Big Shift the thought came to me, that this is really a re-definning of knowledge management within a framework that would be acceptable to a social-cultural-constructivist.  Here are a list of Hagel’s definition categories and my thoughts about them.
From knowledge stocks to knowledge flows: I interpret this as a shift from an attempt to objectify knowledge to the recognition that knowledge is bounded by people and contexts, and that knowledge becomes useful when actualized in real-time processes.  You don’t need a database of content that was written for different contexts and different times.  Instead you need access to conversations with people who have a degree of shared understandings (cognitive contexts).
From knowledge transfer to knowledge creation:  Constructivism is often considered synonymous with discovery learning and I don’t think that is correct, but learning is a building process.  Except for modeling (think: mirror neurons) transfer isn’t a valid metaphor for learning.  Better metaphors are creating, building or growing.  These are literal metaphors if you think of learning as the neurology of synaptic development.  Knowledge creation is often achieved by synthesizing new connection between previous knowledge in new ways and learning is represented neurologically by making new connections between existing neurons.
From explicit knowledge to tacit knowledge:  I really don’t like the term tacit knowledge; I’ve never seen a good definition.  Sometimes it’s explicit knowledge that hasn’t yet been well expresses, sometimes it refers to contextual elements.  I’ve always believed that knowing only exists for doing things, the idea that the deed preceded the word.  Sometimes explicit knowledge is just about trying to ascribing more capability to abstract knowledge than it is able to handle.  Let’s just accept that knowing is for doing, it’s one of the main reasons for getting learning out of the classroom and into the world.  Hagel doesn’t seem to realize this yet and why I don’t seem to get much value from his paragraph on tacit knowledge.
From transactions to relationships:  Trust is indeed becoming more and more important.  I also relate the idea of trust to Umair Haque’s idea of profiting by creating thick value, doing things that make peoples lives better.  I really believe that the transition from transactions to relationships and from thin value to thick has a lot more to do with financial and accounting frameworks than it appears on the surface.  The financial set up has to fit the situation correctly, especially if finance is driving your activity.
From zero sum to positive sum mindsets:
This has a lot to do with boundary crossing, open source, and the aforementioned transaction to relationships paragraph.  A major goal of all organization should be identifying their zero sum process pockets and thinking about moving them to positive sum frameworks.  Often the key is not in the processes themselves, but in the frameworks and cultural understandings that support those processes.
From push programs to pull platforms:
People tend to think of social media here, but that’s just a technology platform.  What is needed first is a cultural platform that makes employees partners and then a relationship platform that blurs organizational boundaries so there is a network to pull from.  While technology can facilitate much, people are the foundation and institutions are important facilitators.
From stable environments to dynamic environments:
This is not a choice, environments are becoming more dynamic, the trick is to develop resilience, the ability to identify when change is needed and the ability to adapt in a timely fashion.  The trick is to not let change become disruptive from a cognitive and a work-flow standpoint.  Sense – learn – respond, it needs to happen all the time and at all levels.  Organizations can cope if individuals are always learning and striving to improve, (something I believe is a part of human nature, that is if organizations do not make structures to stifle it) and if organizations take steps to be flexible in their policy structure.  Refer to the previous paragraph on transactions and relationships.  It is important the employees trust their organization and that their organization must trust their employees.  It;s about creating thick value through and through.
Again this is all pretty much consistent with a social cultural constructivist psychological and educational framework.  Previous ideas about knowledge management could be thought of as a management corollary to positivist psychology.  A rational view that just doesn’t square with the way things seem to work in real life.

Network ROI

Interesting IBM article I was thankfully pointed to by the Evidence-based Soup Blog
Wu et. al. (2009) Value of Social NetworksA Large Scale Analysis on Network Structure Impact to Financial Revenue of Information Technology Consultants
Care is needed interpreting this research as it is correlational and can not imply causation, but I would emphasize three findings:

  1. I believe there is evidence that diversity in project teams (maybe coupled with good communication skills) improves performance.  Wu provides evidence that this could apply to communication networks as well.
  2. Having access to powerful individuals (in the hierarchy) improves the performance of project teams
  3. The social networks of the entire project team seems to be more important than the networks of individuals

This would imply that companies should encourage the development of strong diverse project teams networks and should  support the involvement of upper level management in project team networks.

Channeling Pandora: Ideas from a 2007 Interview with Management Professor Bill Starbuck

Reading through documents from the Stanford Evidence-based Management Blog Site, I came across an interesting article (Un)Learning and (Mis)Education Through the Eyes of Bill Starbuck: An Interview with Pandora’s Playmate (Michael Barnett,(2007) Academy of Management Learning and Education, 6, 1, 114-127).

Starbuck seems to be concerned with two things: (1) methodological problems in research and (2) un-learning or fossilized behavior in organizations.
On methodology:  You can’t just apply standard statistical protocols in research and expect to get good answers.  You must painstakingly build a reasonable methodology fitting your methods to contexts and tasks, much like you are fitting together the pieces of a puzzle.  In a personal example: I consulted practically every text book I had when developing methods for my dissertation, but the most common were introductory statistical texts.  I kept asking myself: what am I doing, what are the core statistical concepts I need to do this, how can I shape my methods to fit my tasks to these core concepts.  Almost all advanced statistical techniques are an extrapolation of the concepts found in introductory statistics and your can’t really understand how to use these advanced procedures until you understand their core and how they fit your methodological circumstances.  As Starbuck points out, the concept of statistical significance is the beginning of results reporting, not the end.  You must go on to build a case for substantive importance.  He points out that mistakes are common in reporting effect sizes.  I believe that this often happens because people simply apply a statistical protocol instead of understanding what their statistic are doing.

A favorite issue of mine (that Starbuck implies, but does not address directly) is the lack of a theoretical framework.  Without theory, you are flying empirically blind.  Think of the four blind men empirically describing an elephant by holding a trunk, leg, body and tail.  Vision (or collaboration) would have allowed the men to “see” how their individual observations fit together as a hole.  You must begin with the empirical, but reality is always larger than your empirical study and you need the “vision” of a theoretical framework to understand the larger picture and how things fit together.  Theory is thus an important part of your overall methodological tact.

On (Un)Learning: Starbuck discusses the need to unlearn or to change organizational processes in response to the changing environment.  It is a problem where past success cloud your vision obscuring the fact the what worked before is no longer working.  The problem is not that people can’t think of what to do to be successful, it’s that they already know what to do and their belief keeps them from seeing that there even is a problem or seeing the correct problem.  Starbuck talks about problem solving in courses he taught.  He found that people often found that the problem they needed to solve was not the problem they initially had in mind.  Their biggest need was to change beliefs and perspectives.

The psychologist Vygotsky spoke of something very similar as fossillized behavior.  As someone is presented with a unique problem, they must work out the process solutions to the problem externally.  Later the process is internalized to some extent and become somewhat automated, requiring much less cognitive load.  After more time this can become fossilized, that is, behavior that is no longer tied to a process or reason, but continues as a sort of tradition or habit.  This would apply at the organizational level as well as the individual posychological level.  I would like to investigate the concept of organizational resilience as a possible response to fossilized organizational behavior as well as a way of responding to extreem events.  This would  emphasize an ability to change in response to environmental demands.  Starbuck thinks that constant change is too disruptive to organizations, but I believe that there may be a combination of processes, capabilities and diversity that enable organizations to sense and respond, not necessarily constantly, but reasonably on an ongoing basis as well as when the enevitable  black swan happens.

A Learning Infrastructure: Where to Begin

My post on 6-18 needed elaboration.  I elaborated on strategic aspects on 6- 30 and today I will elaborated on my 6-18 point #3 – the need for an organizational learning infrastructure.  By infrastructure I mean the technical structures that support enterprise functions.  In often refers to physical assets such as computer networks or electric grids.  These are important, but in learning, I’m primarily referring to intangible assets such as management concepts, learning constructs and knowledge networks or more specifically the following:

A knowledge network supporting the integration of evidence-based research knowledge into daily practice.

Management support for change, appropriate to a knowledge intensive environment (Appreciative Inquiry or other forms of leadership that reject a hierarchical top down approach are frequently called for)

A collaborative culture organized for agility (Agile Management), resisting departmentalization and supported by an integration of social media and other collaborative forms into daily work processes.

These are intense and complex ideas that I plan to delve into in more detail this month.  More to follow. . .

A Good Plan for How to Make Learning Strategic From Charles Jennings

On June 18th I responded to Michele Martins thoughts,  That post needed more elaboration, which has been supplied by Charles Jennings’ post.  This is about my suggestion #1: Strategy must play a stronger role where learning is part of the organizational narrative not just an afterthought.  Jennings excellent post suggests how to go about supporting a vision of learning and development (L&D) as a strategic business tool in 5 basic actions:

1. L&D departments need a strategic departmental vision, aligned with the organization’s strategic vision and priorities, and supported by an appropriate model of governance that includes senior business leaders. The following graphic is an example of a governance structure from Jennings Blog:


3. Integrate frontline managers into all aspects of L&D.  The managers are the point at which learning must take place or it is likely to be ineffective.  Jennings draws on the information from the Corporate Executive Board/Learning & Development Roundtable in this graphic to support his view:


3. Embrace Innovation (such as social media and informal learning trends).  This would also be supported by fostering a creative environment.  My previous post on Supporting and Developing Creative Environments has much relevant information.

4. Use technology and tech tools in innovative ways.  Don’t just put course work into technological forms, but use technology to rethink learning experiences

5. Develop internal departmental capabilities and skills such as consultancy skills, (communication and skills in leading and persuasion), a deep understanding of L&D contexts

A Networks Model for Evidence-based Management and Knowledge Transfer

Couple of interesting reads this morning (Bandura 2006 and Guest 2007) that are relevant to the topics of learning, performance support, knowledge transfer and evidence-based management (EBM).  The bottom-line:

(From Bandura) Knowledge transfer in many situations can be seen as a form of learning that proceeds through ongoing modeling with feedback and increasing approximation, not by an explanation of abstract information.

(From Guest) Practitioners do not generally change their practices as a result of abstract knowledge, but from the example of others in their organization or field.  (e.g. bankers looking to other bankers or retailers looking to other retailers)

Furthermore – Guest laments the current state of EBM.  Changing it requires attention to the communication process (communicator, message, medium and receiver) and the building of bridges (both traditional and non-traditional) between research and practice.  Guest is pestamistic about the readiness of the management field to address EBM.  I would disagree and suggest the following based on Guest’s communication process analogy:

  • Communicator – The concept of EBM is not an outcome, it is the bridge that can close the gap between researchers and practitioners. However, the communicator must stand on this bridge, not on either shore.
  • Message – Standing on the EBM bridge, the most important aspect of research is validity.  It is a view of validity that begins with the whole of the concept (not the narrow view of traditional research validity).  Research is not valid until the consequence of it use in practice can be demonstrated.  See a previous post on validity here although I may need to do additional work on the validity concept.
  • Medium – In the light of Bandura, the real medium of concern, in fact, are the people in the practitioner’s network.
  • Receiver – We need to build up the scope and diversity of practitioner’s networks and the ability of these network to act as learning models for evidence-based practices.

Research Recommendations on Supporting and Developing Creative Environments: A review of the article: Creative Knowledge Environments

This is a follow-up to my creativity post of 6-15-2009

Reference: Hemlin, S., Allwood, C.M. & Martin, B.R. Creative Knowledge Environments (2006). Creativity Research Journal, 20, (2), pp. 196-210.  Which is also available online here.

First the results.  In my reading, the recommended conditions should have:

  • clear (and coordinated) objectives;
  • a research culture built over time;
  • a supportive and cooperative group climate that seeks and respects diverse thought in identifying salient problem features (sense-making);
  • strong vision with leadership in stimulating, structuring and promoting ideas;
  • a flat decentralized organizational structure giving appropriate autonomy to individuals linked by collective goals;
  • supportive and clear communication styles in a highly interactive environment (high levels of social capital) able to mediate any potential clash of ideas;
  • adequate resources (time, funding, equipment, library materials . . . etc.);
  • diverse individual characteristics (discipline, institution, cultural, social, geographical, motivational, etc. . .) with strong and varied individual competencies appropriate for the discipline(s) or field(s) involved;
  • appropriate quality control (although not in too excessive or intrusive a form);
  • an institutional base with an established reputation and visibility’
  • strong but flexible links with individuals both inside and outside of the organization
  • respect for breaking routines when necessary and for taking appropriate risks.

The paper notes that innovation tends to cluster geographically and reasons that:

innovative activities involve a significant element of tacit, embedded, and to some extent locally bound, or ‘sticky’ knowledge that is best communicated face-to-face . . . facilitated by small distances (Asheim & Gertler, 2004)


Looking primarily at the processes that lead to creative products this paper is attempting to identify what:

. . . types of factors are thought to either enhance and hinder creative output, but we are still in the early stages of finding empirical correlates as well as potential constellations of policies and leadership initiatives.

A definition from the paper (paraphrased):

Creative Knowledge Environments are environments, contexts, and surroundings (teams, companies, regions, nations) that exert a positive influence on human beings engaged in creative work producing innovative products.


The authors consider 3 levels of environments macro, meso and micro levels.  In addition they also provide the following more detailed classification schema for characterizing details of specific environments:

Components of Knowledge Environments and their Characteristics

Task characteristics: short-term/long-term, simple/complex, routine/novel, modularised/integrated

Discipline/field: natural sciences VS engineering VS social sciences VS humanities, theoretical VS experimental VS modelling, basic/applied, single paradigm VS multiple paradigms VS pre-paradigmatic, reductionist/‘holistic’, discipline-based/inter- or multi-disciplinary, influence of ‘epistemic community’

Individuals: knowledge, skills, abilities, cognitive style (e.g. broad/narrow, focused/eclectic), motivation, interests, career plans, values, beliefs, other personality properties (e.g. introvert/extrovert)

Group characteristics: size, integrated/loosely coupled, inward looking (‘group think’) VS outward- looking, leadership style, degree of group tension/harmony, heterogeneity/homogeneity of group members, ‘chemistry’ of personalities in the group, composition of knowledge, skills and abilities, agreed on or contested beliefs or underlying assumptions

General work situation for individuals: number of different work tasks or projects, features of time available for research (e.g. sparse/abundant, fragmented or concentrated), job ambiguity (total autonomy VS narrowly defined goals), quality of IT available (including the usability)

Physical environment: facilities, buildings, architecture, location, climate, equipment

Organisation: income sources, economic situation, organisational structure and culture, reward profile, leadership and managerial style (e.g. controlling/allowing), degree of organisational tension/harmony

Extra-organisational environment: small/large economy, expanding/decreasing economy, market characteristics (e.g. open/restricted, global/regional, competitive/monopoly), reward profile, information availability (open/closed), job opportunities and mobility, regional, national and cultural characteristics

Each of the above classifications can be divided into the elements of the social domain (“openness to new ideas or innovation, relations between colleagues or organisations, and routines for the upkeep of equipment”) and the cognitive domain (“bodies of knowledge and skills, cognitive work style and thinking style (e.g. adopting an experimental or ‘trial and error’ approach).  The cognitive domain can also be analyzes for spacial distributed aspects.  Environments can also be classified by the Triple Helix of industry, government, and non-profit.  Although these classifications will effect environments, it is likely that creating positive conditions for creativity will share much more across institutional types than they will differ.


These are general findings the authors derive from the literature and could be considered general recommendations for looking at the creative potential of individuals:

Internal motivation is generally seen as more important in relation to creativity. . .

Many researchers stress the prior need for quite extensive knowledge of the domain.  . . .

individuals should be given sufficient time and opportunities for practice and learning to occur.  . . .

creative problem-solving might rely more on “weak methods” (general problem solving skills)

In general, creative persons show greater openness to experience(,) higher tolerance for ambiguity (and have) flatter hierarchies of associations’ (i.e. having many associations for a particular stimulus with a fairly equal probability for each association, and with associations being easily affected by internal and external events)

Although the authors discuss other personality aspects of creative people, but the research seems more correlational than causational and sometimes conflicting.  I’m reluctant at this time to place much weight on other recommendations regarding personality.

Ideas and paradigms are Important Enablers of Creativity

A timeout from research to respond to a relevant blog post.

George Siemens posted about studies from the PsyBlog relevant to my current  research on creativity.  The PsyBlog post ended with the recommendation to go it alone if creativity is important to you.  I think this is counter productive.  Many important outcomes require group work and diversity in groups can be an important source of creativity when it brings together different perspectives.  Certainly one important factor encouraging creativity in groups are shared ideas and a paradigm based that supports creativity.  The base of my thoughts are in the following comment made in response to George’s blog post.

Good discussion George and Ken;
It reminds me of Vygotsky’s lower and higher mental functions. What Ken describes sounds like a group manifestation of lower mental functions ( a level of thinking shared with animals). Valuing something like diversity may only occur if a higher mental function regulated this primal instinct for conformity.
I would not call this type of creativity destruction a problem with norms, but a problem of lower levels of responding. Something like diversity may require a higher level of function with ideas, paradigms and the sort, mediating the thinking, whether it is a group or an individual. Valuing diversity could be a norm too! Creativity may need to start with an individual, but for a group to participate, you may need relevant shared ideas to be present in the group. As a metaphor, think of shared ideas and artifacts like the neurotransmitters of the group.
In the study referenced by the PsyBlog, we don’t know what kinds of paradigms underly the groups thinking or of the study. Science in my view is a blend of theoretical and empirical. This study sounds like it over emphasized the empirical without a good theoretical understanding of creativity. Sort of like a hold over from behavioral experimental psychology that thought of the individual as a black box where you only measure the inputs and outputs. Measure group creativity, but keep their shared ideas and paradigm base as a variable in the equation.