A reflection on action research “storytelling”

What follows is a 1st person, stream of consciousness  reflection written to my mentor & committee chair.  

I describe  what it was like to record a 10 min video “telling the story” of  some preliminary findings emerging from my action research cycles into curriculum and adult learning. 

The video is hosted  at YouTube.

It will be shown at an international conference in Athens, as part of the Collaborative Action Research Network (CARN) annual conference, as part of a bundle of reports from the Future(s) of Education project, an international  participatory action research network.  

Dr Alana:  

i am just glad to get it out of my head  

i had a real out of body experience recording that one;  

i  am a very effective briefer in person, because i can read the audience pretty well.  

i have recorded hundreds of mini lectures etc for my business and for use here at the college on various topics.

i have never, ever needed more than a single take to record, decent and sometimes even inspired voice-overs  until  last night and that briefing.  

I literally needed about 30 takes to get thru it; most i stopped when less than a minute into it because the tone just didn’t feel right

 i think it has to do with being a fish out of water, and the difficulty i felt in trying to tune my story for an audience i couldn’t see, but more importantly didn’t have empathy for

because the audience characteristics still feel fuzzy to me, i couldn’t call up the right tone, voice, persona to apply  

 this caused me to have almost a split personality in the moment, when i am ordinarily dialed in

 i had a “talking part” and a “look ahead part” that is concerned with shaping the transition to the next point/slide  

but now i had a disconcerting 3rd part that was trying to anticipate the possible reactions of an unfamiliar, and hard to imagine audience  

this is what made me feel so out of sorts

 until i “wore out” the last, 3d part and was able to trust in just telling the story, and accepting the vulnerability of knowing that i couldn’t know the audience, i found i just couldn’t get thru it.  

this is the same phenomenon I spoke with Prof Mike Wesch, the digital anthropologist at Kansas State University, and world thought leader on social dynamics in social media: the camera eye represents the unlimited, unfathomable infinite future of all possible audiences across time and space who can be looking in on the “telling moment”.  

in a sense, its like coming face to face with the unblinking eye of God and wondering what she is thinking  

 it is trust that lets us get thru that moment, the accepting of vulnerability, that creates the empathy that hopefully fills the story, as told, with hope.  

that’s a clumsy way of trying to express my meaning of the risk and vulnerability to “telling” and why it can be such a powerful learning moment, and why we need to model it, embrace it, encourage it, and support it. 

Your “producer’s draft” was exactly what i needed to be able to get out of my own comfortable fishbowl; 

you gave me a bridge to the audience that i could not create on my own.  

this has become an interesting reflection to me already  

please put the video on the website, and any or all of this reflection as you deem suitable  

have a great time at the conference!

Reflections on myself as an adult learner

Who am I as an adult  learner:

I am framing the answer within the context of my “Big 5” (Strelecky, 2007). The “Big 5” focus my thoughts about self, purpose, mission and values.  In Strelecky’s work, the Big 5 are 5 things you want to accomplish in your life. My “Big 5” are all states of being, roles that I want to live with the highest quality (arête). My Biog 5 are:  father, husband,  teacher, student, warrior,

 

Student:

Myers-Briggs Type Indicator:  ENTJ  I am getting closer to the “I” as I get older which moves me from “Leader” to “Scientist” in the typology.  My scores are very high on the NT domain, which gives me a global, theoretical perspective. I notice that I am always searching for the broadest generalizations that can be made from an incident, or the widest application of an idea. It doesn’t take much for me to go off on a tangent. I am least happy when bringing a project to a conclusion, as it feels stifling and disconnected from the dynamic world around me. Finality and endings are disturbing to me, and I dislike graduation ceremonies above all else. I am much more at home in the developmental and conceptual phases of any project. I get bored easily by data gathering and have learned to offload that task to others. I am a good project manager, as I have learned to build teams of various skills and aligning tasks with strengths.

 

Kolb Learning Style Indicator:

The Kolb LSI measures self-reported preferences along 2 dimensions:  Concrete experience-Abstract conceptualization and Active experimentation-Reflective observation.  These 2 dimensions reflect how we prefer to gather our information about the world and then how we prefer to make sense of it. The intersection of these 2 dimensions establishes 4 quadrants, and can be used as a way to describe a classroom population as well as individual learners. We use this model extensively at the Command & General Staff College, and I have become convinced of its practical uses when used within reason.

In this model,  I am classified as an “Assimilator”, which combines a preference for Reflective Observation and Abstract Conceptualization. This means I don’t need to spend too long “in the moment”, fully experiencing every nuance of the moment; I am always ready to begin reflecting on its dimensions, characteristics, descriptions and classifications. As an Abstract Conceptualist, I proceed to place experiences within my larger world view, as a particular example of a class of experiences. I spend little time in active experimentation to validate the data, once satisfied that it makes theoretical sense. 

                These preferences are helpful when approaching new material where the connection to theory is strong or explicit, because it satisfies my need to be situated in the world. I am comfortable with complexity and nuance and am  competent at brainstorming and imagining future scenarios.

The downside of my preferences is that I am prone to overlook deep subtleties in experiences especially if the situation is slow moving. The idea of sitting in a duck blind for hours waiting for birds is my idea of hell on earth. I am also prone to accept theoretical justification as truth and am willing to short change practical validation of new concepts simply because of the theoretical elegance.

                As a consequence of knowing this about myself, I find it necessary to do sitting meditations to work on my mindfulness and presence in the moment, to learn to appreciate the experience simply on its own merits, without a need to explain it or frame it as part of a larger construct. On  group projects I am careful to include pragtmatists and  naysayers who will insist on evidence and results from fair trials before we adopt policy changes.

                These strengths and weaknesses, and my accommodations to the limitations of my learning preferences are an integral part of my business success as an equity trader which puts a value of new ideas, but also on backtesting and forward risk management.

 

Brainmodepower typology: AVK, global. 

I am off the chart on the audio learning, and on the globalization scale.  I have now noticed that when I am really trying to concentrate on learning I do not look at the person talking, but need to doodle in order to free my ears to hear. Doodling helps me occupy my eyes and hands (visual and kinesthetic modes). This has been a problem for others in the past when they would say “Look at me and pay attention!”  when I was doing my best to pay attention.              

2 stories from combat on this topic which reinforces the power of the insight: On a night attack, wearing night vision goggles I had high explosive rounds land near me and “whiteout” my night vision goggles, and I lost my night vision for about 15 minutes: I was able to command my company though because I could hear what was going on via the radio and I had a sense of where things were based on noise, sounds, and the volume of fire. A few days later, in the daylight, I had a hand grenade land very near to me and I didn’t have my earplugs in. I was deafened for about an  hour before my hearing returned, and it was the most frightening experience I had ever had. I felt absolutely cut off from the world and was unable to command effectively. It was terrifying, even though I could see everyone around me and could consult a map.

 

Learning techniques:

I am a fast reader and I prefer to read in burst of 10-20 minutes, rendering my notes in visual, mindmapping form.  I will generally  develop detailed cognitive maps and turn them into slides as cues for recalling detail and cognitive structure. I take semi-structured notes on standard note-taking forms that I have developed over the years to suit my style. I will often color code the notes to make structure even more apparent. When I review notes from my Masters program (15 years ago), they make perfect sense to me and I can recall the circumstances of the classroom and the moment as if no time has passed.  This form of “chunking” supports my assimilating style. 

At any given moment I may be engaged in reading up to 20 books at a time in various locations, and I follow my mood or sense of urgency for picking up the next book to read. When I find myself drifting I stop and do something else until my attention is focused, rather than trying to force concentration.

 I can concentrate for hours at a time in reading if needed, but I prefer the shorter bursts when my mind is feeling especially sticky.  Learning to crate feelings of “sticky mind” is an essential part of my practice of sitting meditation, which Buddhists call “child’s mind”.

I will rarely read a book from cover to cover, preferring to read from top down and outside in, by examining the covers, introduction and forward, table of contents, index and references and chapter summaries first, and then come back to the book after 24 hours when that has had time to digest and become embedded. I will then skim chapters based on my interests, and finally skim the whole book. I have adapted this technique from  Mortimer Adler’s “How To Read A Book” (Adler, 1940) and it has helped me integrate a lot of material from a broad array of fields.

I am not very good in free form dialogues of material, preferring to hear structured presentations that reflect deep inquiry on the part of the presenter. Lectures are excellent for me as I can listen carefully, while doodling and seeming to daydream in my own personal comfortable space.  I enjoy writing and working on a topic while having a background lecture playing, trusting that if something interesting is being said that I will tune in to it. Some of my most creative work is done in this manner in the apparent cognitive dissonance set up by 2 different information streams. I am listening to a Teaching Company presentation on Chaos by Dr Stephen Strogatz as I write this.

My biggest problem as an adult learner is procrastination and time management, since I am always eager to read one more thing before generating my final conclusions. I also find it difficult to recast my theoretical framework of information once established and will generally try to find ways to accommodate pieces of my original insight in an evolving understanding. I try to delay taking final positions in order to gather more information for this reason.

I find it amusing that despite a strong rational component, and a structured approach to learning, that my decisionmaking and sensemaking is much more intuitive than rational.  I trust my instinct far more than my conscious mind. This is a habit perhaps ingrained into me from 15 years of  being an infantryman in combat and trusting my senses in dangerous situations. This habit of mind is so odd that it is even the subject of discussion among peers who know me well and wonder how I can be so rational and yet make instinctive, intuitive decisions.

 

Teacher:

I have been teaching in the Command & General Staff College for 8 years and have reinvented my whole approach to teaching as a result of the action research inquiry while attending CTU. While I acknowledge the need for competence at the data level I also have become much more aware of the importance of the social level of learning.  I no longer think that learning and education are like filling up a pail, but are rather like lighting a fire (to paraphrase  Yeats).

                I am trying to create an educational space in the classroom, in the college, and in my professional work that encourages and supports free inquiry, a commitment to truth and academic freedom, and both a respect for and a seeking out of diverse perspectives and points of view. As a teacher in the classroom  I try to model the behavior I seek from students, by the quality of my preparation, a concern for the learning and perspectives of others, and a willingness to be vulnerable in my ongoing search for knowledge. I am encouraging as many means of formal and informal feedback as possible to help students shape their own educational programs and outcomes. I encourage and support their inquiry in my classes and through support of their independent studies. I reach out to other colleges and programs to create networks of learners and to act as a catalyst for learning.

                I respect the action research construct of multiple ways of knowing (experiential, presentational, propositional, and practical) and acknowledge the learning that can happen through 1^st person, 2d person and 3d person action research.

                I favor the connectivist learning school of thought being developed by George Siemens and Stephen Downes at University of Manitoba, as I believe it represents a realistic, sound, robust and challenging way of developing knowledge and practice to appreciate and thrive under conditions of uncertainty.  More at:  http://www.elearnspace.org/

                As part of my research I am looking carefully at how to add Voice to the environment by encouraging, supporting and promoting the diverse needs, intentions and inquiries of faculty, students and curriculum developers in a way that advocates a move away from an industrial age view of curriculum and towards one of connectivism and individuality. In this sense I have taken on an advocacy perspective that is values-based but which respects the perspectives of other members of the action research teams that make up the projects.

 

 

Father: 

My role as a father influences my role as a student. One of the important reasons for me to begin the doctoral program was to set  a personal example for my kids, who at ages 18, 15, 11 are getting to see their dad doing his homework and reading books every night as a priority.  My father set the same example for me as a kid as he went to night school to work his way up the engineering ladder from “shop rat “to full-fledged design engineer.  I’ve been trying to re-learn math and physics to be able to keep up with my son who is getting ready to go to college next year to be a physicist or an engineer, but just like in video games, I believe he has passed me for good. I am content to listen to him and get him the occasional book to feed his curiosity.

 

Husband: without my wife’s support I could not have dreamed of taking on the active role of student once more; in fact she finally told me to stop moping around and dreaming about it and just get it done. I need that boost from her to get moving at times. I want her to be proud of my work and my goals.

 

Warrior: 

I use Warrior in the eastern sense, as one who is called, by his dharma, to seek mastery of self first in order to protect the weak and promote justice and compassion in the world. This calling is well described in  Trungpa (1984). In this sense, my role as an adult learner is to focus on those things that I ought to be learning in order to improve my practice; to find worthy teachers and learn from them; to questions my own assumptions and preconceived knowledge in order to step outside what I already think I know and to follow my beliefs to their core to find the source.

Warrior learning also has a strong service component, and so the topics for inquiry, the choices for action research must satisfy the “so what” question, must be directed towards a virtuous end. For me, the choice to do action research within my college represents a way to do the right thing in support of my duty to country and soldiers whom I support. Action research’s methodology strongly supports these values, particularly when fellow inquirers are positioned as co-researchers.

 

References:

Adler, M. & Van Doren, C. (1940). How to read a book. New York: Simon & Schuster.

 

Strelecky, J. (2007). The big five for life: Leadership’s greatest secrets. New York: St Martins’ Press. 

Trungpa, C.  (1984).  Shambhala: Sacred Path of the Warrior. Boston: Shambala Publications.

Management games for deep insight

Peter Checkland’s Soft Systems Methodology (SSM) describes the use of models to help us frame questions to ask of the world, and which help us become explicit about our world views, assumptions, frames of reference, theories of cause and effect, values, and desired outcomes.

Checkland, P. (2006) Learning for action: A short definitive account of soft systems methodology and its use for practitioners, teachers and students.  Chichester, England, Oxford Press

 We’ve developed a deceptively simple Force Mgt practical exercise in the form of a card game. The complete rule set is simple; takes 5 min to scan and understand. 

Rapid rule summary:

1. Students buy forces (5 cards) from a production table (a limited deck) and in each of 5 rounds,  deploy them into 5 regions to compete for Victory Points

2. Win:  first one to 51 victory points OR most points after 5 rounds

3. Game: lasts up to 5 rounds

4. Each round has 5 hands , each hand is worth Victory Points (VP)

5. Hand 1 is worth 6 VP, hand 2 is worth 5 VP etc…

6. Player 1 buys from the red deck, player 2 from the blue deck)

7. After you buy your 5 cards, you place 1 card face down in each region (hand)

8. Once all cards are placed,  cards are flipped over and you determine results

9. If your card wins the hand you get the victory points and keep you card; if you lose the hand, you get no victory points and lose your card. If it’s a tie, you keep your card and no one gets points.

10. Each player has an identical deck to buy from.

 It turns out that the development of strategy and then fielding an appropriate force really matters, AND there are distinct choices that are meaningful, available and feasible.

If you are interested, we’d like you to review the rules, and :

• 1. Buy your first round of forces
• 2. Deploy them into the 5 regions for turn 1.
• 3. Send your “Round 1” move to long-kenneth@conus.army.mil, along with a short description of your strategy

 We are interested in examining the variety of forces and the strategy employed in round 1.  Do you, for example:

1. Buy 4 ea 10s and a Joker to kill any enemy aces and retain max budget  flexibility to see what he has remaining?

2. Buy aces early to get a lead on victory points and then protect them?

3. Buy Jacks to kill 10s while still preserving SOME budgetary flexibility?

4. How do you balance economy of force with winning victory points? (efficiency vs effectiveness)

5. Variations?

 And then tactically employing forces, do you:

1. Put aces against 6 and 5 victory point regions?

2. Put 10s against 6 and 5s to hunt aces?

3. Aim for maximum victory points each round?

4. Aim to capture 11 of the 20 available points each round? (ie bluff on 6 and 3, but try to win 5,4,2?)

 In the actual play of the game we’ll look for adaptability and learning, and how strategies change after teams have played each other a couple times etc. 

We’ve play tested it enough to know there is a rich source of insights available in the game and that it is simple to play. We’ll  play it with decks of cards in the classroom 

We prototyped the game in our Force Management elective and are satisfied that that we generate student interest and insight into broader questions of Army force management in an interesting way.

 Here are some student insights gleaned from our playtesting:

1.  Round 1 results dominate the rest of your strategic choices, so getting Round 1 is crucial.

2.  Round 1 strategies are dominated by uncertainty because you have no information about your opponent’s strategy or adaptive style yet.

3. You have to decide when you want to buy strength: early and aim for quick wins, or later after you have seen pieces of the opponents forces and strategy.

4. Forecasting your opponents moves is problematic and make this more like poker than chess or bridge.

5. Aces are like the FCS: dominating until low-cost alternatives found the weakness. It wasn’t unit Aces were developed that the 10s became meaningful, so be alert to deep flaws in complex technologies.

6. Kings are costly but dominate the field; An opponent with Kings drives you to buy Aces but make you vulnerable to 10s.

6. Jacks (J) are a low cost success strategy against 10s, but can be incrementally be defeated by other mid-weight forces.

8. The costs of transforming cards between rounds is significant but manageable and may lead to strategic advantage. Scenario: You buy Aces on the first round and are successful, opponent buys 10s to kill your aces in the second round, but you trade down to Kings which dominate, and which remain difficult to defeat in subsequent rounds.

9. Deciding where (in what regions) to selectively deploy strength

10. Tactical results can overcome strategic insights and strategic failures. Tacrtics can be game changing.

11. What if the enemy has different victory conditions? Price points? Has different rules?

12. What if new cards are introduced after the first rule set is established?

13. How much would you pay to see the opponents’ hands?

14. What if there are partial wins? Or more than 2 teams playing?

15. Simple games can be powerful learning strategies

 Conclusions: the game serves as a way to dramatize very clearly many of our force management challenges and is a useful way to create rapid, deep awareness of prime issues in this domain.

 Here are some insights from a dedicated gamer and management game modeler:

I suspect that for most people’s first play they are strongly influenced by a form of Confirmation Bias: the As are priced higher, therefore new players conduct their analysis from the assumption that As are more valuable. Depending on the goals of your concrete experience, that may be the best argument for keeping the current price structure. However, an ace of spades loses to seven cards, including four cheap ones, where a KH loses to only four cards that are both expensive and vulnerable — the KH is easily the strongest card in the deck.

I assume trade-ins are secret — in fact that for all practical purposes players are operating behind a screen during their setup phase — because knowing whether your opponent has made any trade-ins is very valuable information. You may want to specify that in the rules.

 Given the prevalence of 10s in everyone’s first turn strategies, it seems like the second-cheapest strategy is far more optimal than the cheapest — that is four tens and a jack of spades. That marginal $15 gives you a pretty good shot at a victory somewhere, and a decent chance of carrying more net capital forward.

 Here are a selection of previously submitted moves for  Round 1: (* = Joker)

Strategy 1
Region	Cards	Strategy:                  Cost:  102   Carry forward: 48
6	10h	I’m trying to kill aces while creating and deploying one, but putting it where it is unlikely to run into an ace-killer unless the other guys is trying an ace-killer strategy like mine.  I’ve got cheap on the ace I bought, which is a risk that may not be worthwhile. I’m expecting to kill an ace in either 6 or 5, win 4 outright, and lose in 3 and 2.  Expected results are thus 9.5 points to me, 10.5 points to the bad guys, I will lose approx $35 worth of cards and kill approx $70 worth.  The enemy is expected to have spent rather more than me, so I will have more cash with which to restructure in light of what I find out. Cost: 102
5	10c
4	As
3	10d
2	10s
Strategy 2
Region	Cards	Strategy:                   Cost:  150   Carry forward: 0
6	10s	10 is the ace killer on 6, then we try to overpower each successive category on the way down.  Assumes aces go to 6, which rapidly becomes a tail-chasing assumption.
5	As
4	Ks
3	Qs
2	Js
Strategy 3
Region	Cards	Strategy:                   Cost:  123   Carry forward: 27
6	Jh	Hunting the ace-killers, retaining some flexibility, winning early points
5	10h
4	Ah
3	Jc
2	*
Strategy  4
Region	Cards	Strategy:                   Cost:  150   Carry forward: 0
6	Ah	Maximum strength in every region
5	Ac
4	Qs
3	*
2	*
Strategy  5
Region	Cards	Strategy:                   Cost:  145   Carry forward: 5
6	Ah	Maximum strength in main regions, try to hunt an ace and kill 10s; accept risk in small region
5	Ac
4	10s
3	Js
2	*
Strategy  6
Region	Cards	Strategy:                   Cost:  149   Carry forward: 1
6	10s	Hunt aces and accept risk in regions 5,6, steal points with aces & J in regions 2,3,4

A reflection on mixed methods research in adult education

Introduction

The Research Problem

The purpose of this paper is to offer one vision of developing a methodological theory of mixed methods research co-equal with that of quantitative and qualitative research. I use a case study of the US Army Command & General Staff College engaged in a redesign of its curriculum, its teaching practices and its design process itself in a period of revolutionary change while supporting a nation at war. I describe circumstances and worldviews in which I argue that only mixed methods research may be employed to simultaneously develop a deep appreciation of uncertainty, improve decision making through an appropriate gathering, mixing and analyzing of quantitative and qualitative data, and applying “learning in action” as a strategy to manage success.  I contrast the view of research  as a process of increasing knowledge for control  with a worldview of research as a learning-in-action that allows for deep appreciation of complexity but without the assertion that appreciation and research can lead to prescriptive measures of control.  I examine the merging feedback system of the CGSC curriculum redesign as a mixture of qualitative and quantitative data. The concept of “Voice” that emerges from the CGSC action research process will be described, along with a multi-phased, multi-year research plan that demonstrates the practical development of an interactive dynamic research plan that is also adaptive to interim and periodic results. The paper  reflects a pragmatic worldview as it focuses on practical outcomes inside an organization concerned with real-world results, but acknowledges the importance and utility of the other 3 worldviews described by Creswell (2007, p.6), namely advocacy/participatory, post-positivist and constructivist.

Waldrop (1992, 2008) described the emerging science of complexity in a rich description of the inter-disciplinary work developing at The Sante Fe Institute. Sixteen years later (Waldrop, 2008) he found that the pioneer days of complexity research had evolved into a rich diversity of programs in major and minor universities worldwide, with lines of business and cognitive domains each finding ways to apply the ideas of emergence, uncertainty and complexity in new and profound ways. What remained unchanged from the origins of the research were the questions of what next and so what and how much more is there and what does it mean to apply an appreciation of complexity to everyday problems and opportunities.  The field of education is only beginning to appreciate how complexity and uncertainty may change the dynamics, structure, content and practice of adult education (Siemens, 2004). Professions in particular will be challenged by educating for complexity, since deep, profound, and reliable bodies of knowledge are at the center of professional practice. Educators, themselves members of a profession, are examining what it means to educate, teach and instruct in light of an emerging awareness of complexity.

The US Army Command & General Staff College (CGSC) is a self-described “learning organization” (Senge, 2000), engaged in a revolutionary re-design of curriculum and teaching practice, with a mission to educate 1500 US Army Majors for uncertainty and complexity, while engaged in a global war on terror and in direct combat in Iraq and Afghanistan (Long, 2009). This provides an opportunity to examine reflective learners and practitioners in action (Schon, 1987) using mixed methods research and using multiple worldviews (Creswell, 2009; Creswell & Plano Clark, 2007).

Past Research on the Problem

Edmondson and McManus (2003) propose a structured approach to selecting research methods that fit the state of theory in a given field. Their 3 archetypes of the state of theory: nascent, intermediate and mature  are connected to qualitative, mixed methods, and quantitative research methods, resulting in an appropriate methodological fit that aims to meet the needs of researchers worldwide. Creswell(2009) offers a systematic approach to analyzing: researcher worldview, research purpose,  research questions, the state of theory, data collection, populations and situations to be studied, and data analysis in order to further refine the methodological fit  and better connect purpose with practice across all 3 methods. Creswell and Plano Clark (2007, p.8) offers a functional working description of the state of mixed methods research, which proceeds from a deep review of current field practice, establishes a superb framework for classifying current choices of mixed methods research design and the means by which methodological fit may be refined, but stops at the boundary of developing a deep theory of mixed methods of research.

Deficiencies in past research and Need for Mixed Methodology

Conventional professional education processes have been adapting at an increasingly frequent rate as a consequence of Army senior leader directives and direct field feedback. The adaptive processes and decisions to date have been single issue, single iteration problem solving exercises inherited from an environment in which incremental change was the norm and most appropriate. These processes are less and less suitable as rates of required change increase and the relevance of existing processes and curriculum are increasingly called into question (Long, 2009)

Audience

The audience for this research include: staff, faculty and students of CGSC; educators of military professional schools; curriculum developers in graduate schools and organizations engaged in preparing leaders for uncertainty; scholar-practitioners of mixed methods looking to adapt practical field methods of for mixing qualitative and quantitative data; scholars examining the deep theory of the methodology of mixed methods.

Purpose

Purpose of the study, and reasons for a mixed methods study

The purpose of this study is to examine the CGSC curriculum redesign project and the emerging feedback system that guides design decisionmaking, which incorporates both quantitative and qualitative data. The project can be described as intermediate theory in the Edmondson  and McManus ontology, and therefore suitable for mixed methods research since I am introducing an emerging concept (“Voice”), focusing on the exploration of theoretical propositions (the theory of mixed methods), the availability of sets of rich theory that inform the research (adult learning, decisionmaking, complexity, design, learning organizations, narrative inquiry and action research), and incorporating multiple data types and analysis (Edmondson& McManus, 2007, p.1165).

Research Questions and Hypotheses

Quantitative Questions

H₁: Student satisfaction measured on the Noel-Levin Adult Learner Satisfaction Survey is not different than their reported overall satisfaction

H₂: Student education priorities measured on the Noel-Levin Adult Learner Satisfaction Survey are not different than those of faculty and college leadership as measured on the same instruments

H₃: Student education priorities measured on the Noel-Levin Adult Learner Satisfaction Survey do not vary through time in the course of the academic year

H₄: Student education priorities measured on the Noel-Levin Adult Learner Satisfaction Survey do not vary after graduation and reassignment to field units

H₅: Student satisfaction measured on the Noel-Levin Adult Learner Satisfaction Survey do not vary from satisfaction as measured by existing CGSC Quality Assurance  surveys

Qualitative Questions

What are the dominant and subordinate narratives that emerge from focus group discussions on educational priorities and practice and environment within CGSC?

How does the curriculum design decisionmaking process respond to similarities and differences in narratives that emerge from groups of students, faculty and senior leaders?

Describe the development, emergence of the construct of “Voice” from the CGSC PAR cycles, and how this prototype construct is evolving and being applied by various sub-groups within and associated with CGSC, by applying various interpretive methods of the narrative inquiry tradition.

Mixed Methods Questions

1. To what extent are qualitative insights generated from PAR cycles, focus groups, and individual interviews supported by quantitative data generated from surveys and actual use data of digital communication and collaboration mediums?

2. How are various organizational narratives constructed by sub-groups within the CGSC curriculum design process in order to make sense of quantitative data?

3.  What insights are offered by the application of  various narrative inquiry traditions? Which traditions are favored or overlooked or rejected by curriculum design decisionmakers?

4. What happens within CGSC when students and faculty are given opportunities to exercise “Voice”?

Philosophical Foundations for Mixed Methods Research

Quantitative research literature review

Student satisfaction surveys built on consumer theory (Watkins, 2009) are broadly applied in colleges and universities, and  treat students as  free-willed individuals that choose between alternatives for an educational institution and particular fields of study. They are seen as rational actors with definite expectations about what they want in their educational experience, and that satisfaction occurs when their expectations are met or exceeded. Smart, Feldman and Ethington (2006) note a decline in the attention being paid to the attitudes and behaviors of faculties, administrators, and the college and university environments as contributors to student success.(p.2.). These insights are related to the “college impact” model of student success. Applying the Noel-Levitz  Adult Student Priorities Survey leverages a robust, nationally recognized, validated research instrument whose dimensions reflect the areas of importance emerging from the CGSC Participatory action research  (PAT) study (Long, 2009) and enables  quantitative research into the existing database of historical satisfaction measures currently applied in the college’s curriculum design process.

Qualitative research literature review

The James, Milenkiewicz and Buchnam (2008) application of Participatory Action Research (PAR) develops measureable action steps that can lead to  revolutionary transformations within educational institutions.  The use of measureable qualitative and quantitative data gives power and legitimacy to the insights it generates inside an organization that values rigor and validity, while respecting the intuitive insights of qualitative research.  Prasad describes many techniques of Narrative Inquiry that offer many techniques for interpreting and making sense of qualitative and quantitative data.  Reason & Bradbury, (2008) and Clandinin, (2007), describe these disciplines and crafts of action research and narrative inquiry as having a relatively mature foundation of theory and best practices, with enough variation between sub-disciplines as to create real and significant choices for researchers . Various methodologies in each discipline can be characterized according to their own logic that connects their particular world view (Creswell, 2009), ontology, research technique, data requirements, classification and analysis protocols, and strategies for sense-making of the results of inquiry. The combination of PAR  and narrative inquiry offer a robust set of strategies for generating insightful qualitative data with connections to quantitative data sets, which make them especially useful and practical to mixed methods researchers.

Mixed methods research literature review

Creswell and Plano Clark (2007) provide a broad yet detailed overview of the current state of the art of mixed methods research. They offer a working definition of the field derived from a survey of practice which proceeds from a deep understanding of high quality methods of practice, through choices of design and point to potentials  for the development of deep methodological theory. They offer mixed methods as an appropriate research strategy as a way to improve on the use of either qualitative or quantitative research alone.  In their view, mixed methods are more comprehensive, can answer more  types questions, encourages collaboration  and the deliberate incorporation of more than 1 worldview and is especially well suited for situations where practicality and pragmatism are prized (Creswell & Plano Clark 2007, p8-11).

Methods

A definition of mixed methods research

Creswell and Plano Clark (2007, p.5) define mixed methods research in the following way:

“As a method, it focuses on collecting, analyzing and mixing both quantitative and qualitative data in a single study or series of studies. Its central premise is that the use of quantitative and qualitative approaches in combination provides a better understanding of research problems than either approach alone”.

The type of design used and its definition

In this section I will briefly describe the theoretical shortcoming of treating mixed methods merely as a practical solution to improving upon either the qualitative or quantitative approach alone, and why a broader and deeper theory of mixed methods is appropriate for developing deep appreciation of complexity and uncertainty. I will briefly describe two different designs that would pass the test of the Creswell and Plano Clark ontology of mixed methods designs. With appropriate development, either would be approved for research within CGSC.

Comparison table  (adapted from Creswell & Plano Clark (2007)

	Design 1	Design 2
Choices	Explanatory	Exploratory
Theoretical Description	A 2 phase  design, where qual helps explain or build upon initial quan results (p.71)	2 phase design where qual results help develop or inform 2d phase quan inquiry
Description of application to CGSC	Round 1: the Noel-Levitz Adult Learner Satisfaction Survey is applied to a population, and results are tabulated, analyzed, compared against national  graduate student norms and in  a time series from the beginning, midpoint and endpoint of the academic year. Insights are developed   Round 2:  A series of focus groups and individual interviews are used to develop qualitatitive insights to  make sense of the quantitative findings	Round 1: a  set of participatory action research cycles identify areas of pressing concern to leaders, faculty and students within the college. A grounded theory is developed and constructs are defined by the community of practice, informed by theory from PAR outsiders.   Round 2: A quant survey is developed to explore deeply into issues and constructs developed by the PAR teams to cross check for validity, to confirm or deny, to support or modify the emerging grounded theory and provide the basis for future inquiries as selected by PAR teams. (Note: this is s summary of the actual process used at CGSC as the basis for this case study. The various tangents deriving from the initial rounds of inquiry generated my epistemological concerns with the pragmatic assumptions of mixed emthods)
Design notation	QUAN->QUAL	QUAL->QUAN
Justification	Needs qual to help explain significant, , non-significant, outlier or surprising quant results	Exploration is needed because:   1. no existing instrument 2. unknown variables 3. immature theory or framework Well suited for exploring a phenomenon or when researcher wants to generalize to other populations, test emerging theory or classifications (p. 75)
Variants	1. Follow-up explanations (quan results, insights need additional explanation)   2. Participant selection  (where a sampling of representative outliers are selected for follow-on inquiry)	1. instrument development model   2. taxonomy development model
Strengths	Straightforward implementation   Feasible for single researcher 2 section report of results Supports both single and multiple phase studies Appealing to quan researchers	1. easy to design, describe, implement and report   2. although initial emphasis is on qual, the quan phase makes it easier to appeal to quan audience 3. both variants supports multiphase studies well
Challenges	1. Time consuming   2. Decisions on which individuals to use by phase w/justification 3. Difficulties with IRBs 4. Deciding which results to explore 5. Specifying criteria for follow-on inquiry (before or after results?)	1. time consuming   2. difficult to specify phase 2 construct for IRB prior to phase 1 results 3. deciding up front which individuals to use in phase 2 4. which data to use in phase 2 instrument 5. deciding relevancy of phase 1 results for phase 2 taxonomy
Timing	2 phase sequential model	2 phase sequential model
Weighting	I think the QUAL(quan) model is more likely. This design relies on at least a mature enough state of theory to allow for initial quan inquiry, but we are more concerned with the interpretation and application of insights than in model or theoretical validation	the equal weighted choice is more logical; the desired outcome is an improvement to state of theory (quan) by either a better instrument or by an improved taxonomy (ontology). Yet the reliance on initial qual inquiry as a guide makes it at least co-equal to quan.
Mixing the data	Either Merging or Connecting is more likely than embedding.  Embedding implies a single phase, whereas this is defined as a 2phase design.  The improved explanation of initial quan findings is how the design could be “connected”. If the interpretation or meaning making is intended to create “rich description” then either variant of merging is logical	Connecting is by far the most logical design choice, as the 2 phases are explicitly linked; quan follows qual and the connection is either an instrument or a taxonomy.   There is a distinct “manufacturing or processing” aspect to this design, which does not seek to produce a rich description that is a blend, but rather produces a better quan framework as guided by the initial qual inquiry

Diagrams:

Research model 1: Explanatory:

Research model 2: Exploratory:

Analyzing the data:

In both models of mixed methods design the quantitative data would be subjected to power analysis, tests for relationship and causality.  The quantitative hypotheses are framed in the form of null hypotheses in order to determine if there were differences that could be attributed to a difference in instruments and what they are measuring (existing survey vs the Noel-Levitz survey); through time series tests to see if there is a treatment effect, and with the samples subjected to control variables to examine the effects of demographic, career experience, educational goals and faculty specific effects on the measures of satisfaction and importance.

Qualitative data would be subjected to thematic analysis according to the practices of the grounded theory, narrative inquiry and PAR traditions/  Narrative inquiry traditions are especially important here as outlined in Boje (2001).

Analyzing the mixed data would be drivcefn by the specific design selected as noted above.

Theoretical analysis of the consequences of choice in mixed methods design

Both designs would be interpretable as providing a deeper insight and understanding than a study restricted  to either of their individual qualitative or quantitative components. The functional definition of mixed methods (Creswell & Plano Clark, 2007, p.5) would lead decisionmakers, particularly of the pragmatic worldview, to ‘receive the wisdom of experts” and seek to straightforwardly apply the insights based on a justifiable belief that they now knew more about what was going on, and had in some fashion reduced the amount of uncertainty about the world around them. My central argument is that there are situations so complex and uncertain that no amount of research and conclusions drawn from best practices of the traditions of both quantitative and qualitative inquiry, and the best practices emerging from mixed methods as described by Creswell and Plano Clark. In fact my use of the word “situation” in the preceding sentence, is an implied assumption that there is such a thing in the real world as a definable “situation” or problem set  which may be bounded and contained by a problem solving, decisionmaking entity. While this construct is the basis for the post positivist tradition, which has endlessly proven its utility in countless settings, it is normal for pragmatists to conflate utility with reality.

It can be argued that there is could be a tacit agreement between constructionists and post-positivists to allow each other the primacy of method and interpretation based on typical problems, and indeed much work is being done to increase cross-discipline understanding, cooperation and integration. The common assumption between these two worldviews is that the end product of such effort is a measureable increase in our knowledge of the world as it is, from which we may exert more control and prediction by having reduced the amount of uncertainty by some amount. This tacit shared assumption I submit is expressed through the research practice of pragmatists who are “naturally” drawn to the mixed methods designs and practices described so well by Creswell and Plano Clark. Given the fertile and as yet only partially explored areas best suited for mixed method research it would be natural for the deeper philosophical theory or theories of mixed methods to be postponed, much as Smart, Feldman and Ethington (2006) found a willingness for researchers to revert to their preferred and more easily measured  research domains and begin to neglect the messy and challenging issues of environmental factors affecting student success. One is reminded of the story of the man who’d lost his key in a dark alley but was searching for it under the streetlight because the light was better there.

I am arguing that there are situations where even mixed methods are properly and rigorously applied, and interpreted in best professional practice, that the insights may serve only to help decision-makers appreciate the vastness of what they do not understand, and better act within an uncertain environment, humble in their ignorance, yet moved to action from values and on the basis of principles informed by the best practice of inquiry.

It is my contention that in those situations described so aptly as “wicked problems” by Rittel & Webber, (1973) that a  deep theory of mixed methods may be developed that is co-equal to that of qualitative and quantitative methods. I argue that mixed methods not only are useful in solving less-than-wicked problems, as described by Creswell and Plano Clark,  but most appropriate to engage with uncertainty and complexity for the express purpose of appreciating deeply the current situation. The deep theory of mixed methods I anticipate would require explicit inclusion of all 4 world views, since there is no a priori basis for excluding any of the 4. I thinkit quite likely that a reasonable assumption of a deep theory of mixed methods in fact could require an explicit inclusion of the best practices of each world view in some fashion, details to be determined, of course.

The shift in epistemological  perspective seems important to me,  and which should be developed in tandem to the directions for improvements in design and pure method described by Creswell and Plano Clark.  Checkland’s application of soft systems methodology,  artfully describes  “learning towards success” in a satisfying way (Checkland & Poulter, 2006).

The best expression of the theoretical stance towards irreducible complexity intersecting the human need for the state of nature or through any objective criteria (Boje, 2001) and in the work of Hayden White (1987) concerning the relationship between narrative discourse and the historical representation. These insights are causing me to reflect deeply on my own essentially pragmatic worldview and its underlying assumptions, and lead me inevitably back to the proposition that we need the methodological theory of mixed methods developed simultaneously with that of its design choices and specific methods.

References

Argyris, C.  (2008). Teaching smart people how to learn. Boston, Harvard Business Press.

Astin, A. (1999) Student involvement: A developmental theory for higher  education.  Journal of College Student Development (Sep/Oct, 19990 (Vol 40, No 5)

Beltyukova, S. & Fox, C. (2002) Student satisfaction as a measure of student development: Towards a universal metric.  Journal of College Student  Development (Mar/Apr 2002)

Checkland, P. & Poulter, J. (2006). Learning for action: A short definitive account of soft systems methodology and its use for practitioners, teachers, and students. West Sussex, England,  John Wiley & Sons Ltd.

Clark, M. & Rossiter, M.  (2008). Narrative learning in adulthood.  In S. Merriam (Ed) Third Update On Adult Learning Theory (pp 61-70). San Francisco,  Josey-Bass.

Creswell, J. (2009). Research design: Qualitative, quantitative and mixed methods approaches (Third ed.). Thousand Oaks, CA: Sage Publications, Inc.

Dorner, D. (1996). The logic of failure: Recognizing and avoiding error in complex situations. New York, Metropolitan Books.

Edmondson, A., and McManus, S. (2007). Methodological fit in management field research. Academy of Management review 32: 1159-1176

Fenwick, T.  (2008). Workplace learning: Emerging trends and new perspectives.  In S. Merriam (Ed) Third Update On Adult Learning Theory (pp 17-28). San Francisco,  Josey-Bass.

James, E. A., Milenkiewicz, M., Buchnam, A., (2008).  Participatory action research: Data driven decision making for school leadership. Thousand Oaks, CA: Sage Publications

Kearsley, G. (1997). The Virtual Professor: A Personal Case Study. Retrieved from http://home.sprynet.com/~gkearsley/virtual.htm

Lincoln, Y. (1983) Expectancy theory as a predictor of grade-point averages, satisfaction and participation in the college environment.  Annual meeting paper, Association for the Study of Higher Education, 1983.

Long, K. (2009). Participatory Action Research pilot study notes. Ft Leavenworth, KS: CGSC (unpublished).

March, J. (1994). A primer on decision making: How decisions happen. New York, The Free Press (Simon & Schuster Inc).

Prasad, P. (2005). Crafting qualitative research: Working in the postpositivist traditions. Armonk, New York, M. E. Sharpe.

Rittel, H., and M. Webber (1973). Dilemmas in a general theory of planning.   Policy Sciences, Vol. 4, pp 155-169.

Schon, D. (1987). Educating the reflective practitioner. San Francisco, Jossey-Bass.

Senge, P., McCabe,N., Lucas, T., & Kleiner, A. (2000). Schools that learn: A fifth discipline fieldbook for eductaors, parents and everyone who cares about education. New York, Doubleday.

Siemens, G., (2005). Connectivism: A learning theory for the digital age.  International Journal of Instructional Technology and Distance Learning 2 (10), 2005.

Smart, J., Feldman, K., & Ethington, C. (2006). Holland’s theory and patterns of college student success. Commissioned report for the national symposium on postsecondary student success: Spearheading a dialog on student success

Taylor, K. & Lamoreaux, A.  (2008). Teaching with the brain in mind.  In S. Merriam (Ed) Third Update On Adult Learning Theory (pp 49-60). San Francisco,  Josey-Bass.

TRADOC Pam 525-5-500. (2008).  Commander’s Appreciation and Campaign Design.

Trochim, W. Research methods knowledge base (2006). Types of designs.  Retrieved from http://www.socialresearchmethods.net/kb/desdes.php

Waldrop, M. (1992,2008). Complexity: The emerging science at the edge of order and chaos. New York, Simon & Schuster, Inc.

Watkins, T, (2009). Consumer theory in economics. San Jose State University Economics Department website. Retrieved from http://www.sjsu.edu/faculty/watkins/constheo.htm.

White, H.  (1987). The content of the form: narrative discourse and historical representation. Baltimore, The Johns Hopkins University Press.

Reflections on Validity in qualitative research

 

Was working on my final draft for my research proposal and had the following reflection about the issue of Validity in qualitative research. Since I am aiming for transformational changes in our strong military culture, and am using individual Voice (narratives, stories , interviews etc) as the basis for describing the current situation and desirable paths for change, the concern for Validity comes not from individual stories per se, but in the comparison between my entire dataset and the “Word as it really is” in the eyes of decision-makers.

My sense is also that the quality and insights that arise from individual narrative Voice will have an authenticity that must be considered valid as a data point. The validity concern I now perceive as being a function of having the narratives and stories cast broadly and fairly enough that they represent something that will be seen as truth. The threat to validity from individual narrative will come from assertions that the stories are not representative of the whole, and not from questions of individual perspectives, which will be honored as a matter of course.

In other words, a decision-maker who may not like the reports from the street may try to say

“…all the stories you report are “True” and valid, because they are perceptions, Mr Long, but you have not covered the whole topic/school/population to give a fair reading of the Real Truth, s we will persist in our ways. Thank you for your interest in national defense…”

So, now it is clear to me that I must look to answer the issues of Validity from a methodological and sampling perspective, and not just from the rigor of individual data point collection.

 

Reflections on Qualitative Research techniques: interviewing

Wow 1: Lofland and Lofland p.37.  “…It is precisely the “spy quality” of covert research in closed settings that raises questions about it propriety in social science”

 It strikes me that even if you take care to protect individuals by withholding their names, your results may end up introducing harm if the organization you are reporting to or for, takes actions to “get” the problem makers thru changes of policy.  These are the kinds of 2d and 3d order effects that you may end up subjecting subjects too even though you are hiding names and identifying features. I am thinking hard about the privacy rights of people in public places, wondering when the need to know and study outweighs the rights of privacy.

 

Lofland & Lofland, p.41 “…the ethical concerns engendered by covert research do not fully disappear with the decision to be a known investigator, but are merely muted….” 

I think that because they are muted, and therefore more easily overlooked, they are all the more important. It’s not enough to announce once at the beginning; you almost need to have visible “nametag” to remind people of your role, and your dual role if you are researching inside your own organization where you have other usual, normal roles in play. Its like you have to remember to keep reading people their rights, not just the first time, but in each session. 

Lofland and Lofland, Ch 3 Notes: 

Key takeaways:

1. Ethics of power relationships in the roles between investigator and subject. I am attracted to AR precisely because of the equality that is possible when subjects are acting as co-researchers.
2. Relationship characteristics drive ethical and power issues,
3. The ethical status of covert research: and 3 types; deceit by omission is crucial concept
   1. Public research at a distance:
   2. Quasi private: a good discussion of the effect of intent vs results that informs the ethics of being a hidden researcher. The example of the opportunistic researcher in the factory, conducting a study because he is already there to earn tuition vs that of the deliberate hidden researcher is instructive.
   3. Private; norm-fitting behavior that serves to “fit in” and simply to be a member of good standing take on an ethical quality of legitimizing the group norms simply to be able to study.  Has a moral quality to the decision.
4. Good section on ethic resources on page 39
5. Known investigator: sacrifices anonymity for public acknowledgement, at the risk of studying inauthentic behaviors.
6. The importance of social group connections and navigation to assist you in gaining access to key people, events and decision-making. If you have gained social trust you may be entrusted with insights into what are normally private considerations by groups and people.
7. Candid, brief, direct accounts of your research, with a view to explaining simply “why” they should participate, an explanation that avoids a dissertation of an answer.
8. Adopting a “learner” attitude is smart and often productive; should avoid “gaming” or being smug about it though.
9. The issue of respecting boundaries of the subjects and organizations being studied.  Can present some moral decisions of course.  A decision to halt the research because of behaviors you observe that you cannot tolerate will run into the ethical dilemma of non-disclosure that you may have negotiated initially.  The priest’s or lawyers dilemma ethics apply here.
10. Confidentiality issues are central when we are looking for transformational behavior changes. The Vidich case study is a good discussion of these implications on page 51.
11. have to learn to anticipate the kinds of findings that might put you into these decision spaces prior to negotiating for confidentiality and boundary agreements.

 Rubin & Rubin Ch6 Notes: The Responsive Interview as an Extended Conversation

1.      The differences between ordinary conversations and  responsive interviews:

     a.      Continuity over time

     b.      Focused thematic  attention

     c.       Explicit clarification of meaning and intent (spend more time in interviews making the tacit explicit than is usually the case in conversation)

     d.      Difference in purpose between narratives (construct of what happened) and stories (themed to make a point or represent a point of view)

2.      The interview is guided towards the purpose of the researcher; I would agree if you already have a specific informational goal in mid, but how much does this become confirming your own bias and not being open to where the story wants to go needs to go?  Is there a problem with confirmation bias in that observation from p.110?

3.      Guiding away from formal answers through artful question design, although you do have at least a semi formal relationship at work as a researcher. Is this disingenuous?

4.      I like the metaphor as a series of linked stages to structure the interview

5.      The formulas for establishing rapport are at the same time helpful advice, but ironic in that they are rules for being spontaneous and “real”

6.      Easy questions, tough questions, concluding questions, with management of emotional states, and ending with open ended contact options

7.      Good discussion of evaluating the interview as a way to study your own process as well as examining the data itself.  Could have had more advice on how to rehearse the interview ahead of time and having a purpose statement to keep you focused during the process. Ie, “What I must get from this interview is….” 

Lofland, J., David, S., Anderson, L., Lofland, L. (2006). Analyzing Social Settings A guide to qualitative observation and analysis (Fourth ed.). Belmont, Ca: Thomson.

Rubin, H., Rubin, I. (2005). Qualitative interviewing: The art of hearing data (Second ed.). Thousand Oaks: SAGE Publications. 

4 Reflections on Design

4 Reflections on “Design” that are guiding my inquiry, inspired by thought provoking article and commentary at the School for Advanced Military Studies, which is engaged in deep think on the entire military decision making model: 

1.  The challenge of codifying artful design. I am trying to integrate these 2 statements from the article; It may simply be that we want to get the principles right as opposed to creating a checklist, template, and standard process for Good Design:   

“The Art of Design is a cognitive paradigm shift from a 20^thCentury doctrinal model that emphasized check-lists, templates and processes.” 

“Our collective goal is to codify these practices for doctrine.” 

2.  Multiple ways to Design? 

“The Design approach makes explicit what Commanders have been doing intuitively in combat as they learn from acting within complex and complicated systems.  Our collective goal is to codify these practices for doctrine.” 

We have accepted within the college that there are multiple learning styles. Action Research is a tradition of inquiry associated with discovery learning in cycles of “plan-act-observe-reflect” and which appreciates multiple ways of knowing (experiential, presentational, propositional, practical).  If the essence of the Commander’s art that we want to codify in Design  is ‘learning from action’, is there a sense that there may be more than one way to design? Particularly in light of later statements that: 

“Because conflicts are unique, they require individually tailored solutions. There is no recipe or formula to choose between competing solutions for resolving this tension, because resolution depends upon the creative application of military judgment.” 

3. The relationship between Design and the current body of knowledge of Operational Art 

 “Design provides the logical connection between strategic ends and tactical means that is the foundation for adaptive action in the face of novelty and complexity.”  

• Is Design taking the place of what we called operational art?
• Or the operational level of war which connected strategic goals to tactical means?
• If so, is it replacing the construct of operational art?
• Or being offered as an alternative?
• Or to be used in lieu of current operational planning when complexity and uncertainty make current planning models unsatisfactory?
• Is “logical connection” the right characterization of the output of design? What would “artful” or “hypothetical” or “tentative” do to the meaning of that sentence and our appreciation of what Design is engaging with? 

4. Certainty in Framing so we can proceed to Planning?

 “Creation of an adequate Systems Frame allows the commander and his design group to cognitively map their environment.  They create a relevant Problem Statement and Theory of Action for moving an unsatisfactory state of affairs to a manageable condition that is within tolerance of the political leadership or of a senior military commander.” 

I had a vision of a Commander and staff  working on their systems frame until they judge that they have a workable model that accommodates multiple points of view. I asked myself: “I wonder how they will know that this systems frame is finished? Or good enough? Or relevant? Or satisfying to stakeholders? Or representative of the current situation? Or offers enough insights to begin taking actions? 

Could this be construed as putting the cart before the horse? On what basis will the Commander’s and staff judge that their map is sufficiently like the territory that they can proceed? That their problem statement is relevant? That their Theory of Action provides the basis for informed action? That their plans will have a causal connection to a change in the state of the mess they propose to manage? 

My sense is that a situation that calls for design will not let you judge the efficacy of your design without many and continuous iterations of experimenting , and that your design will become an evolving set of continuous approximations that are informed by feedback. And because we know that the environment is dynamic, it will learn from (or at least react to) our iterative experiments, there is every reason to believe that we will not be able to “Design Once, Plan Many”, but engage in an open ended  of iterative Design/Planning cycles that are integrated and continuous; that these Design/Plan are not separable, until we have evidence that the situation has become stable enough that our existing planning doctrine is sufficient for ongoing operations. 

If this line of inquiry has merit, I would be looking to see the sense of open-ended action research cycles of “Plan-Act-Observe-Reflect”, with conclusions about our new unsderstandings, theories and concepts evolving through each cycle, with an adequate Systems Framework  and Theory of Action emerging only after much hypothesis testing in the environment we propose to manage. The very uniqueness of the situation that demands we respond with Design is unlikely to become logical and understood at the front end of  Design/Plan campaign. 

Is there such a thing, then as The Design, or only continuous “Designing” to inform planning, and that we discover that we no longer need to change the design when the environment no longer demands it but not before? And then, only for as long as the environment remains in an acceptable range of  dynamic equilibrium? 

Imagine we are at the end of a successful campaign and look back at our path.  Would we be surprised if we had a series of “designs along the way”, each good enough and relevant enough for action based on previous feedback and our then-current level of appreciation validated by stakeholders, and that as we acted/observed/reflected we gained a new appreciation of how the situation was also evolving around us, so that we had a sufficiency for action, without ever having a complete Truth, and that the working model of the world that informed our design was unique and quite different, yet strung together by our reflection and learning until we reached a state where we had something like an acceptable, stable-enough reality. 

At what point did we “know” we had it right? That we had made a “logical” connection between goals and means? That we NOW understood what was going on in the environment? My sense is that those only become clear in hindsight when we construct the narrative of the “secrets of our success” and convert the winning narrative into our next template, until the world decides to tip over our applecart. 

The commentary above about humility and learning (and we can learn a lot of things without getting closer to the Truth) is inspiring and challenging. We may be in the situation Argyris and Schoen (1974) described as “Few espoused theories but effective theories in use” in Theory In Practice: Increasing Professional Effectiveness.

Cultural context in qualitative research: can it even be done?

  A “critical friend” in the action research process is a trusted agent who gives deep insights from the outside into the nature, quality, and path of  your introspection. They help keep you grounded, and offer triangulation points in the sense-making process as you grapple with your own questions and insights. They act as sounding boards and mirrors, acknowledging that we can’t really ever know in the scientific sense about the truth of our propositions when dealing with complex human terrain. However we can pursue quality and appreciation instead, and that’s whata  critical friend’s role is.   

A good friend, acting as one of my  ”critical friends” in my 1st person action research process offered the following for me to consider after a journal entry concerning a confrontation I had over curriculum transformation with a peer.

Ken, this journaling effort seems to carry itself in your research.  The fact that I have a military background provided me with certain comforts in reading this, but for those without the cultural insights…they may be a little lost.  If you approach your writing with a bit more cultural sensitivity, or as a “foreigner” your words of wisdom would reach a wider audience.  Military culture is very specific in communication styles, so standing in a circle outside the backdoor ***ing about directives and expectations is very NORMAL to soldiers, of which I contribute to trust among the brotherhood, but please know that many organizational structures do not support this special sharing and feeling process among its members.  You identified what is normal for a DC, so from that baseline you should be able to point to cultural inconsistencies which are now creating ripples.

 

I really like this 1st person reflection, but I feel a sprinkle of cultural unraveling/description is necessary to fully understand the phenomena. 

I replied:

I think you are exactly right Jeff.  The deeper into a subculture we go, the more unpacking and “scaffolding” we need to provide a framework of meaning for outside readers. I was thinking of a translation of a Chinese classic novel I have started to look at. It has 45 pages of dramatis personae before the opening scene and covers many generations of the family that is the central to the plot and narrative. I am exhausted by that already   I just can’t upload that much into short term memory and hold it there to make an informed reading. So I am having to nibble on it as I go. 

I wonder how much like that the study of other cultures are for researchers? The ethnography chapter describes that dilemma: the knowability of other cultures, no matter how immersive you become. Could Jane Goodall every really be “of” the apes she studied?  The French sociologist Francois Jullien wrote with a certain despair of trying to fully grasp for himself the subtle elements of Chinese thought  in his book “In Praise Of Blandness’, despite a life of scholarship. Then he considered just how hard it is, if possible at all, to communicate the essence to another culture, which he was no longer fully a part of precisely because of his immersion in the Chinese culture. He was seeing himself in a no-man’s land of “between” 2 cultures.

In another sense that’s kind of where we all are: between our sense of self and our sense of the dominant culture and other subcultures in the soup we swim around in.

I say all that simply to say that what i am trying to do on my journaling is to capture the moment as quickly and deeply as I can, without over-thinking it and the first order immediate reflection, in order to build up some snapshots in time of my own role inside of my research. Dr Alana has recommended some qualitative analysis software called Atlas/ti that should assist me in identifying and linking narrative themes connecting the snapshots in time.  I am excited to see where it goes.

It is clear to me that I need to provide an intermediate layer of context around the raw entries in order to improve the accessibility to key points and themes as they emerge.

Jullien, F. (2004) In praise of blandness: Proceeding from Chinese thought and aesthetics. 169 pages. Zone Books, New York.

 

Blogs and democratization within hierarchical organizations

an update on “giving voice to important matters”

after we briefed him on the power of blogging, our commandant, the senior military officer in charge of the college has embraced the idea and has started asking provocative questions about the future of the college’s curriculum and modes of instruction.  Students and faculty are starting to give voice to their concerns and ideas, and it seems to me we have passed the tipping point, The culture is changing around the former centers of power and there is a new spirit of democratization in the air.

I have appended an example of one of his questions, extracted my commentary from the discussion thread, and then provided a 1st person AR reflection as part of my self examination which i share with critical others as part of my efforts at transparency. I include some comments on “playing multiple roles” as part of my reflection in the practice of dramaturgy, one of the qualitative methods we are looking at in MGM815 with Dr Wishart.  Coughlan & Brannick discuss “roles” in their excellent book “Doing Action Research Inside Your Own Organization”.

this blog thread comes from the college’s top level blog. meanwhile my departmental blog, started  last term, continues to grow in content and usefulness as documented in the chart at the end.

 

 

PAR Journal entry 2009-02-05

Subject: giving voice to the future of education.  Flexibility vs Standardization 

The thread was begun by our deputy commandant, the senior active duty military officer in charge of the college. I have extracted out my commentary from the discussion thread and provided a reflection below

=================================== 

Is it possible to address contemporary, real-world problems while still upholding our learning requirements? Is it possible to address contemporary, real-world problems while still upholding our learning requirements?  Let me give you an example. 

From November 2008 to January 2009, the Command and General Staff College conducted a pilot special research study with two staff groups in the 09-01 ILE course to analyze the impact of the “surge in Afghanistan.”  The plan was to study the time period from December 2008 to August 2010, a critical time for operations in Afghanistan, in part because of the election of a new administration and the subsequent shift in focus for the United States military from operations in Iraq to operations in Afghanistan. In this 18-20 month time frame, a critical planning factor was to show “discernible progress” in the security of Afghanistan. 

This pilot study was intended as an alternative approach to achieving the same purpose, outcomes, and learning objectives as the common curriculum AOWC 1 block.  The study focused on the intangible aspects of battle command — understanding and visualizing – using current and emerging doctrinal concepts.  Additionally, the pilot study targeted the following objectives: pilot the use of real world topics to meet learning objectives in the CGSC classrooms; provide research products to the operational force through specialized studies to enhance CAC as the “Intellectual Center of the Army;” shift focus from Iraq to Afghanistan within the leader development community; conduct parallel strategic engagement opportunities.

 Among both our faculty and students this study demonstrated the potential to break traditional educational paradigms and explore progressive methods.  This is in line with a shift in CGSC to refocus on the leader development in ways that will be vital to winning this war…and the next one. We are always looking for new and unique ways to educate our students with practical, contemporary issues.  The pilot study demonstrates what is possible – is this methodology appropriate for our Field Grade education? 

My response: 

Can “manufactured” scenarios against notional opposing forces prepare professionals for real demands? yes. 

The Great Krasnovian Wars, fought in CBS, on the plains of Kansas and Nebraska, prepared officers for challenges of planning, synchronizing and executing complex conventional operations which contributed in some fashion to victories in Desert Storms and OIFs. 

What were the equivalent real world challenges we’d have been working on instead of the Krasnovians? I suppose how to conduct small scale humanitarian aid interventions. 

Exclusive focus on the immediate problem set would have left the Army less ready to adapt to the discontinuous challenge of large formation conventional operations. I think that remains true now as it was true 10 years ago, 100 years ago. 

Because we can’t predict the future with certainty, and time is our limiting factor, we cant afford to pick a single strategy and bet the farm on it. I think our conception of Full Spectrum Ops is sufficiently complex and challenging enough that we need to be educating adaptive leaders to perform in all of the dimensions, with more focus on the leader qualities and skill sets, and less on content-centric curriculum. 

If you believe in the description of future dynamic uncertainty found in the Capstone Concept for Joint Operations, Commander’s Appreciation and Campaign Design, the TRADOC concept for the Human Dimension in 2014-2025, in FM 3-0, and in the importance placed on Commander’s judgment expressed in FM 7-0 

If there are real world problem sets available that allow us to get to the educational outcomes of producing agile, adaptive, educated leaders, then by all means we should take advantage of the situation, provided it supports our design goal of having the officers experience problem framing, problem solving, decisionmaking, leading and managing at multiple echelons, with multiple mission profiles, with sufficient complexity to challenge their belief in school solutions, and enough human terrain variables to keep culture, media and world opinion firmly in mind. 

That said, every time we incorporate real world problems in lieu of standard curriculum,, we’ll need to be able to rapidly craft a study proposal that can be compared to the design criteria of our student education outcomes, tailor an assessment and evaluation plan that meets the standards of accreditation and our Accountable Instruction System. 

 the infrequency with which we actually leverage real world scenarios suggests this will be quite a stretch for our culture and routine processes. There will be tension between the desire to incorporate the newest real world scenario, and the standardization, uniformity and stability that AIS values. 

We’ll have to let go of the comfort of a stabilized curriculum, that seeks minimal changes year to year, of year over year trend analysis. We’ll need to do it in AOWC, because that is where we do much more application of theory to practice, whereas Common Core emphasizes the basic cognitive and affective skills that are broadly needed in the force (probably our CMETL). 

We’ll have to accept variation in content and delivery across the multiple settings where our course is delivered. A shift to student-centric education outcomes, away from content based standards of performance will help, but not completely solve this problem. We’ll have to have a rapid prototyping and approval process for proposed studies that can routinely make the assessments as to the business-case merits of research proposals, so that we can maintain our commitment to evidence based educational assessments which represents our institutional committment to intellectual excellence. 

At CGSC, we are precisely at the intersection of theory and practice, with a requirement to make sure that our doctrine is communicated and exercised in the classroom, yet acknowledging the primacy of the practical and immediate lessons we are learning in the war we are fighting to win right now. i think the risk is well worth taking. 

Ask yourself when was the last time we were prepared, as a faculty, to be genuinely surprised and delighted with the results of student inquiry into a wicked problem set. When was the last time you went into the classroom excited about the uncertainty of the direction our investigation into our craft and profession might lead us? When was the last time we modeled the kind of adaptive ingenuity, innovation and measured risk taking we assert we are educating our officers for? 

At times it seems we have carefully scripted the curriculum to beat innovation into submission. When was the last time we had the time and flexibility to carefully examine the results of our first round of inquiry with our officers and collectively assess the results, and then decide where the next round of inquiry should take us? This kind of living action-research, is risky, the results aren’t preordained, but we have good processes and passionate, committed leaders and I am confident we can do a lot more in this area than we currently do. 

a follow-up example of the kind of analysis i described above.

In the W100 block the hour operational logistics lesson is designed for every officer to get to the apply level  of of the sustainment warfighting function. To assess that, we designed the lesson  so that each officer participates in log prep of the theater, does individual deep analysis on a commodity or service area, participates as a member of the staff to complete and brief a concept of support, and then write an individual logistics estimate (employing materials contributed to the group effort by others).  This lesson is designed to address a persistent educational gap in the field: that commanders have not always appreciated the effect of sustainment on their vision and plan, and that staff officers need a better personal understanding of logistics basics in order to be more effective as a member of the team in any capacity.

The analysis and decisionmaking that allocated 16 precious hours to these educational outcomes and to perform them at the apply level was non-trivial.

I believe that a study of operational problems in Afghanistan can meet every design goal of the existing curriculum. I know that we know a lot more about the theater in GAAT than we do in Afghanistan, simply from the knowledge that is created through hundreds of staff groups examining GAAT over multiple years and the deliberate analysis and research we do to improve the quality of the scenario materials every year. I also know that it is not the complexity and level of detail we get to that matters in this lesson, it is the process we follow, and the questions we ask, and the answers we tentatively form in the time available.

If we are professionally sure that we have a good set of education outcomes, then when examining a proposal to do a study into a real world problem set, the crucial question becomes: how do we ensure the outcomes are achieved in this new context, and if there is something we dont get to, what gets cut? This should be a routine professional judgment call, which gets documented and accompanied by evidence after the execution phase in order to learn and grow for the next round

My reflection on action: 

The posting above will be read by several hundred high ranking people in our college. It represents the kind of discussion which until this year and this doctoral program, I would make verbally to my peers or in meetings where there was little likelihood of bveing heard. It was a safe existence but frustrating in many ways.

I confess to feelings of apprehension mixed with excitement as I hit send, knowing that once posted, the words will stand on their own merits and my standing in the college will undergo some change, in unpredictable ways. I enjoy the risk and appreciate the new opportunity to be heard directly. 

In terms of action research and dramaturgy, this particular Q&A thread has many touch points. I am voicing a role as a professional curriculum developer for a particular department. My example of how important it is to analyze the standard curriculum for the real world study that replaced it could be seen as an indictment of the Afghanistan study as executed, because they blew off the logistics requirements that are an important part of the standard curriculum. There is no doubt their study was valuable, but logistics instruction was sacrificed without a hearing.  There was no due process. 

I am also speaking as a change agent in this thread, independent of my “hired job”.  I am on the side of those who are pushing for more change, not less, in the curriculum, and because this represents a challenge to established authority and more work for the faculty to adapt, it is not popular. There will come a decision point and I will support the decision even if I don’t like the outcome, because due process will have been followed  to my satisfaction because I am now certain that my voice has been heard by those charged with the decision, and I trust in their personal integrity and judgment. 

I will end up playing the role of advocate and supporter of the final decision when it comes to preparing the department and faculty to  implement. I will also maintain my role as change agent in the next rounds of change 

Until the advent of the blogs on our college homepage, it is also the kind of dialogue which would never have a practical chance of being heard given the real constraints on time and the formality of a command hierarchy for decision making. Only the dominant narratives survive to make it to the top level decision makers, except under exceptional leaders. 

The infrastructure of the blog, and the willingness of the current crop of leaders to engage personally and professionally in bloggery prior to decision making, is a democratization and a loosening of the formal process, one that is changing the nature of the culture in significant ways. 

PAR Journal entry: initiating the “Force generation” project

this is the text of an info paper I wrote for our Deputy Commandant to send to the 3 and 4 star generals in charge of Army leader education and training. It describes the motivation for and purpose of our inquiry into “the education gap of force generation”, and lays out a methodology and strategy for accomplishing the mission. Upon close examination, you would recognize all the elements of Participatory Action Research and a collaborative inquiry in the action research tradition. It’s easier to get the project moving and going thru the PAR and AR steps without getting bogged down in the “naming”, so I intend to get irresistable momentum built and introduce the concepts as required and as we go.

Force Generation Curriculum Project:

 

Educational gap: based on feedback from the DC, the field, and  our current students and faculty, we have identified “Leading Army units through the Force Generation Process to build ready units” as one of our most important educational gaps in the field.

 

Results of initial curriculum analysis:

1.       All departments teach on some aspects of the Force Generation process, but we need to do more in AOWC to get our officers to the “Apply” level of learning.

2.       Existing curriculum can  be integrated and leveraged for part of the solution, but we need more attention to and information on the Reset phase, and on integrating the insights of supporting organizations.

3.       A Terminal Learning Objective  for AOWC that incorporates “Force Generation”, that is developed through Enabling Learning Objectives and the appropriate standards which includes insights from all departments, fully integrated through the ongoing AOWC Working Group will lead us to an integrated solution, and follow our own Accountable Instructional System (AIS)

 

Force Generation. Research Questions:

What we want to learn:

by focusing on the specific topic of “force generation” within the CGSC curriculum, we want to answer the following questions:

Our top level questions are:
1. What do Army Majors (and sister service equivalents) need to know about Force Generation?
2. What are the qualities of the educational environment that will best support their learning? translation: How can we best create the conditions for learning? How sho9uld we teach it?))

Our supporting questions: (these are what we are asking of all the stakeholders to help construct our group knowledge)
1. Who are you and what do you know about Force Generation?
2. What does your organization do in Force Generation?
3. What do you think Majors need to know?
4. What can you contribute to our knowledge base?
5. What can majors do with your knowledge?
6. What are your questions about force generation? (we want everyone to learn)
7. What would you do with the answers and why is it important?
8. Who else should be part of our team?

Initial stakeholder list:

 

 

 

·         Students

·         DLRO faculty

·         CD & Faculty of other departments

·         Army staff: G1, G3, G4, G8, ACSIM

·         AMC

·         HRC

·         JFCOM

·         FORSCOM

·         CASCOM

·         CALL

·         BCTP

·         1/1 HBCT (partnership with field unit entering reset & train/ready pools)

·         Senior Warrant Officers at proponent schools

·         IT support staff (for digital learning environments)

·         BCKS forums (watering holes for interesting ARFORGEN conversations)

 

Methodology:  Co-operative inquiry  with all stakeholders in the Force Generation process, focusing on support to the education of field grade officers attending CGSC

 

Initial actions (already underway):

1.       Establish POC at each stakeholder and begin staff work to answer initial supporting questions

2.        Announce to all stakeholders we will have a 3 day workshop in May at Ft Leavenworth to review all supporting question inputs and then craft an answer to the top level questions

3.       Begin planning for said conference

4.       Identify wiki solution for  an adaptive knowledge base that will support ongoing curriculum and reach back capability for FORGEN (done: Blackboard, in the reachback area)

5.       Integrate concept of threaded Force Gen lesson theme into AOWC WG process with 1^st order estimate of hours required (done: 10)

6.       Plan for an AAP to catch any spillover from conference and this inquiry that will not fit within AOWC curriculum.

7.       Develop project management framework to monitor progress and include in DLRO SigActs routinely (LTC Judy, LTC Hart)

8.       Draft DC info summary describing FORGEN initiative (Mr Long)

 

Expected outcomes:

1.       Quality staff recommendation for AOWG FORGEN curriculum and an AAP for “spillover” material

2.       An  interdepartmental “application” level curriculum fully integrated through the AOWC Working Group process

3.       An inquiry- based research process that models the principles of group-learning in real time, which can serve as an example for other inquiries in important topics

4.       An infrastructure that supports student learning while in attendance, and after graduation as a reachback

5.       A knowledge base that focuses on support of our student and faculty population, and wehich synthesizes quality research & knowledge from Army staff and support organizations

6.       A documented staff process that will support our ongoing accreditation and scholarship standards