ASSESSING
TO LEARN - LEARNING TO ASSESS:
THE
FIRST TEN MONTHS
DECEMBER
1998
Stella Humphries
INTRODUCTION
The Assessment Initiative is a work in progress. This is a first attempt to pull together and make meaning of the many conversations I've had on "assessment for learning" both within and outside the SoL community since the Assessment Research Forum of January 1998 (See "Assessing to Learn -- Learning to Assess" on the SoL website at http://www.SoLonline.org/co m/AR98/index.html
The January Forum set the tone and gave initial direction to the conceptualization of assessment as a means for enhancing learning within organizations. Since that time I have engaged with companies, researchers and consultants to explore the assessment-evaluation domain and to learn what projects are in progress or would be useful to create.
I'd like to first give a snapshot of the challenges as I see them now - given that we are in a changing landscape, with often difficult conceptual terrain. I will then give an outline of how projects have begun within this Initiative, which I will follow with a brief description of those underway or proposed. Finally and most importantly - a request for your input on going forward.
THE CHALLENGES
The nexus between evaluation and assessment
At the January Forum, the distinction between assessment as an evaluative process and assessment as a reflective process became a focal theme of conversation. The implications are still reverberating. At the January Forum we [1]came to define "evaluation" as a third party judgement on the worth of others' performance, project, program, etc. Consequently, "evaluation" is more about "success" or "failure" according to criteria external to the learner. Since inherent to being judged is fear of failure, third party evaluation tends to elicit learning for compliance and inhibits generative, self-sustaining learning. Evaluation for accountability, as it has been conventionally practiced, is therefore likely to be a powerful force that counters the intentions of organizational learning initiatives. The predominant feeling in January was to focus on assessment for learning to:
- build our own learning about organizational learning - what changes as an organization becomes a learning community - and to articulate these changes to others who have not experienced the process directly.
- develop the framework, tools and procedures for self-assessment - how do we systematically learn about our learning and know do know we are making progress?
Despite reluctance to have evaluation be a driving force for research within SoL, evaluation continues to re-appear on the agenda in different ways. First, there is more or less pervasive pressure from executive management to provide "proof" of value to business: "what is the value of organizational learning to the company? " Second, some practitioners have asked: "Having sent so many people to course A - how do we know it has been effective?" Third, how do we, as a community hold ourselves accountable for the quality, effectiveness and safety of the capacity building processes and products we provide and use? Where are the checks and balances for ensuring what is safe as well as what is effective? Who are the gatekeepers?
The questions arise - when is evaluation for accountability "legitimate" in a learning context? Can it be done in a way that does not inhibit learning?
Elucidating the nexus between assessment for learning and evaluation for accountability is an ongoing process. Assessment for learning and evaluation for accountability are on a continuum and the boundary is not clear. The polar ends are distinguishable, but in a large intervening area, the distinction is blurred. In part this is due to the fact that being accountable is the other side of having freedom. Thus evaluation has a place in a learning community. But what is the nature of such evaluation? Whose responsibility is it to evaluate and how can it be done? We know that in practice, evaluation by others raises fears. Particularly if the others are outside the established work relationships within the group. Unless there is a foundation of trust within the group, a fabric of resilient relationships and cohesiveness of understanding of what is needed and why - the fear will continue.
Most SoL member organizations are either heterogeneous with respect to or are in transition between "command and control" management and more "self-directed" management. Hence creating and sustaining conditions for building the needed levels of trust in the relationships is usually difficult. Different parts of an organization are not always aligned in their commitment to or understanding of learning principles so one part may ask for an evaluation of another. Also, those who ask have typically not participated in and experienced becoming a learning community so have not felt the changes it brings. In such situations the evaluation is, of necessity, an abstracted set of indices and measures which are a substitute for the experience itself. As much of the changes are "intangible" the evaluation cannot reflect the subtlety of what has occurred. For example, new insights, new levels of relationships with co-workers, more resilient networks, broadened changes of reference, changed self-perception, increased creativity and risk taking, etc. What develops is a credibility gap - those who want to know have not had an experience of the changes and the "measures" which they are given are inherently inadequate to capture many of the "softer" changes in the learning domain. Many of the changes, moreover, are gradual and intuitive and participants themselves are not always conscious of their transformation. Another difficulty is that where one or two people undergo a change process, their efforts are not supported within the culture so they may be subsumed.
Heterogeneous levels of commitment within an organization thus may cut across efforts by a sub-group to implement learning practices. Use of top-down, abstract evaluation measures may be not only inhibitory to learning but they are likely to give an inaccurate or incomplete picture of what is occurring.
When the whole of the organization is, for practical purposes, aligned, the issue of upper management asking for an evaluation as an abstracted activity is unlikely to arise. That is, where assessment is integrated into the ongoing procedures of an organizations activities, it becomes a navigation tool for self-correction - for making continuous adjustments. I came across a corporate example of a feedback link between ongoing, systemic assessment and continuously self-correcting management. I met with Berth Jonsson and Mikael Paltschik of SMG Inc., a Swedish company that specializes in the management of intangible assets. They referred to the "third wave" [2]of management in which there is a continuous assessment and adjustment of the system using various lenses such as instruments, interviews, surveys, dialogues etc. to reflect, monitor and understand what a whole company and its components are doing. Here assessment is intimately interwoven with management throughout an organization. Such integration is made possible when alignment, trust and mutuality have been built over a long time. The social relationships are then resilient as illustrated by the following example. One of the instruments they used with a Swedish bank was a scattergram showing each branch in relation to its score on a "customer satisfaction" survey and an "employee satisfaction" survey. Clearly visible were those that ranked high on both and those that ranked low on both. This could be a devastating "evaluation". However, the point, according to Berth and Mikael is that this assessment was seen as a valuable tool for improvement and for collaboration in sharing what works. The level of trust was high enough for the assessment to be experienced as helpful rather than threatening. This condition was made possible because the "tools" of assessment were regarded merely as a vehicle for conversation. The tools were not ends in themselves, they were not abstracted from but embedded in the social-matrix. Moreover, assessment was based around three "bottom lines" that had equal importance: customer satisfaction, employee satisfaction and profitability. Profitability was not the only measure of "success".
Although I did not experience first hand what occurred in the Swedish bank - it does seem from the reports of Berth and Mikael, that the ways in which the consultants interacted with the people in the implementation and interpretation of the instruments is at the heart of the difference between learning from them and having them be external measures of success or failure.
This story and analogous ones lead me to emphasize that the difference between assessment and evaluation is less in the method and more in the intention. That is, is it intended to support the learning of the employees in a genuine way? Is the process integrated into the ongoing work? Does it engage the people themselves to self- observe? Is it contained within robust social relationships? Generating operational systems such as the one at the Swedish Bank that integrate work practices with interpersonal dynamics and with supporting principles of management requires commitment, time and conscious intention. To have a system that operates in this way also requires that procedures of self- assessment are seamlessly built-in in order for ongoing self- correction to take place.
At this juncture the "command and control ethos" of management and "learning community" ethos meet -- uncomfortably. The question is how to begin moving toward an integrated multiple bottom line/means oriented focus from a single bottom-line/results oriented focus. Moving from external evaluation for accountability to self-assessment for ongoing learning is more than a decision - it requires providing a context and continuously navigating the polarized force fields as they play out in local circumstances. Among the challenges is to begin to articulate the value, conditions, outcomes and procedures of assessment as a tool for individual and collective learning. Another is developing principles and a conceptual framework for designing and applying assessment-evaluation processes to serve a collective learning purpose. Yet another is to develop procedures and tools for ongoing self- observation. That is, to develop assessment as a process; a process integrated into the operations and workplace culture in a way that supports self-sustained learning and self-correction.
The new discipline of assessment for learning
The questions on the value of organizational learning from those outside the community are serving to focus attention more sharply on the internal need to reflect on our learning. We are now asking ourselves: what does change as a result of introducing principles and practices of learning in community? How do we know we are making progress towards our learning goals?
Building assessment capacity as a deliberate and systematic practice for learning is essentially developing a new discipline. A new discipline of assessment that would support organizational learning as a journey and not a destination. As such, assessment would inherently become a vehicle to sustain and diffuse learning.
As already mentioned, a critical aspect to assessment is to begin the shift from a results-oriented focus for assessment towards a work practices orientation. This has methodological and cultural ramifications. Most of what we ordinarily notice remains tacit, as current work environments are not focused on informed observation of the every-day actions and interactions of how work is done. (see also Appendix E - Tom Johnson's article) . Neither does the work culture support the development of awareness around the quality of relationships we have with each other. There is little appreciation of the fact that interpersonal relationships are the very foundation on which operational connections are made. How, then, do we develop a capacity to focus attention on what has leverage in the relational and the operational domains of the everyday practices of the workplace?
At its simplest, this requires time set aside regularly for reflection on the individual and group levels. At its most developed, it requires systemic changes in principles, practices and culture.
I had the good fortune to visit Creswell School District in Oregon on the invitation of research member Dennis Sandow and Jim Ford the Superintendent. They have created a monthly District Improvement Day ( DID days) during which teachers and staff spend the day reflecting on what is working, what isn't, what needs to be done, who can help. My observation over the course of two days was of a social network comprised of high quality personal relationships exhibiting creativity, trust and commitment. I now know that such a learning community can exist. It has for me become a reference point for what is possible. Watching the interactions and hearing the stories, I also know that it was carefully built, one person at a time, moment by moment over sustained effort of five years against ongoing opposition from State Authorities and entrenched cultural norms. I also know that the student achievement results have been steadily climbing over that time without achievement being the focus of attention.
How do we begin cultivating work environments and cultures that value cultivating relationships and value reflection? Particularly in the present when taking time out to reflect, to be with, is counter-cultural.
Again and again I hear: How do we "capture" the changes, much of which are intangible? How do we describe and articulate personal learning in meaningful ways? Because so much of the change comes from shifts in the quality of our relationships with each other, the very act of trying to describe and finding language to describe the changes can be a source of problems. In my recent discussions with Etienne Wenger, it became clear that describing and paying attention to something is a double edged sword. On the one hand we can become more aware of it and thus can "grow" it in our culture. On the other, as we try to analyze what has changed in an attempt to define it in order to rationalize resources to sustain it, we can too easily lose the living essence of the change. We begin to concretize it and it becomes reified. That is, we abstract and externalize it so it loses its meaning in our daily lives. A mundane example is asking "how are you?" No-one expects an authentic answer because it is no longer an authentic question - it has become a formality and a surrogate for genuine connection. So the challenge is not only to capture for ourselves what is changing but also not to lose the meaning of the words as we discover them. This calls on us to stay present to the value of the living moment - to continually renew the experience of the "now".
My experience at Creswell School, my conversation with Etienne, Berth and Mikael (see above) leave me with a sense that the processes we build for assessment for learning have to be living ones. We should not try to build protocols or methods that take away the quality of "being with" and reflecting together in whatever ways are needed. I believe it is the in the act of meeting on a regular basis among work colleagues with the intention of reflecting on progress and learning that will emerge the means and methods uniquely suited to their situation. Moreover, as researchers, we should not try to come in as experts with pre-conceived ideas of what is needed, but to build relationships founded on partnership and understanding, out of which the way to proceed will emerge.
I want to revisit the distinction between assessment in the operational domain and in the relational domain. The work of Tom Johnson provides us with a powerful conceptual foundation for developing capacities in the operational domain. Once his book is published, the ideas will become accessible for us to study ways to incorporate them practically within the Initiative. We are hoping to arrange a site visit to Toyota sometime next year to experience first hand a culture built on the premise of attention to process integrating empowered people, appropriate tools and foundational principles - the Toyota Production System.
Dennis Sandow has played a leading role in helping us to develop our collective thinking in the relational domain. He very clearly makes a stand for the primacy of relationships and has in his own business demonstrated the quality of social environment and work effectiveness that ensues. I would like to echo Dennis' vision for what I too believe we should be working toward. Dennis quotes Humberto Maturana as the eloquent spokesman:
I want a cultural change, I want to contribute to a work of art in the domain of human existence, I want to contribute to evoke a manner of coexistence in which love, mutual respect, honesty and social responsibility arise spontaneously from living instant after instant.
At the Creswell DID day lunch Dennis asked a very simple question to begin the dialogue on assessment: How would you assess your day yesterday? Each of about 15 senior administrators from 3 school districts gave their personal assessment. It was such an easy task yet out of that - effortlessly - began a rich dialogue on the deeper issues around assessment: the State-school tensions; the struggles of living in the two worlds; and a practical solution to the local illiteracy problem. My take home lesson was that in the new tradition assessment has to be collective, simple, supportive, concrete and regular.
Economic
value of organizational learning
The issue of demonstrating the links between organizational learning and financial results is at the extreme "hard" end of the continuum to demonstrate value. At the January Forum there was a move away from having this objective be a driving force within the Initiative for practical and philosophical reasons. The practical reasons included the impossibility of causally linking specific practices to financial performance due to such things as time-lags, externalities, intermediate effects - all of which confound the links. Moreover, abstracted measures are so removed from the reality of the experience that they should not be used for decisions. The prevailing feeling was that assessment of learning in organizations should focus on more systematic learning about the changes that occur and in this way build a credible body of evidence that intermediate performance measures are influenced by the nature of the ongoing work practices. The philosophical reason was concern about focusing on "proving" worth to others instead of having an intrinsic motivation to improve. Nevertheless, I get questioned about why we are not addressing economic value.
It is important for us to explore how this tension of "proving" economic value can be transcended, rather than resolved in one direction or another. While we consider this option, it would also be useful from my perspective to have a paper that systematically explores the practical and philosophical issues. SoL could provide real leadership in this crucial area by striving to demonstrate "best practice" in holistically linking learning initiatives to a range of intermediate performance measures (e.g., personal skills and behaviors, work practices, social networks) and overall business performance - including social measures applied within and outside the company. The focus of this assessing could be third party evaluation -- but it would be done in such as way as to intrigue people to learn more, not just to judge passively. It would ideally include involving those assessing in the experience of how the work is being done. In this sense, assessment for evaluation could become part of a larger learning process.
To illustrate, we could examine a few cases where organizational learning work has been going on for 3-5 years or longer, and document the different aspects of this work against the backdrop of the overall business performance. These cases should show and document in some detail the range of personal, interpersonal and structural or process changes that can be linked to this work, emphasizing changes in work practices. We would also interview different people associated and ask for their judgments and reasoning about the linkages between these different levels (learning initiatives, intermediate performance measures, and overall business performance). Doing this for several cases would enable us to raise core questions about what is happening in these cases and what might be gleaned as lessons or guidance for other situations. The key is to approach such in-depth case studies from an inquiring perspective, rather than from a perspective of offering any sort of definitive judgment. Such cases could then be "cross-examined" by independent observers, who could then offer differing interpretations and raise further questions.
None of this would ever definitively "prove" the link between enhanced learning capabilities and business performance. That would not be the goal. Rather, it would be to illustrate a level of serious inquiry that would provide a model to encourage others to approach evaluation in a more productive manner.
We are open to other ideas for next steps in this area from others in the community.
Evaluating
for accountability within SoL
Another powerful assessment challenge for the SoL community came from John Sterman in a talk to the research members at the 1998 SoL Members Meeting at Amherst:
What is the extent to which the Organizational Learning community and SoL encourage the skeptical assessment of the various methods that fall under the Organizational Learning umbrella? It would be remarkable indeed if all of the many methods to support and enable organizational learning embraced by SoL proved to be 'safe and effective', or if these methods could be extended to all the various contexts for which they are used and for which their use is advocated by SoL members. In system dynamics we have learned (usually painfully) the limits of applicability of the tool and method. We try to train our students to be skeptical about its applicability when approaching a new problem, to consider alternative approaches, and to be cautious about claims for efficacy and scope. I hope this is happening within the SoL community. It would be interesting to hear about the methods that SoL members have tried, evaluated, found wanting, and abandoned, and the ways in which these conclusions were publicized.
At the end of his talk he posed the following questions:
- To what extent does the growth of
"organizational learning" activity
resemble the dynamics of medical and
management fads?
- How can the effort devoted to and the
quality of follow-up studies and
rigorous assessment be enhanced?
- How can the positive loops driving growth be weakened so that
adoption and
use proceed at a rate consistent with the ability to do
follow up
studies?
- How can the skill dilution, infrastructure
saturation, superstitious
learning, and overexpansion of the scope of
organizational learning be
avoided?
- Is it too late?
John recognizes that the definition of "rigor" should be cause for conversation: in any enterprise as difficult as assessing the effectiveness of organizational interventions, we need multiple approaches. Still, not all approaches will be equally useful and reliable. Designing and testing useful approaches to assessment is part of the dialogue that needs to take place.
This evaluation issue is also at the heart of the initiative that Linda Booth Sweeney And BC Huselton have been guiding: to provide a coherent, integrated framework for SoL's capacity building role within companies.
For the evaluation aspect of SoL's own capacity building efforts to go ahead , there has to be a dedicated person to lead it. While it is central to assessment, it is also distinct enough and important enough to warrant special attention.
Allied to an internal evaluation focus is an idea that Peter Senge has been suggesting: as a community we carry out an assessment of what is already being done in individual companies around evaluation and assessment and what needs to be done. The aim would be to glean what knowledge and methods are already available but also to explore how to achieve strategic integration, diffusion and follow-up (sustaining) of learning capacity with a view developing these capacities.
Balancing the long-term with the short-term
How do we address the question of long-term sustainability in the context of SoL's Assessment Initiative?
Bill O'Brien at the January Forum (See Appendix D in "Assessing to Learn -- Learning to Assess" on the SoL website at http://www.SoLonline.org/co m/AR98/index.html) laid a foundation for thinking about the issues from his perspective as a long-term successful CEO. (He since referred us to a specific example of how current systemic norms led to bankruptcy of one company: "Garbage in Garbage Out" in Fortune May 25, 1998 pp130 -138).
Bill distills his wisdom and experience into a number of frustrations. Many of his frustrations illustrate how the prevailing forces allied to short-term profit-making preclude long-term generation of wealth. Bill emphasized that to invest in the long-term is to internalize the understanding that creation of financial capital is dependent on investment in human capital. By contrast, the pursuit of profit is ultimately at the expense of wealth. He gives compelling business reasons for investing in the long-term as well as stating it as simply "the right thing to do". Investment in human capital to him was inseparable from the "practice of virtue".
It would trivialize the issues to speak of realizing that degree of corporate and social change through the means of assessment. Yet, when viewed as a reflective tool, assessment gives us an opportunity to examine the values and intentions behind what it is we are trying to learn. Are we making choices in our operational principles and work practices that are consistent with long-term sustainability? What would it take for us to do?
At the recent Systems Thinking in Action Conference in San Francisco Stanford Business School Professor Jerry Porras gave a summary of his findings of the attributes of the leaders and companies that are "built to last". I believe these findings are an important key in reconciling the dilemma between evaluation which tends to be focused on short-term results and assessment for learning, much of which requires investment in long-term building of relationships and resilient social networks.
In response to an earlier draft of this report member Prasad Kaipa commented:
My belief is that both the short-term and the long-term are necessary and it is our role to ...understand how to get to the long term by first helping [companies] succeed in dealing with short term or to realign their work in a developmental way that is meaningful to them.
For this initiative to succeed, we need to look at both short term benefits and long term sustainability. If we can come up with ways for short term actions to be taken in the context of long term systemic perspective, we will have a win-win. That means, there needs to be evaluation and assessment combined in what we do and learning has to show up connected to living, working, feeling, thinking and being of those participants and organizations.
INITIATING PROJECTS
I have engaged with companies on an opportunistic basis to collaboratively explore existing interests and activities that could be turned into research projects.
The projects that are currently under the umbrella of the Initiative are described below. Some were already in train and are not and outcome of but support the aims of SoL's Initiative.
I am beginning to see common patterns in assessment interests among companies while simultaneously seeing how unique each company is with respect to how the principles and practices of learning are positioned, accepted and implemented. This implies that opportunities for projects need to be individually conceptualized around the specific initiatives and particular conditions within each company - and that takes time.
Examples of activities that could lead to research projects are: a capacity building initiative which a group has already undergone or is about to undergo; a new affiliation that requires social /organizational changes; an operating system that is particularly successful but unconventional within the culture. Any of these have the potential for development into research projects that would make the tacit learning more explicit and available for analysis and further use. The focus of research would be context specific and depend on the learners and on the domain of learning in which they are interested.
A frequently expressed wish by practitioners is for a retroactive assessment of what has been learned either through a specific initiative or after the years of implementing learning tools and principles. One reason is to encourage and support diffusion of learning capacity to other parts of the company. Whether to those who are interested but want to know more about the benefits or to create interest within a new group. A second reason is communicating the benefits to executives to maintain support. Some companies have invested resources on a large scale in a particular program and want to know if it has been effective. For example, Jean Tully of Hewlett-Packard asked: - having sent hundreds of people through a systems thinking course, how can we know if people are now thinking systemically? Has that investment made a difference? Are these the right questions to be asking?
Some companies are beginning to think about proactive assessment of intentional change processes. How do we assess our capacity building needs and match these with the products and providers? How do we do this in a strategic, sustained and integrated way? Others are asking: how can we track our learning as it is happening during the change? What procedures are available to begin such a process? What information would we need/metrics should we use to make real-time adjustments? What data would we want to record for analysis and synthesis over time?
By being opportunistic in starting projects, research questions arise from the practical needs of the companies. While this is consistent with staying in the emergence and working with what has immediate relevance for people, I feel that we are not addressing the systemic challenges that underlie our daily activities. The challenges I outlined in the previous section require intentionally addressing issues and tensions that underlie our social constructs of how we work and our assumptions about what has value in the long term. To what extent have we really even begun to cultivate learning communities? To undertake evolving a learning community would require a wholistic vision and approach which extends beyond focussing on particular capacities. I believe we could use the vehicle of assessment to ask ourselves such questions and to grapple with these challenges. I believe we need to define projects that address our deeper concerns. We need to collaboratively find the people and resources to undertake them on everyone's behalf. I would appreciate some guidance from the SoL community on how to move us into this deeper inquiry and wholistic (systemic) approach to learning and assessing our learning.
EMERGENT
PROJECTS
Below I briefly describe some of the projects underway or under discussion that illustrate the many forms that assessment has so far taken. The italicized questions are my interpretation of the focal question, but clearly each project is richer in its intent and its outcomes.
- How do we diffuse
learning?
At Harley-Davidson, the benefits of organizational learning are widely accepted among those involved in learning activities and the challenge is to help other groups get started and describe the benefits to those not directly involved. The question for them is: How can we usefully map our learning process and convey that process to others?Even when favorably disposed, people are asking - what exactly can I expect to get for the investment of my time? In response, we are considering a series of interviews with individuals who "have a story to tell" and capturing on video the "spark" of enthusiasm and heightened energy when a genuine insight or transformation has taken place. Such a video could be widely used to engage people with the heart and soul of organizational learning and provide some practical guidance around methodologies, tools and resources. Used in combination with the map, it would assist those wanting to get started and provide a diagnostic framework for those already involved with learning initiatives. Ideally, this video/interview process could be used across SoL member companies to test and modify the proposed "learning map". Additionally, this map could provide the conceptual framework for core learning competencies/skills being defined under another SoL initiative."
- How can we show others that
something really changes?
One company is considering a linguistic and ethnographic analysis of a series of "before and after" video tapes to systematically describe notable differences in behavior after two years of implementation of a program led by Claire Nuer of Learning as Leadership. The tapes could not be used directly for confidentiality reasons but a composite summary of indices of the behavioral shift would not only preserve anonymity but also provide one type of "objective" analysis of change. - Are people
thinking differently as they engage in intra- and
inter-personal
learning?
Dawna Markova is looking at cognitive changes in individual capacity by studying shifts in the level of activity in the domains of the brain correlated with empathy, analysis, organization and innovation/synthesis. She has a study under way at Shell Oil and Royal Dutch Shell in which she assesses habitual thinking styles before and after exposure to a learning initiative in an experimental group and similarly sequenced measures in a control. From any significant shifts that occur in the aggregate patterns for the groups, she will be able to make inferences about changes to breadth and versatility of thinking capacity. - What kinds of social structures support
innovative learning?
The Inkjet Business Unit of Hewlett Packard has commissioned a study with Dennis Sandow. of how knowledge is created within informal social networks. Using Humberto Maturana's biological theory of cognition and autopoiesis (self-producing systems) Dennis is studying how recursive conversations form knowledge creating social networks. Preliminary results demonstrate that knowledge flows through collaborative social networks. - How do people
organize themselves to carry out the work?
Shell Oil is in the process of entering into formal affiliations with others is considering a study to track what happens as formal and informal networks emerge. What are the conditions that make networks effective? How do leaders emerge and how do they lead? What "data" do we need to notice and track to describe and interpret what is occurring? How do we recognize and support "the energy" of high levels of motivation and creativity among large numbers of geographically dispersed people? -
How do we measure means?
Tom Johnson spoke at the January Forum of the systemic societal damage we have caused by the perversion of means in the pursuit of ends (see Appendix E, in "Assessing to Learn and Learning to Assess"). Abstraction of complex inter-relationships into simplistic measures for the purpose of measuring "results" has led us not only to create a mechanistic society but also leads to bad financial decision-making in the long-term. His research focuses on learning to recognize the patterns and relationships that comprise the dynamics of systems and to use these to guide operational processes. In particular he has been working with Toyota and Scania to develop ratios of scalars and rates of change among elements and their functional relationships as indicators of systemic health. Tom is in the last stages of completing a book on these ideas and they will soon be available to us more fully. - How can we assess and
sustain our learning as we work?
Detroit Edison is taking steps to build their capacity for assessing and sustaining learning during and after change initiatives: how can you make changes that are sustainable and not eventually overtaken by the larger system? And how do we know we are really making progress? What are the metrics that can be used to say in fact we are improving? How do we become a learning organization? We are planning to develop assessment procedures with a pilot group aimed to enhance personal and collective learning through regular self-observation; we will look for linkages with changes in performance. - How can we follow and support large scale
cultural change?
Research member Jim Walker from Sydney, Australia is conducting an assessment research initiative within Centrelink, a newly established Australian federal government department of some 25,000 employees. Centrelink was created a year ago to provide all government cash payments (e.g. unemployment benefits, old age pensions, student allowances) previously made by a variety of government departments. Staff of Centrelink, mostly employees of the original departments, are now expected to provide associated services such as counseling and referral to persons undergoing significant changes in their lives. Such duties are well outside of their original role as bureaucratic processors of applications for payment. Centrelink has embraced a business philosophy and a "quality first" policy focusing on improved performance. Integral to this is an Organizational Learning Project (OLP), which has attempted to build in assessment from the ontset. Research is being conducted on the progress of organizational learning and assessment, as seen within the context of an ambitious attempt at cultural change in a large organization.The OLP has three successive stages. (1) A project team is working with staff in two sites using organizational learning methodologies, including assessment methodologies, to assist staff in changing their roles. (2) Depending on the outcomes of stage 1, the project team will work with up to six further sites, building on the learning generated in the first two. (3) Depending on the outcomes of stage 2 further OLP initiatives will be supported throughout the organization.
Throughout the project, there will be two assessment processes. Evaluation is being conducted by the Centrelink Audit Office, using performance sampling and best practice methods. Assessment to support learning is being conducted through individual and group self-assessment by project team members and staff in the sites. The self-assessment is supported by experienced coaches/mentors, who will also assist in the documentation of the research. Thus it will be possible to compare evaluation for accountability and compliance with assessment for learning; and both will be available to decision makers as they consider the future role of organizational learning in Centrelink.
- How can we better
articulate outcomes of organizational
learning?
The assessments that actually drive decisions in organizations are shaped in conversations among key players, including those directly involved in an initiative, learning leaders, sponsors, and those responsible for resource allocation. A good way to begin improving practice and generating knowledge is by reflecting together on how we are conducting those conversations now and how we might make them better. Bob Putnam of Action Design has offered to facilitate a reflective workshop of this kind. - What are we learning about organizational
learning?
George Roth is proposing a conference to use a series of learning histories to collectively explore "what are we learning about organizational learning?".The intention is to involve the three groups of SoL members (company members, consultants and researchers) to focus on building actionable knowledge on learning patterns in organizations. This would bring together those with an understanding of which capacity building strategies and skills are effective; those with a rich set of data on learning and those whose an interest in the development of theories of learning and change in organizations. The participants would study three learning projects by working closely with the learning history manuscripts, extending what happened in these companies by comparing and contrasting them with their own experiences, and creating propositions for the patterns of learning in large organizations.
- How can SoL evaluate its
capacity building?
SoL ( previously as the Organizational Learning Center at MIT) has been providing opportunities for members to learn collectively i.e. the five-day core course, company-sponsored meetings, the liaison group, annual meetings, SoL web site, etc. A new phase of the SoL capacity building effort is being conceptualized to include a network of experienced teachers, facilitators, researchers and practitioners to provide integrated learning and development opportunities in the concepts, methods and tools of organizational learning. Linda Booth Sweeney and BC Huselton have sought SoL member input and have led the efforts of the SoL Capacity Building steering group [3]to develop a framework and plan for such an integrated, practical curriculum. They report that in their focus groups concern was raised with regard to ensuring and maintaining high quality standards across a wide range of courses, teachers, facilitators and implementation sites. The analysis of John Sterman (see above) has raised similar, generic questions about accountability. The issues of quality, effectiveness, safety and standards have thus been tabled and the next step is an inquiry into how these can be addressed practically. - How can companies
learn from each other's experience of assessment and
evaluation?
Many companies have undertaken assessments of learning initiatives and have experience with what works and what does not. We will experiment with creating websites where such information can be shared, with facilitated on-line dialogue, and with open-space technology to invite self-organizing around topics of interest.
We are looking to interview individuals who have personal experience of substantive or transformational changes in the workplace and to document their story and their judgments and reasoning about the linkages between learning initiatives, intermediate performance measures, and overall business performance.
ONWARD
FROM HERE
In summary, we now have a five-pronged emphasis:
To continue to:
a) support the above projects and foster the development of others as they naturally arise from interests in the community.
b) research and develop the concepts, principles and practices of self-assessment as an integral component of the work culture and operations.
c) foster ongoing structured and informal conversations within and outside the community on issues related to assessment and evaluation.
To develop the means to:
d) explore and document the links between changes or transformation in the relational and conceptual domains with changes in performance and business results
e) engage, as a community, with the challenge of self-evaluation of SoL's capacity building methods, tools and training.
I welcome your comments, stories and suggestions . Send these to me at: stellaH@mit.edu
Copyright © 1998 The Society for Organizational Learning. All rights reserved.