From Theory to Practice:
Research Territory, Processes and Structure at the
MIT Center for Organizational Learning
SUBMITTED TO:
Journal of Organizational Change Management
Special Issue on Organizational Learning
July 17, 1995
George L. Roth
Peter M. Senge
Center for Organizational Learning
MIT Sloan School of Management
30 Memorial Drive
Cambridge, MA 02139
-
Abstract
1. Introduction
2. Research Territory
3. Research Outcomes and Methods
4. Field Project Research Design and Process
5. Summary
References
Figures
From Theory to Practice:
Research Territory, Processes and Structure at
the MIT Center for Organizational Learning
The purpose of this paper is to outline the territory, goals, approach, and process underlying field research projects within the MIT Center for Organizational Learning. In the first section we provide a brief history of how the Center came about and the historical threads of theory and method which are the basis of our present research efforts. In Section 2 we develop the premise for focusing organizational learning efforts on a territory characterized by high behavioral complexity and high dynamic complexity. The implications of this territory for research methods and field research projects is discussed in subsequent sections. Section 3 describes the research outcomes and methods, and Section 4 draws upon those outcomes and methods in describing how our field projects are organized. In the final section, in Section 5, we summarize our thinking on the integration of research territory, methods and field research projects. Our intention in this paper is to describe one model for addressing the research and practice issues associated with promoting organizational learning in a way that we believe leads to practical testing of new ideas while leading to generalizable insights and new methods.
1.Introduction
The ideas described in this paper reflect the progress and frustrations of the first four years of the MIT Center for Organizational Learning. The Center was established in 1990 in order to apply and test the concepts proposed in Senge's The Fifth Discipline (1990) and Sloan Management Review article, "The Leaders' New Work: Building Learning Organizations" (Fall, 1990). These writings, in turn, built upon many years of research in system dynamics (Forrester 1961, 1969, 1971; Meadows, 1982), action science (Argyris and Schon, 1978; Argyris, Putnam and Smith, 1990; Argyris 1982, 1990) and related approaches to group process (e.g., Schein, 1987), the personal creative process as understood from the creative arts (Fritz 1989, 1991), dialogue and collective thought (Bohm, 1987; Isaacs, 1993; Schein, 1993) and practical experience in developing organizational learning processes (e.g., de Geus, 1988; Stata, 1989). (See References.)
In starting we were interested in developing a critical mass for fundamental change. The intent was to foster serious experimentation in a group of large corporations and to nurture a community of practitioners from these corporations who could learn from and with one another. This intent required developing a partnership between researchers and practitioners to design, implement, and study learning processes. Through such learning processes we might test theories and tools in realistic practical settings, leading to both improved theories and to better tools that could then be used more widely. We also hoped to show what is possible, in terms of dramatic improvements in business results, when teams and larger organizations begin to internalize new learning capabilities.
In the year that it took to establish the Center, and the ensuing three years of work with sponsoring companies, we have gained invaluable experience in putting ideas into action, observing results, and have received critical help from many colleagues.[1] From its inception, the Learning Center was a collaborative effort, and much of the work in the first several years has been in developing partnerships between researchers and practitioners in member companies. Today, there are twenty corporate sponsors, with in-depth projects in about half of these companies. [2]
The Center is now at a crucial juncture in its evolution. During the first years of operation, projects focused more on producing business improvements than on producing research reports. We worked to establish a "proof of concept" -- to show that a powerful synergy was possible, between tools and methods, and that the two were connected. We believed that typical work environments characterized by reductionist thinking, defensiveness and top-down goal setting inherently produce mediocre results. Having succeeded in showing what is possible we are now shifting toward accumulating findings across multiple field projects and documenting research findings. In the remainder of this paper we describe our focus in terms of three areas: specification of the research territory, consideration of research methods, and the specification of a structured research process that applies those methods in the research territory.
2. Research Territory
Research at the Learning Center focuses on situations characterized by high degrees of behavioral complexity and dynamic complexity (see Figure 1). Taken together, the dimensions of dynamic and behavioral complexity describe a "problem space." Managers and decision-makers face increasingly difficult challenges as either dynamic complexity or behavioral complexity increases. When both dynamic and behavioral complexity are high, the challenges can be overwhelming. This is especially so because the types of skills required may diverge: dynamic complexity requires high level conceptual and systems thinking skills, behavioral complexity requires high levels of interpersonal and facilitative skills.
The territory defined by dynamic and behavioral complexity can be divided into quadrants of four different types of problems. As King (1993) shows in the context of the nuclear power industry, catastrophic consequences can result if our efforts don't first recognize what type of problem we face. Solving the wrong problems not only fails to solve the right problems, but "we unwittingly undermine what it takes for us to solve the right problems" (King, 1993: 106). For example, blaming operators in the accident at Three Mile Island was politically expedient but it failed to consider the "insidious accumulation of delayed-action human failures occurring primarily within the organizational and managerial sectors" (Reason, 1990: 476 quoted by King, 1993: 107). In this example, a "mess" -- the complex interrelated set of problems associated with operating a nuclear power plant -- is treated like a "tame problem" -- operator error. Once it is "solved" by eliminating the identified error and retraining operators, the incentive to further inquire into the more systemic causes of failure disappears.
Behavioral complexity characterizes the extent to which there is diversity in the aspirations, mental models, and even values and basic assumptions of decision makers. High behavioral complexity is characterized by deep conflict in assumptions, beliefs, and perspectives. Under conditions of high behavioral complexity, it is difficult to get people to agree on what should be done because they see the world very differently and because they have different "agendas" or goals. For example, entrenched management-union conflicts are situations characterized by people negotiating with one another with different understandings and goals. Political situations with high behavioral complexity include Catholics and Protestants in Northern Ireland, Jews and Palestinians in Gaza, and blacks and whites in South Africa. On the other hand, when behavioral complexity is low, people share underlying assumptions and values from which they can develop common perspectives and alignment in their actions. An example of low behavioral complexity is a group of financial analysts solving a technical problem. Whenever groups are composed of people with homogenous backgrounds, behavioral complexity is low and problem solving emphasizes data gathering, analysis and rationality.
Dynamic complexity characterizes the extent to which the relationship between cause and resulting effects are distant in time and space. Large and complex organizations are good examples of high dynamic complexity. In situations of high dynamic complexity, the causes of problems cannot be readily determined by first-hand experience, and few, if any, of the actors in the system may have a sound understanding of the causes of problems. So, for example, when a large organization makes a change in its reward system in order to improve its bottom line, it is very difficult to link the action - change in how people are compensated - with the expected result - change in profitability. There is considerable opportunity for other factors to influence profit, and the elapsed time between the change in policy and the consequent change in the bottom is long. When dynamic complexity is high, management interventions tend, at best, to improve matters in the short run, only to lead to more problems in the long term. Even worse, many of the most pressing problems people face are actually the unintended consequences of past "solutions." By contrast, low dynamic complexity occurs in situations where it is easier to link actions with outcomes. When a supervisor, for example, working with an operations analyst, implements changes in the order in which tasks are performed on a production line it is generally possible to directly observe the impact of those changes on production rates.
Research suggests that decision makers have great difficulty learning from experience in the face of dynamic complexity. In experimental studies, decision-makers take actions which are ineffective and their effectiveness does not improve with repeated experimental trials (Paich and Sterman, 1993; Kampmann and Sterman, 1994; Diehl and Sterman 1994). This research suggests that just getting people to communicate more effectively is inadequate because our cognitive maps are much simpler than the real life systems we routinely encounter.
When problems of low dynamic complexity combine with problems of low behavioral complexity, the result is a "tame problem" (Rittel and Weber, 1973). Tame problems can be solved using conventional analytic methods involving data collection and "static" analysis (i.e., analysis that does not require dealing with delays, multiple feedback loops, and nonlinear relationships). Tame problems can be solved in isolation. Traditionally, tame problems are broken down into parts which can be solved independently by different groups of people. Solutions to different parts of larger problems can then be integrated into an overall solution because (1) there are no significant dynamic interconnections between the parts and (2) different actors share common values and goals.
"Wicked problems" are those where behavioral complexity is high; where complex underlying social realities are inescapable; and where different groups of key decision makers hold different assumptions, values, and beliefs which are in opposition to one another. (King, 1993; Rittel and Weber, 1973). Geertz (1973) describes the "loss of orientation" that arises in the absence of an overriding social theory or ethic. When there is no overriding social theory and ethic, people see the situation from different perspectives and plan strategies for what could and should be done based on different mental models. Moreover, these different mental models remain in the background and are typically "undiscussable." "Wickedness," according to King (1993), "occurs when people confer immutability on value assumptions and ideological considerations."
"Messes" (Ackoff, 1974) arise when dynamic complexity is high. These are puzzles that are not so much "solved" as sorted out in terms of their inherent complexities. Messes cannot be solved by solving component problems in isolation from one another because there are significant couplings between isolated problem symptoms. For example, the breakdown of discipline in the classroom cannot be addressed effectively by stricter teacher control because the larger parenting and community systems out of which students come have also broken down. Sorting out messes is complicated by "vicious and virtuous cycles," "tragedies of the commons," "shifting the burden," and similar dynamics which are often neglected by individual decision makers under pressure to "fix" problems.
Much research has been done on behavioral complexity and dynamic complexity. What befuddles real decision making is that the two co-exist and interact in what we call "wicked messes." The fact that problems cannot be solved in isolation from one another makes it even more difficult to deal with people's differing assumptions and values: people who think differently must learn about and create a common reality, one which none of them initially understand adequately. Systems of interlinked problems interact with the misunderstandings, divergent assumptions, and polarized beliefs of different groups of people. Improving communication and trust among different camps is not enough; people are still likely to focus on symptoms rather than deeper causes and pursue low-leverage changes. Conversely, even if deeper understanding of the systemic forces at play is achieved, such understanding will be viewed with suspicion by the different, competing interests and mental models.
This territory is important for three reasons. First, dynamic and behavioral complexity characterize the most vexing social problems, both within organizations and within society. Examples include global environmental problems, government deficits, erosion of public education, and the gradual decline of a corporation's vitality and competitiveness. Second, such problems go largely unrecognized. There is a tendency to treat such problems as if they had either purely technical solutions or purely behavioral solutions -- as if the key were to simply gather the right data and analyze it correctly or to get people communicating more effectively. Lastly, theory, tools, and methods for addressing such problems are largely under developed.
Behavioral complexity and dynamic complexity have been the foci of different and unconnected academic fields of study. Behavioral complexity is the traditional domain of "soft-science" inquiry and intervention-based techniques associated with the fields of organizational development, negotiation theory, conflict resolution, and labor relations. On the other hand, dynamic complexity is most associated with "harder," more technical fields like operations management, system dynamics, and related analytic model-based, mathematically oriented problem solving techniques -- especially those that explicitly deal with multiple feedback loops, delays, and nonlinear relationships. Consequently, the intellectual foundations for dealing with wicked messes is almost non-existent.
Our approach to the territory of wicked messes assumes that new syntheses of previously disconnected approaches will be required, along with new theory and methods. We assume that effective syntheses will entail a blending of "technical" and "behavioral" approaches -- conceptual and analytic tools developed to understand complex dynamics, along with the thinking and inquiry skills needed to surface and suspend mental models and assumptions.
Undergirding the entire effort is a recognition of the complexity and the newness of the territory and a belief in the power of a community of researchers and practitioners working together. Unless we learn to learn from and with one another, across traditional organizational and cultural boundaries, little real progress is possible. Finding and developing research methods appropriate for the field projects in this area has been an important endeavor.
3. Research Outcomes and Methods
The expected outcomes of our research activities are three fold: living examples of companies developing new learning capabilities, new tools and methods that can be used widely, and improved theory that can be tested and lead to replicable outcomes (see Figure 2).
Living examples are vital in order to demonstrate what is possible. We believe that developing capabilities to deal with both dynamic and behavioral complexity can lead to quantum improvements in productivity, effectiveness and people's excitement and commitment in their work. Moreover, we believe that the type of synthesis of technical and interpersonal skills fostered by approaches like the "five disciplines" can contribute to developing such capabilities. But, this belief must be tested by attempting to bring about just such improvements in realistic settings. Existence is a compelling proof of possibility. Such "living examples" will leave many questions about replicability and generalizability, but we believe that they will also create the impetus to go further -- especially, if these examples involve practical tools and methods.
General tools and methods are vital to make new knowledge useable. Without generalizable tools tested in many different settings there is no way to know if improvements achieved are due to better theory or to clever interventionists and unusually competent managers. [3] As tools are used, we develop practical know-how on what is needed to make existing tools effective, modifications to improve them, and insights into their limitations and the need for alternative methodologies.
Broadly applicable tools and methods are only possible when there is a firm underlying body of theory. In addition to the core theories of nonlinear system dynamics, collaborative inquiry and the creative process, we are also developing new general theory that will eventually give rise to new tools, such as work on dialogue and collective thought (Isaacs, 1993; Schein, 1993) and on learning consortiums and organization sets (Schein, 1995). In addition, as existing tools and methods are applied to particular classes of management issues, new substantive management theories are developed in areas like product development (Seville and Kim, 1993) and managing service quality (Oliva and Senge, 1993).
Our goal of developing theories and methods that are practical has several implications for research methods. First we need to work in realistic field settings. In field research projects we draw predominantly on the qualitative research methods found in action research (Argyris, Putnam and Smith, 1990) and ethnography (Schwartzman, 1993). The complex, collective learning phenomena we seek to influence and study requires integration of science with social practice. The research process we propose includes the components of action research (see Argyris, Putnam and Smith, 1990: 8-9) as follows: 1) learning projects are with company partners; 2) the learning cycle is used in planning research and project activities; 3) tools and techniques for thinking and learning are taught; 4) learning and development are promoted by building capacities of people in the organization; and 5) while developing and testing new theories, methods, and tools for learning we seek to improving management practices. Our research process is itself an ongoing process that encourages continuous learning by both researchers and managers.
We supplement action research by making the research process part of the learning process. In action research activities, researchers and managers often find it difficult to abstract meaningful general insights from the idiosyncrasies of particular settings. We have developed an organizational reflection and documentation process called "learning histories" (Roth and Kleiner, 1994). The learning history approach supplements action research, program evaluation and ethnographic methods. The learning history approach helps subjects assess and evaluate themselves, as researchers capture the data which also allows the larger learning process to be documented. The learning history approach combines techniques from ethnography (Spradley, 1979; Sanday, 1979; Van Maanen, 1979) with oral history (Yow, 1994) to promote reflection among participants as data are collected.
A dilemma inherent in most evaluation approaches is that experts' assessments are separate from participants' learning. The dilemma is that the feedback which people require to assess themselves is often only collected in organizations through experts' involvement, and as part of their involvement, the experts make evaluations. The learning history approach has researchers and management practitioners working together as an insider/outsider team (Louis and Bartunek, 1992). The "learning historians" work with learners to help them reflect and assess their own efforts, and then utilizes the data from people's reflection and assessment of themselves as the basis for documents which are more broadly disseminated. Multiple perspectives are sought - those of the people in the organization involved in the efforts, other people in that organization, and the action researchers. The learning historian team creates "jointly told tales" (see Van Maanen, 1998: 136-138) that describe work issues and learning experiences from multiple and often contending perspectives. Learning histories are proving effective in engaging and influencing readers because of extensive use of participants' own narratives to capture multiple, coherent "stories" about complex realities. A two-column format is used to keep the research team's commentary separate from participants' own descriptive and evaluative narratives.
In addition to field research methods, we also use more normal science methods, in our theory building work. For example, system dynamics methods are used for formulating and testing dynamically complex theories projects (Forrester, 1961; Forrester and Senge, 1980; Sterman, 1985) of product development, service quality, supply chain dynamics and other substantive issues in field. In studies of behavioral decision-making, we rely on experimental research methods developed by Sterman (1994a).
The challenge of conducting research in this complex territory is further complicated by having to manage an organization to do so. Not only are many researchers involved as managers in the Learning Center itself, but the nature of field settings - teaching learning tools and methods, and working with managers to plan and deliver interventions - requires that virtually all researchers have a role in managing people and projects. We manage as well as help others manage. This challenge fosters both humility and empathy, and deepens our understanding of the practical as well as theoretical challenges of this work.
In particular, we have found that growing such a research organization requires holding in balance a number of "essential tensions," similar to those our partner business organizations must themselves face. People are pulled by multiple, often conflicting, forces. On the one hand, there might be genuine interest in improving the organization's capabilities to learn. On the other hand, people are rewarded for business results, usually short-term, measurable results. On the one hand, there may be a commitment to reflection and inquiry. But, on the other hand, especially in American business cultures, "doing" is more valued than "thinking," and the more abstract, conceptual, and future-oriented activities associated with learning and research are typically associated with staff, not line, jobs. The desire to balance conflicting forces of learning and performing requires a continual awareness of those forces, inquiry to surface and articulate contradictions, and concern for producing project outcomes which balance research findings with business improvements.
4. Field Project Research Design and Process
In order to coordinate across multiple field projects, we strive to follow a consistent research process. This approach both helps in formulating particular research questions in each project and in providing check points to assess business outcomes and research progress. First and foremost, we view the research process as a learning process for managers and researchers alike. In so doing, we build upon a long tradition dating from John Dewey (1896), Kurt Lewin (1951), David Kolb (1984), and more recent advocates of "continuous improvement" like Edward Deming (1982), who all view learning as a continuous process that connects thinking and acting over time. We refer to the experiential learning cycle as the "OADI" (observe-assess-design-implement) cycle (Kofman referenced in Kim, 1993a: 38-9).
The OADI cycle involves observation of concrete experience; assessment reflecting on observations to interpret and their meaning; design of possible actions based on the assessments; and testing the design by implementing it, leading to further observations (see Kolb, 1984 for more detailed description of experiential learning). Kim (1993a) extends the OADI cycle to include shared mental models in developing an integrated framework for organizational learning.
The only basic problem with the OADI cycle in business settings is that it doesn't work very well -- while it characterizes how learning might occur, it also shows why often little learning does occur. Central to the OADI cycle, and to all experiential views of learning, is that learning occurs as human beings observe and reflect on the consequences of their actions, leading to new understandings and actions. Put differently, learning processes often involve "mistake making" and then learning from those mistakes. Managers have limited opportunity for such learning. The consequences of bad managerial decisions can be catastrophic, both in financial and human terms, so every effort must be made to avoid making mistakes. When mistakes are made, there are psychological and social pressures to cover up the mistakes rather than to learn from them, what Argyris (1990) calls "organizational defensive routines." Moreover, it is extremely difficult to learn from decisions whose consequences may unfold over years, and where those consequences may be distant from the original decision makers. Consequences may also be ambiguous and/or influenced by forces outside of your control.
Understanding these breakdowns in the learning cycle has led to a central strategy underlying all our field research projects: designing, implementing and studying "managerial practice fields." Managerial practice fields are designed learning spaces where decision-makers can experiment, make mistakes, accelerate learning and test new behaviors. We believe that such practice fields, which we often call "learning laboratories," will eventually become an essential element of the new infrastructure for learning that will characterize organizations that can develop, capture, and disseminate knowledge in ways that traditional organizations are unable to do (Kim, 1994; Senge et. al. 1994:32-36). Just as it is unimaginable that sports teams or theater troupes would never practice, so too will it be unimaginable that managers never practice.
There are several basic features of managerial practice fields. First, they allow for reflection. The "observe and assess" phases of the learning cycle are frequently compromised in real decision-making settings. The quality of data observed in real decision making settings is often incomplete or distorted, and the opportunity to seriously reflect on how we interpret that data and form assessments of our decisions is often missing entirely. Especially in the West, and particularly within American firms, the "bias to action" often means a bias against reflection. The hectic pace and continual problem solving, often makes life feel like a perpetual "fire drill" for American managers. Managerial practice fields can encourage personal reflection on aspirations and assumptions. They can also encourage dialogue to help understand different people's aspirations and mental models and to build shared visions and mental models. This can also enhance the "design" phase of the learning cycle. Typically, new actions are often nothing more than new reactions, rather than thoughtful crafting of new strategies based on reflection and rethinking.
Second, practice fields allow for experimentation -- experimenting with alternative decisions and alternative ways of interacting. Experimentation is discouraged in real decision-making because of the costs of mistakes, and the psychological and social pressures to conform. This greatly limits the "implement" phase of the learning cycle. In practice fields, mistakes are not financially costly and are recast as learning opportunities. Experimentation in practice fields is actively encouraged, followed by extensive conversation to surface the different assumptions upon which people base their thinking and action.
Thirdly, practice fields can compress time and space to see more clearly the systemic and longer term consequences of decisions. Using simulation, for example, individuals and teams can work through a product development cycle which normally takes three to four years (as is the case in the automobile industry), in an afternoon (see Booker, 1994). Computer simulation based on system dynamics modeling (Forrester, 1961; Senge 1990: 313-338; Senge et. al. 1994: 529-560) has been a basic component of Center's research since its inception.
Practice fields only matter if they connect to "performance fields." A second major goal of our research design is to understand the interplay between learning in the practice field and learning in "real" decision-making settings. For example, in learning laboratory sessions, managers are taught tools and methods that allow them to inquire and reflect on their own and others' mental models. Eventually, they develop a repertoire of new behaviors that enable them to inquire into each others' thinking without invoking defensiveness. In the practice field sessions, people also learn how to step back and conceptualize the larger systemic forces driving problematic situations. The practice field allows the manager to work "on" the system - developing and improving theories of how larger systems work, including how they themselves work in those systems. This insight can lead to greater awareness and changes in decision-making when they are subsequently working "in" the system, and to redesigning the "physical" aspects of that system, such as reward systems, information flows, or the physical structure (Kim, 1994: 6). In our research, we are also interested in studying whether, over time, the "distance" between practice field and performance field shrinks, so that the capacity to reflect, conceptualize, collectively inquire, and act in more coordinated ways that characterizes the full OADI learning cycle, is evident in "real" decision-making settings.
A learning lab project with Ford Motor Company illustrates these ideas and the research process through which they are implemented (Roth, forthcoming, describes this project in more depth). A course in systems thinking offered through Ford's Executive Development Center led to interest by some managers in applying these concepts in real work settings. Line managers from a particular vehicle development program volunteered to use organizational learning concepts in their program. A team of researchers conducted initial interviews with engineers and managers associated with the program. The interviews were summarized into themes. These themes, illustrated by representative quotes, were described in a document, whose contents formed the basis for an "engagement clinic" among researchers and managers. The clinic provided an opportunity to test the ability and willingness of managers to engage with challenging issues of the kind that surface through learning initiatives. It also provided an important forum for discussing the implications of conducting research while trying to accomplish business objectives.
The following eight months of activities involved a researcher and graduate students working one or two days a month with a cross-functional core team of fifteen to twenty managers. That core team of managers was responsible for the different vehicle development teams (see Kim, 1993b; Giancola, 1992). During this time the managers learned new skills and tools that helped to enhance their commitment, conversation, and conceptualization capabilities. In the process of applying these new skills, managers investigated systemic conditions that limited effectiveness in the vehicle development process. As the core team learned together and gained new insights, those insights influenced the way they managed. For example, in creating a system causal loop diagram managers realized that the perpetual lateness of parts for prototype builds was caused by many factors. One of those factors was that engineers were measured according to how long engineering changes orders were left open on the computerized reporting system. Thus, engineers were not always forthright in reporting problems, waiting until either they knew the solutions or had to provide parts for prototype builds. In examining the multiple cause and effect relationships which affected lateness of parts, managers on the core team realized that the problem of late parts was inseparable from trust. So long as trust was low, people would put off reporting problems. They also realized that openness and honesty were behaviors they had to demonstrate themselves before they could ask the engineers to be more open and honest.
The initial learning period provided the core team with an opportunity to learn and practice new skills. One of the tasks that the core team eventually took on was designing a "learning lab" for engineers in their organization. Thus, while managers were learning and applying new skills themselves, they were also evaluating them for their usefulness with others. The learning lab for engineers was offered on a voluntary basis. When the first learning lab was announced by core team managers, considerable interest had been building from engineers' curiosity about what their managers were doing and learning. Those learning labs for engineers were taught by both researchers and managers. The experience from each learning lab influenced the design of subsequent learning labs. Engineers in the early learning labs also took roles in teaching sections of the subsequent learning labs for their colleagues.
The vehicle development program led by this core team set numerous internal company records for quality and timeliness of parts. Now that the car the team developed has been launched, other vehicle development programs continue to build on and apply learning techniques. Learning concepts have also spread to other divisions beyond new car development.
The field research project process described at moved through four main phases:
Throughout this process, the OADI cycle operates at both a "macro" level, interconnecting the different phases of a project, and a "micro " level within each phase. The figure below depicts the four main phases of the research process:
The pre-project "Phase 0," develops mutual understanding and appreciation for research issues and business issues, leading eventually to selecting a project site and research focus. Without mutual understanding and strong relationships between researchers and managers, it is not possible to move forward with a field research project. In fact, such understanding and relationship building continues over many years, but it must start in Phase 0, or there is no reason to expect it to continue.
During Phase 0, managers and researchers work together to identify potential project settings that could lead to significant business and research results. Criteria for project selection include:
- wicked messes: from the perspective of the people in the system, they are facing difficult change issues, perhaps ones they feel are impossible to surmount;
- generality: business issues are generic, not idiosyncratic to one company or industry;
- significance: potential for business impact is high;
- leveragability: insights and new capabilities developed in this setting could potentially diffuse widely within the organization;
- line leadership: local line leaders with responsibility can form teams of people with "the power to take action" vis a vis the issues addressed;
- theoretical foundation: past research provides a foundation of prior theory that can be a starting point, especially in understanding systemic issues at play (e.g., field projects on the dynamics involving product development projects and on managing product development (Roberts, 1992).
Phase 0 can easily take a year or more. We have found that considerable effort is needed to find settings with an appropriate mix of critical business issues, committed line leadership and researchers' capability. For several companies in the Center, we have been unable to initiate pilot projects, even after several years. However, the more time and care we take in establishing initial project conditions by surfacing both research and business expectations, the more likely we are to create conditions through which we can achieve desired outcomes. In Phase 0, critical leadership is provided by "internal community builders," individuals within the organization who can search out prospective sites and local line leaders (see Senge, forthcoming).
Phase 1 is the first phase of a field project. It commences with a "project engagement clinic" and an initial "project research clinic." During Phase 1, the core team that will provide leadership for the project is becoming immersed in the basic tools, methods, and principles (Senge, 1990) which will help them improve their individual and collective learning abilities. There is a focus on fostering personal and shared vision, understanding diverse mental models, and appreciating dynamic complexity. People work with tools for reflection, dialogue, and conceptualizing systemic causes of problems. Computer-based "management flight simulators" and manual simulations (like the "beer game" (Senge, 1990; Sterman, 1992) are used to give people first-hand experience of how cause and effect can be distant in time and space, and how well intended interventions can cause more harm than good. In Phase 1, it is critical that the core leadership team move beyond mere intellectual appreciation of learning organization ideas and begin to "walk the talk," or else they will be ineffective in leading subsequent organization learning processes.
Simultaneously, in Phase 1 we are developing shared understanding of the key business issues to be addressed and the core challenges that will lie ahead. In many ways, Phase 1 is a "problem articulation" process -- we judge success by the extent to which established ideas about the nature of the problems in the real setting begin to shift. For example, in the Ford product development project, people initially thought that the fundamental problems were management interference and lack of collective commitment. Eventually, they began to discover how their own ways of interpreting problems ("so and so isn't trustworthy") and their habitual behaviors ("the boss has to be the boss") were creating the inability to put all the pieces together. Typically, Phase 1 lasts six to twelve months. Phase 2 can commence once the core team has developed the skills and shared insight to begin designing a learning process (i.e., a practice field) that could help others in the real system develop similar skills and understanding.
A major challenge that runs through Phase 0 and Phase 1 concerns the understanding of everyone involved, managers and researchers, of what it means to work together within a research project. Managers are used to working with consultants and will invariably see the researchers initially as consultants. Researchers likewise might tend to see themselves as consultants, there solely to help the managers. Of course, this is one of the goals of the research process, but there are some significant differences from traditional consulting. First, we make it clear that we assume no responsibility for producing change within the organization -- our task is to help people (individually and collectively) to develop their capabilities to produce change. Second, we have limited capacity, and the only way the work can spread beyond limited pilot testing is if the organization develops its own capacity. Gradually, the Learning Center is developing a capacity building program to help the organizations grow their capacity. Third, we insist on documenting the process as it unfolds. "Learning historians" are a key component of every project, typically a team of internal (to the organization) and external learning historians who we train.
It takes months and years to break down traditional mental models of "clients and consultants," but gradually a new set of understandings and expectations evolve, and the managers often come to uniquely value this different relationship. A year into one of our first projects, one of the managers in the core team was asked, "How is it different to work with the MIT team than with consultants?" He responded, "I have always felt that consultants told me what they thought I wanted to hear. These people are more difficult, they ask tough questions that often I would rather not hear but which I really need to think about."
Phase 2 involves pilot testing the learning process that emerges from Phase 1. In wrestling with "wicked messes" there may be hundreds of key "decision-makers." These are not typically issues where a small handful of people can design and implement necessary changes. Moreover, a core challenge in developing organization-wide learning capabilities is to embed skills in systems thinking, collaborative learning, and building shared vision throughout the organization. Thus, the practice fields that we typically develop are ultimately aimed toward large audiences, such as product development teams throughout an organization, an entire sales force, or large manufacturing organizations. In Phase 2, we focus on extending the learning process from the core team to a small number of other teams that are representative of this larger audience. The focus is on fostering understanding in terms of inter-related problems and different mental models, and on accelerating the learning process of the core team. We also seek to involve senior management in developing a large system change plan for diffusing learning.
In Phase 3, the learning process is being diffused more broadly. There is a move from pilot and trial testing of learning interventions to broad-scale replication of learning events, formal quantitative measurement of changes, and wide-spread teaching of learning tools and methods. For example, at Ford the Learning Lab pilot tested in Phase 2 was eventually made available to over 200 engineers in the car program. Moreover, it became a starting point for other car programs to begin to design their learning labs as well as a catalyst for simulation efforts in other divisions.
The phases of the field research process build on the OADI cycle in two ways. First, there is the application of the OADI cycle within each phase. Each phase involves cycling back and forth between study and practice - learning new tools, applying them to work issues, and reflecting on application. Within a research phase there is an articulation of what is being observed, what assessments managers have, and how they would deal with these assessments (design and implementation). Each research phase involves both reflective learning - observing and assessing - and action learning - implementing learning interventions. For example, Phase 1 involves developing and testing new learning processes within the core team, helping the team members to operationalize, transfer and test what they have learned. Within Phase 2, the pilot testing can lead to new observations and assessments, as the initial hypotheses formulated in Phase 1 are tested and new insights and skills are put into practice by new teams of managers. At Ford, for example, significant new insights (assessments) of systemic problems in product development continued to arrive during the Phase 2 and Phase 3 learning labs (Kim, 1994).
Secondly, the OADI cycle applies across research project phases. Early phases are predominantly concerned with observation and assessment. Subsequent research phases have increasing emphasis on design and implementation. Phase 1 emphasizes developing skills of observation and assessment, so that managers and researchers can construct rich, grounded, multifaceted articulations of problems that are associated with the "wicked messes" being investigated. Phase 2 emphasizes moving from assessment to design and pilot implementation, testing initial hypotheses through designing and implementing a learning process, and studying the effects in helping other teams enhance their effectiveness. Finally, Phase 3 emphasizes design and implementation of learning processes on a larger scale and study of system-wide change. Throughout, the overall vision is to create managerial practice fields that can be embedded within the workplace, leading to ongoing process of reflection, theory building, and improved decision-making and systemic design.
Successfully carrying out the research process described above is a challenge. It requires significant time commitments from many people playing key roles, including significant commitments from busy line managers. It requires commitments of all parties to research and practice, to producing general knowledge and methods and to improved business results. This means, in many ways, new work for managers, as well as new work for researchers.
5. Summary
It is our hope that the MIT Center for Organizational Learning can serve as a model of sorts for other research centers interested in organizational learning. One goal of initial field projects at the Learning Center has been to develop and test learning methods based on this synthesis in companies. The underlying theory and methods for organizational learning (Senge, 1990; Senge et al, 1994) represent a synthesis of existing fields -- system dynamics, action science, social psychology, group dynamics and the creative process.
In so doing we hope to encourage other to add to the body of practical tools for confronting "wicked messes." Existing problem-solving, decision-making and improvement approaches have not provided satisfactory outcomes for problems where dynamic complexity and behavioral complexity is high. We further believe that meaningful research must force theories into practical test. Traditional academic research is not likely to bridge the gap from theory to practice because it tends to ignore the complex challenges of change in real life organizations: developing practical tools, institutionalizing long-term individual and collective learning processes, and confronting the cultural and organizational barriers to learning in modern organizations. The research methods we have developed are based on three expected outcomes for field research activities: living examples for what is possible, improved learning tools and methods, and better theories to guide development of future tools and methods. We hope that all three outcomes will also be the aspiration of other researchers in organizational learning.
In order to promote learning in the situation we are studying, we base our field research process on an experiential learning cycle. The learning cycle provides guidance for project activities on two levels. First, within each phase of a field project, the learning cycle serves as a guide in moving from activities which are predominately based on observation to assessment, to design and to implementation. This requires managers to articulate the basis for their thinking and action. This deliberate process helps to slow down the typically frenetic pace of organizational activities so that data are captured, information is fed back, opportunities for reflection are created and learning is not left to chance.
Second, the OADI cycle helps to clarify the different phases of the overall project. Pre-project activities, building relationships and shared commitment, are the basis for future project activities. Project activities in the first phase focus more on observation and assessment - articulation and clarification of problem symptoms rather than seeking quick amelioration of problematic conditions. The second phase of project activities emphasizes assessment of symptoms and design of action, while the third phase focuses on implementation of what has been learned and broader diffusion of a learning and change process.
The Learning Center is still very much in its infancy, and the ideas and processes described in this paper will undoubtedly evolve substantially in the coming years, just as we have learned a great deal in the last four years. To date, we have achieved success in launching pilot projects that have helped partner companies. But we have also struggled in getting good quality documentation from some of our initial projects, in part because we did not understand the need for new research methods and roles, such as learning historians. Our overall research focus has become sharper. We have had the idea of managerial practice fields and testing of tools and methods based on the synthesis of different learning disciplines since we started. But, focusing specifically on confronting "wicked messes" is helping to sharpen our research goals and process.
The Spanish poet Antonio Macodo said, "Life is a path that you beat as you walk it." So, likewise, is learning about organizational learning. As we act and reflect, we clarify our thinking and increase our capacity for effective action. Hopefully this paper will help others do the same.
Notes
[1] Colleagues that have been directly working at the center include Daniel Kim, Janet Gould, Bill Isaacs, Fred Kofman, Ernst Diehl and Jeff Clanon. We have also received advice on the establishment, strategy and operation of the research center from Ed Schein, Ed Nevis, John Sterman, Richard Beckhardt, W. E. Deming, Chris Argyris, Ray Stata, Henry Jacoby, Thomas Magnanti, Thomas Malone, Charles Fine, Gabrial Bitran, Arie DeGeus, Joe Jaworski, Rita Cleary and Bill O'Brien acknowledge their contributions. [Back][2] Member companies include Amco, AT&T, Chrylser, EDS, Federal Express, Ford, GS Technologies, Harley Davidsen, Herman Miller, Hewlett-Packard, Intel, Motorola, Merck, National Semiconductor, Pacific Bell, Philips, Shell Oil, Texas Instruments, US West, and the Quality Management Network (a consortium of healthcare organizations). [Back]
[3] Senge, et. al. (1994) provides a good summary of many of
the tools
being used in Learning Center projects, as well as references on their origins
and development. [Back]
References
Ackoff, R. L. (1974), Redesigning the Corporate Future: A Systems Approach to Societal Problems, John Wiley & Sons, New York, NY.
Argyris, C. and D. Schon (1978), Organizational Learning: a Theory of Action Perspective, Addison-Wesley, Reading, MA.
Argyris, C., Putnam, R. and D. M. Smith (1990), Action Science, Jossey-Bass, San Francisco, CA.
Argyris, C. (1990), Overcoming Organizational Defenses, Prentice-Hall, New York, NY.
Argyris, C. (1982), Reasoning, Learning, and Action: Individual and Organizational, Jossey-Bass, San Francisco, CA.
Beer, M., Eisenstat, R. A., and Spector, B. (1990), "Why Change Programs don't Produce Change", Harvard Business Review, November-December, pp. 158-166.
Bohm, D. (1987), On Dialgoue, David Bohm Seminars, P.O. Box 1452, Ojai, CA, 09023.
Bono, E. de (1985), Six Thinking Hats, Penguin Books, New York, NY.
Booker, E. (1994) "Have you driven a simulated Ford lately?" Computer World, July 4, p. 76.
de Geus, A.P. (1988), "Planning as Learning", Harvard Business Review, March/April, p. 70-74.
Deming, W.E. (1982), Out of Crisis, MIT Center for Advanced Engineering Study, Cambridge, MA.
Dewey, J.( 1896), "The Reflex Arc Concept in Psychology", Psychological Review 3: 357-370, reprinted in Dennis, W (ed.) (1948), Readings in the History of Psychology, Appleton-Century-Crofts, New York, NY.
Diehl, E. and J. Sterman (1993), "Effects of Feedback Complexity on Dynamic Decision Making", working paper, Sloan School of Management, MIT, Cambridge, MA.
Dumaine, B. (1994), "Mr. Learning Organization", Fortune, October 13.
Forrester, J.W. (1961), Industrial Dynamics, MIT Press, Cambridge, MA.
Forrester, J.W. (1971), "The Counterintuitive Nature of Social Systems", Technology Review, January, 1971.
Forrester, J.W. (1969), Urban Dynamics, MIT Press, Cambridge, MA.
Forrester, J.W. and Senge, P.M. (1980) "Tests for building confidence in System Dynamics Models," System Dynamics, TIMS Studies in Management Sciences, Vol. 14, pp. 209-228.
Fritz, R. (1991), On Creating, Fawcett Columbine, New York, NY.
Fritz, R. (1987), The Path of Least Resistance, Fawcett Columbine, New York, NY.
Giancola, P. (1992) "A Systems Thinking Perspective of the Product Development Process: A Study of Business and Academic Collaboration," unpublished Masters Thesis, Sloan School of Management, Cambridge, MA.
Hogarth, R. (1987), Judgement and Choice, 2nd Edition , John Wiley, Chichester.
Isaacs, W. (1993), "Taking Flight: Dialogue, Collective Thinking and Organizational Learning", Organizational Dynamics, Fall.
Kampmann, C., and Sterman, J. (1994), "Feedback Complexity, bounded rationality, and market dynamics", working paper, MIT Sloan School of Management, Cambridge, MA.
Kim, D. and Senge, P. (1994), "Putting Systems Thinking into Practice", Systems Dynamics Review, Summer-Fall.
Kim, D. H. (1994), "Managerial Practice Fields: Infrastructure of a Learning Organization", working paper to appear in Learning Organizations: Developing Structures for Tomorrow's Workplace, New Leaders Press.
Kim, D. H. (1993b) "A Framework and Methodology for Linking Individual and Organizational Learning Applications in TQM and Product Development", unpublished Ph.D. dissertation, Sloan School of Management, Massachusetts Institutue of Technology, Cambridge, MA.
Kim, D. H. (1993a), "The Link between Individual and Organizational Learning", Sloan Management Review, fall, pp. 37-50.
King, J. (1993), "Learning to Solve the Right Problems: The Case of Nuclear Power in America", Journal of Business Ethics Vol. 12: 105-116.
Kofman, A. and P. Senge (1993), "Comunities of Commitment: the Heart of Learning Organizations", Organization Dynamics, Fall, 1993.
Kofman, F., Repenning, N., and J. Sterman, (1994) "Unanticipated Side Effects of Successful Quality Programs: Exploring a Paradox of Organizational Improvement" working paper D4390, System Dynamics Group, Sloan School of Management, MIT, Cambridge, MA.
Kolb, D. A. (1984), Experiental Learning: Experience as the Source of Learning and Development , Prentice-Hall, Englewood Cliffs, NJ.
Lewin, K. (1951), Field Theory in Social Sciences, Harper & Row, New York, NY.
Louis, M. R. and J. M. Bartunek (1992) "Insider/Outsider Research Teams: Collaboration Across Diverse Perspectives" Journal of Management Inquiry, Vol. 1, No. 2.
Meadows, D.L. (1982), "Whole Earth Models and Systems", in Co-Evolution Quarterly, summer.
Mirvis, P., and Berg (eds.) (1977), Failures in Organizational Development and Change , John Wiley & Sons, New York, NY.
Moorefield, R. and M. Losada (1995), "Organizational Learning and Leadership Development at EDS", The SystemsThinker, Pegasus Communications, Cambridge, MA, February, 8-11
Oliva, R. (1993), "Service Quality-Service Capacity Interactions: Framework for a Dynamic Theory," working paper D-43761-2 in Systems Dynamics Group, Sloan School of Management, MIT, Cambridge, MA.
Oliva, R. and Senge, P. M. (1993) Developing a Theory of Service Quality/Service Capacity Interaction," working paper, Center for Organizational Learning, MIT Sloan School, Cambridge, Ma. April.
Paich, M. and Sterman, J. D. (1993) "Boom, Bust, and Failures to Learn in Experimental Markets", Management Science 39, 12: 1439-1458.
Reason, J. (1990), "The contribution of latent human failures to the breakdown of complex systems", Philosphical Transactions of the RoyalSociety of London , (Series B) 327, pp. 475-484.
Richardson, G., Andersen, D., Maxwell, T. and T. Stewart (1994), "Foundations of Mental Model Research" paper given at the International Systems Dynamics Conference, Stirling, Scotland, July 11-15.
Rittle, H. and Webber, M. (1973), "Dilemmas in a General Theory of Planning" Policy Sciences, Vol. 4: 155-159.
Roberts, E. B. (1992) "Developing new products and services by listening to the voice of the customer" working paper 3455-92, Sloan School of Management, Cambridge, MA.
Roth, G. (forthcoming) "From Individual and Team Learning to Systems Learning", Managing So Organizations Learn, Cavaleri and Fearon (eds.), Blackwell Publishing Company.
Roth, G., Kleiner, A., Binkert, J., Fritzges, L., Girod, N., Siegel, E., and A. Thomas (1995), "Ford FN74 Program Learning History", working paper for Ford Motor Company, Center for Organizational Learning, Sloan School of Management, MIT, Cambridge, MA.
Roth, G. and A. Kleiner (1994) "Notes from Learning History Conversation", materials presented at Bretton Wood Conference, Bretton Woods, NH, June 27.
Sanday, P. (1979), "The ethnographic paradigm(s)", Administrative Science Quarterly Vol 24. pp. 482-93.
Schwartzman, H. B. (1993) Ethnography in Organizations, Sage Publications: Newbury Park, CA.
Schein, E. (forthcoming), "Organizational Sets," working paper in process, Center for Organizational Learning,Sloan School of Management, MIT, Cambridge, MA.
Schein, E. (1993), "On Dialogue, Culture and Organizational Learning", Organizational Dynamics, Fall.
Schein, E. (1987), Process Consultation, Vol II, Addison-Wesley, Reading MA
Senge, P.M. (forthcoming), "Leading Learning Organizations," forthcoming in The Leader of the Future, special tribute to Peter Drucker, Jossey-Bass.
Senge, P. M., (1994), "Moving Forward, Thinking Strategically About Building Learning Organizations" in Senge, P., A. Kleiner, C. Roberts, R. Ross, and B. Smith, The Fifth Discipline Fieldbook, Doubleday/Currency, pp. 15-47.
Senge, P. M., A. Kleiner, C. Roberts, R. Ross, and B. Smith (1994), The Fifth Discipline Fieldbook, Doubleday/Currency.
Senge, P. M. (1990), The Fifth Discipline: The Art and Practice of the Learning Organization, Doubleday/Currency.
Seville, D. and Kim, D. H. (1993), "Product Development Management Flight Simulator, Facilitator's Guide" working paper, version 2.08, Center for Organizational Learning, MIT Sloan School, Cambridge, MA.
Spradley, J. P. (1979), The Ethnographic Interview, Holt, Rinehart and Winston, New York, NY.
Stata, R. (1989), "Organizational Learning - the key to management innovation", Sloan Management Review, pp. 63-74.
Sterman, J. D. (1994a), "Learning in and about complex systems", Systems Dynamics Review, Vol 10 (2-3).
Sterman, J. D. (1994b) "The Improvement Paradox: Designing Sustainable Quality Improvement Programs" proposal funded by the National Science Program, Transformation to Quality Organizations, MIT Sloan School of Management, Cambridge, Ma.
Sterman, J. D. (1992) "Teaching Takes Off, Flight Simulators for Managment Education" OR/MS Today, October. pp. 40-44.
Sterman, J. D. (1985) "A Skeptics Guide to Computer Models," Proceedings of the International Systems Dynamics Conference, July.
Van Maanen, J. (1988), Tales of the Field, On writing ethnography , The University of Chicago Press, Chicago, MI.
Van Maanen, J. (1979), "The fact and fiction in organizational ethnography", Administrative Science Quarterly , Vol 24. pp. 539-50.
Walton, R. E. (1987), Innovating to Compete: Lessons for diffusing and managing change in the workplace , Jossey Bass, San Francisco, CA.
Yow, V. R. (1994), Recording Oral History, a practical guide for social scientists, Sage Publications, Thousand Oaks.
Figures
Figure 1. Territories defined by Dynamic and Behavioral Complexity
Dynamic Complexity
Low High
Low tame messes
problems
Behavioral
Complexity
High wicked wicked
problems messes
[Back]
Figure 2. Expected Outcomes of Research Activities
[Back]
Figure 3. OADI Cycle
[Back]
Figure 4. Field Project Research Process as OADI Cycle
[Back]
Phase 0 Building relationships, understanding, and knowledge while
searching for project opportunity - PRE-PROJECT ACTIVITIES
Phase 0: Overall Goals Events Outcomes
Determine match of Annual OLC meetings, Center membership
company interests with meetings among
research territory and research center
organizational learning staff and company
approach managers
Develop understanding & 5-day "Core Course" Knowledgeable &
capacity in Attendance committed company
organizational learning people
Project sponsorship by Meetings, Request for project
senior line manager presentations &
visits to other
companies
Initial inquiry and Stakeholder Project engagement
observation of contextual interviews clinic document
project conditions
[Back]
Phase 1 Initiating learning project activities, developing core team and formulating hypotheses - OBSERVATION AND ASSESSMENT
Phase 1: Overall Goals Events Outputs
Definition of project and Project Engagement Establish
associated business goals Clinic interpersonal
relationships and
understanding of one
another's goals upon
which to base project
activities
Project Proposal and
Budget
Research project "Research Proposal Research Proposal
positioning Workshop" Document
Faculty Interest
Core team definition and Meetings, Capacity Development;
development interviews, Problem Articulation
observations, and & Representation
exercises
Observation and interview Ongoing field Phase 1 Learning
data on organizational research including History Document
dynamics, cultures, and interviews and
stakeholder perspectives observations
Review and Phase 1 Research
positioning of Report
research questions
in relevant
literature
Articulation and testing Pilot learning lab Learning Intervention
of preliminary hypotheses or some other (Learning Lab) Design
preliminary learning
intervention design
and test
Engagement of top company Meeting of project Large System Change
management in learning team to plan how to Proposal
process and substantive inform and involve
inquiries for improving top company
business performance management
[Back]
Phase 2 Pilot testing hypotheses and spreading learning opportunities - ASSESSMENT AND DESIGN
Phase 2: Overall Goals Events Outputs
Positioning of project Project Assessment Project Plan & Budget;
activities and their Clinic Executive Support
associated business impact
Qualitative development Phase 2 Research Research Plan; Faculty
and testing of Seminar Support/Sponsorship
hypotheses; design and
testing of learning and
change processes by
managers and researchers
Implement action research Conduct: Learning Multiple and systemic
initiatives Labs, Dialogue, perspectives
Curriculum program
Deepen understanding, and Meetings, Capacity Development;
potential extension, of interviews, Problem Articulation
core team observations, and & Representation
exercises
Observation and interview Ongoing field Phase 2 Learning
data on organizational research including History
dynamics, cultures, and interviews and
stakeholder perspectives observations
Phase 2 Research Report
Articulation and design Meetings with Large System Change
of diffusion and change executive management Plan
process
[Back]
Phase 3 Broader diffusion and transfer of knowledge and learning - IMPLEMENTATION
Phase 3: Overall Goals Events Outputs
Link specific business Project Diffusion Business and Strategic
benefits and desired Clinic Plan which includes and
behavior and policy integrates learning
changes with learning processes; partnership
interventions support
Quantitative testing of Phase 3 Research Research Program;
hypotheses; design of Seminar faculty research
learning diffusion involvement; system
processes by managers and dynamic model
researchers development
Wider diffusion of Curriculum Program Capacity Development;
learning process and Learning Design of learning
Intervention processes into work;
Replication Learning Manuals;
Business Process Designs
Understanding and Meetings, Phase 3 Research
comparison of learning and interviews, Report; Project-based
formal testing of observations, research papers
hypotheses benchmarking and
quantitative
measurement
Observation and interview Ongoing field Phase 3 Learning History
data on organizational research including
dynamics, cultures, and interviews and
stakeholder perspectives observations
Operationalize theoretical design meetings for Changes in job
findings into work, job planning changes descriptions, business
and business process process redesign,
redesign multimedia learning
tools
Design of new learning Meetings with other Initiating learning
programs parts of organization process cycle in other
parts of company
[Back]