Organisational surprise and its relevance to project management

Introduction
As humans, we tend to believe that events and processes will unfold in the way we expect them to.  Unfortunately things rarely go according to our expectations and we end up being, well  surprised! But it isn't just individuals, entire organisations can be caught unawares by events. In this post I draw on a paper entitled,Clues, Cues and Complexity: Unpacking the Concept of Organisational Surprise, to elaborate on the different ways in which surprises can crop up in organisational settings and in particular, in projects.  My focus is on the latter because the temporary and one-off nature of projects seems to render them particularly prone to surprises.
Organizational surprise
The authors define organizational surprise as, any event that happens unexpectedly or any expected event that takes an unexpected shape.  One does not have to look far to see examples of organizational surprises. It is almost certain that any experienced project manager would have encountered a number of surprises in the course of his or her professional work. These could be unforeseen events (such as a project sponsor leaving for greener pastures) or unexpected twists and turns in what ought to have been a straightforward process(such as a software upgrade turning out to be more complicated than expected).
The ubiquity of organizational surprises begs the question as to why we are still surprised by surprises. The short answer is that this happens because we tend to overestimate our ability to control the future.  The authors suggest that we would be better served by regarding surprise as an inherent property of all open systems  (which includes entities such as organisations and projects). After living through the consequences of many over-optimistic managerial actions (both, my own and those of others), I would have to agree.
Classifying surprise  a typology of surprises
Nothing I have said so far would be surprising to readers: indeed, project management is largely about managing uncertainty and all project managers know that. What might be new, however, is a classification of surprises proposed by the authors of the paper. I have hinted at the classification in the previous section where I gave one example each of a surprising event and a surprising process. It is now time to generalize this to a typology of surprises.
The authors classify surprises along two dimensions:
Issue  An issue can occur in one of two ways:
When something unusual happens.
When something that usually happens does not happen.
It is important to note that although the term issue has negative connotations in project management parlance, it is used here in a neutral sense  i.e. issues can be either positive or negative.
 Process  a process is a chain of related events that unfolds in an unusual manner. For example, when an ATM cash withdrawal fails because the machine does not have enough money left to process the withdrawal.
From the perspective of surprise, issues and processes can be either expected or unexpected. This gives us the four categories illustrated in Fig 1.
 
Figure 1: A typology of surprises
Let's take a tour of the four categories
1. Expected issue and process: This is the zone of predictability where one-off events tend to go as foreseen.  An example of an expected issue that was successfully dealt with through planning was theY2K problem. Another example is provided by (successful) risk management activities that are triggered when a foreseen risk eventuates.
2. Unexpected issue, expected process:  This is where a surprising issue occurs, but the consequences follow are expected. An example this would be a chance occurrence (say, a project team member on a troubled project stumbles on a novel technique that saves development time), and this leads to the project being completed within time and budget (expected process following the event).
3. Expected issue, unexpected process:  This occurs when an expected event evolves in an unexpected way  i.e. leads to a surprising process. A common example of this in a project environment is when a  front-end project  decision unfolds in unexpected ways.   Another common example is an organizational change that has unintended consequences.
4. Unexpected issue, unexpected process: This is a situation where both the event and the processes around it are counter to  conventional wisdom. In this case, those involved need to understand, or make sense of the situation and hence the term sensemaking crisis. An example of this is when project managers fail to anticipate factors that turn out to have a major influence on the way their projects evolve.  One could argue that many high profile project failures were the result of such crises.  The Denver Baggage Handling System and the Merck Vioxx affair are good examples. In both cases, the projects failed because those responsible failed to react to certain events that changed the trajectory of the projects irrevocably.
Let's now take a brief look at the usefulness of this classification.
Coping with surprise
Managers expend a great deal of effort in attempting to predict surprises and hence corral them into the zone of predictability (Reminder: this is the bottom left quadrant in Figure 1).  As mentioned earlier, this is difficult because organisations are open systems, and novelty is an inherent property of such systems. The main implication of this is that surprises in quadrants other than the zone of predictability cannot be foreseen.  So, instead of worrying about predicting surprises, project and program managers would do better by focusing their efforts on creating an environment that enables team members to cope with nasty surprises and take advantage of good ones.
What might such an environment look like?
This question is best approached via a related question: what are the qualities displayed by project teams that are able to cope with surprises?
Here are some essential ones that are mentioned in the paper:
1.Vigilance / problem sensing a deep awareness of the project environment, with the ability to sense any changes in it.
2.Resilience  the capacity to adjust to changes in the internal and external environment.
3.Ability to improvise the ability to respond to the unexpected by devising appropriate courses of action under pressure
The striking thing about these qualities is that they are impossible to create or engender by management fiat: teams will not improvise unless they feel empowered to, nor will they be resilient or vigilant unless they are intrinsically motivated to be so.These characteristics are emergent in the sense that they will be displayed spontaneously by teams that are in a frame of mind that comes out of being in  the right environment.
The primary task of a project manager, or any manager for that matter, is to create such a  holding environment that provides psychological safety to the team and encourages rational (or open) dialogue between all project stakeholders (yes, including project sponsors). I won't elaborate on these terms here  since they are dealt with at length in the articles that I have provided links to in the previous sentence.
Different types of surprise require different approaches
Having the right environment is the key to dealing with all four kinds of surprises. However, even within such an environment, it is important to note that different types of surprises have to be tackled in different ways. In particular:
1.Predictable surprises are best tackled through traditional management approaches (as discussed in PMBOK, for example). In view of the prevalence of such approaches, I should perhaps emphasise again that they work only for a small subset of all possible surprises (only those that lie in the first quadrant)
2.Surprising events and surprising processes are best dealt with by the people who are at the coalface of the problem since they are intimately familiar with the context and history of the problem.
3.Sensemaking crises are best handled by collaborative problem solving approaches such as Dialogue Mapping.
The above yet again underscores the importance of the creating the right environment, for although predictable surprises can be tackled through traditional approaches to project management, those that lie in the other three quadrants cannot.
Conclusion
A fact of organizational life is that project managers are often caught unawares by unforeseen events and their dynamics. In this post, I have summarized a typology of organizational surprises and have elaborated on its relevance to project management.  I have also briefly discussed the ways in which different types of surprises can be tackled, emphasising that the key to tackling surprise lies in creating an environment that provides psychological safety and encourages open dialogue.
In closing, I reiterate that projects and organisations are open systems, and surprises are characteristic of such systems. The biggest surprise, therefore, is that we are continually surprised by some of the events and processes that occur within them
