Stories as a vehicle for learning from the experience of others

Senior software engineering positions command higher salaries than junior positions. The industry believes (correctly, I think) that engineers become more effective as they accumulate experience, and that perception is reflected in market salaries.

Learning from direct experience is powerful, but there’s a limit to the rate at which we can learn from our own experiences. Certainly, we learn more from some experiences than others; we joke about “ten years of experience” versus “one year of experience ten times over”, as well as using scars as a metaphor for these sometimes unpleasant but more impactful experiences. But there’s only so many hours in a day, and we may not always be…errr… lucky enough to be exposed to many high-value learning opportunities.

There’s another resource we can draw on besides our own direct experience, and that’s the experiences of peers in our organization. Learning from the experiences of others isn’t as effective as learning directly from our own experience. But, if the organization you work in is large enough, then high-value learning opportunities are probably happening around you all of the time.

Given these opportunities abound, the challenge is: how can we learn effectively from the experiences of others? One way that humans learn from others is through telling stories.

Storytelling enables a third person to experience events by proxy. When we tell a story well, we run a simulation of the events in the mind of the listener. This kind of experience is not as effective as the first-hand kind, but it still leaves an impression on the listener when done well. In addition, storytelling scales very well: we can write down stories, or record them, and then publish these across the organization.

A second challenge is: what stories should we tell? It turns out that incidents make great stories. You’ll often hear engineers tell tales of incidents to each other. We sometimes calling these war stories, horror stories (the term I prefer), or ghost stories.

Once we recognize the opportunity of using incidents as a mechanism for second-hand-experiential-learning-through-storytelling, this shifts our thinking about the role and structure of an incident writeup. We want to tell a story that captures the experiences of the people involved in the incident, so that the readers can imagine what is was like, in the moment, when the alerts were going off and confusion reigned.

When we want to use incidents for second-hand experiential learning, it shifts the focus of an incident investigation away from action items as being the primary outcome and towards the narrative, the story we want tell.

When we hire for senior positions, we don’t ask candidates to submit a list of action items for tasks that could improve our system. We believe the value of their experience lies in them being able to solve novel problems in the future. Similarly, I don’t think we should view incident investigations as being primarily about generating action items. If, instead, we view them as an opportunity to learn collectively from the experiences of individuals, then more of us will get better at solving novel problems in the future.

What Deming got wrong

One of my Father’s Day presents this year was The Essential Deming, an anthology of Deming’s shorter writings. I thoroughly enjoyed Deming’s Out of the Crisis and was looking to read more from him.

Reading this book, I was surprised to discover that Deming was opposed to workers training workers, which he considered a faulty practice. The most effective way to become an expert is through the apprenticeship model, where a novice works alongside an expert and directly observes how the expert does their work. That Deming would reject this model, and would believe that an outsider could more effectively train a worker than someone who actually does the day-to-day work is, frankly, bizarre.

Deming also asserted that the most effective teachers at a university were the professors that were the best researchers. This also seemed to me to be an extremely odd claim, and one I’m extremely skeptical of.

Deming had a deep understanding of systems thinking, and the importance of holistic, expert judgment (he used the term leadership) over chasing metrics, and that shines through in this book. However, while Deming seemed to recognize the value of expertise, he did not seem to have a good understanding of how people acquire it.

Postmodern engineering

When I was younger, I wanted to be a physicist. I ended up majoring in computer engineering, because I also wanted gainful employment, but my heart was always in physics, and computer engineering seemed like a good compromise between my love of physics and early interest in computers.

I didn’t think too deeply about the philosophy of science back then, but my beliefs were in line with the school of positivism. I believed there was a single underlying reality , the nature of this reality was potentially knowable, and science was an effective tool for understanding that reality. I was vaguely aware of the postmodernist movement, but mostly by reading about the Sokal hoax, where the physicist Alan Sokal had demonstrated that postmodernism was nonsense.

Around the same time, I also read To Engineer is Human: the Role of Failure in Successful Design by the civil engineering researcher Henry Petroski. The book is a case study on how civil engineering advanced through understanding structural failures. Success, on the other hand, teaches the engineer nothing.

Many years later, I find myself operationally a postmodernist (although constructivist might be a more accurate term). When I study how incidents happen, I no longer believe that there is a single, underlying reality of what really happened that we can access. Instead, I believe that the best we can do is construct narratives based on the perspectives of the different people that were involved in the incident. These narratives will inevitably be partial, and some of them may conflict. And there are things that we will never really know or understand. In addition, contra Petroski, I also believe that we can learn from studying successes as well as from studying failure.

I suspect that most engineers are steeped in the positivist tradition of thinking as well. This change in perspective is a big one: I’m not even sure how my own thinking evolved over time, and so I don’t know how to encourage this shift in others. But I do believe that if we want to learn as much as we can from incidents, we need to work on changing how our fellow engineers think about what is knowable. And that’s a tall order.

Not apprenticeship!

Mark Guzdial points to an article by Nicholas Lemann in the Chronicle of Higher Ed entitled The Soul of the Research University. It’s a good essay about the schizophrenic nature of the modern research university. But Lemann takes some shots at the notion of teaching skills in the university. Here’s some devil’s advocacy from the piece:

Why would you want to be taught by professors who devote a substantial part of their time to writing projects, instead of working professionals whose only role at the university is to teach? Why shouldn’t the curriculum be devoted to imparting the most up-to-the-minute skills, the ones that will have most value in the employment market? Embedded in those questions is a view that a high-quality apprenticeship under an attentive mentor would represent no loss, and possibly an improvement, over a university education.

Later on, Lemann refutes that perspective, that students are better off being taught at research universities by professors engaged in research. He seems to miss the irony that this apprenticeship model is precisely how these research universities train PhD students. For bonus irony, here was the banner ad I saw atop the article: skills-webinar

Software Analysis

At McGill University, the computer engineering program evolved out of the electrical engineering program, so it was very EE-oriented. I was required to take four separate courses that involved (analog) circuit analysis: fundamental of electrical engineering, circuit analysis, electronic circuits I, and electronic circuits II.

I’m struggling to think of what the equivalent of “circuit analysis” would be for software engineering. To keep the problem structure the same as circuit analysis, it would be something like: Given a (simplified model of?) a computer program, for a given input, what will the program output?

It’s hard to imagine even a single course in a software engineering program dedicated entirely to this type of manual “program analysis”. And yet, reading code is so important to what we do, and remains such a difficult task.

Training is a dirty word

Two posts caught my eye this week. The first was Anil Dash’s The Blue Collar Coder, and the second was Greg Wilson’s Dark Matter, Public Health, and Scientific Computing. Anil wrote about high school students and Greg spoke about scientists, but ultimately they’re both about teaching computer skills to people without a formal background in computing. In other words, training.

In the hierarchy of academia, training is pretty firmly at the bottom. Education at least gets some lip service, being the primary mission of the university and all. But training is a base, vulgar activity. And it’s a real shame, because the problems that Anil and Greg are trying to address are important ones that need solving. Help will need to come from somewhere else.