FYI.

This story is over 5 years old.

Tech

Rock Climbing, Jet Planes, and the Delicate Art of Technological Redundancy

Why systems fail in unanticipated ways.

Systems fail. Even very simple ones—the staggering complexity of a fly-by-wire control system in a modern jetliner seems especially prone to failure, but so does the gnarly old timing chain in my pickup truck. If we were to add up all of the timing chain failures in all of the 1990 Toyota pickups in the world and stack them against all of the (unprovoked) flight control system failures in the world, timing chains would win (or lose) and it won't even be a contest. Which is good—no one's ever died because a failed timing chain caused a pickup truck to drop out of the sky.

Advertisement

It's an apples and oranges comparison, sort of. The airplane is on a regular maintenance and inspection schedule, for one, while my truck is months past due for an oil change. Secondly, timing chains in pickup trucks are not redundant. There's no other timing chain ready to take over. An aircraft, however, is from the ground up a marvel of redundant systems engineering. A hydraulic system might be replicated three or more times in a single aircraft, basically requiring a complete severing of the jet's fuselage to completely fail (which has happened).

System redundancy is an under-appreciated artform. This is a lesson I learned not from timing chains or jet planes, but from one specific and increasingly popular misuse of climbing equipment.

I'm currently learning a somewhat esoteric variety of climbing known as roped soloing. This is more or less what it sounds like: self-sufficient, self-contained, and reasonably well-protected—compared to unroped free climbing, anyhow—climbing above heights from which a complete groundfall is likely to result in death or serious injury. Its practice is enabled by devices called ascenders, the basic function of which is to allow rope to pass through unimpeded in one direction, and to stop it from moving in the other direction. It's basically a pulley with sharp teeth set at an angle.

What makes ascender-based soloing interesting is that there aren't really devices expressly designed for this purpose. Ascenders are properly meant for aid climbing (perhaps climbing directly up a rope itself) and shuttling gear around mountainsides and rock faces—they can hold weight, but they're not meant to take the shock of a high-velocity fall. A good ascender will most likely hold in a fall, but rope soloing lore is rife with stories of the devices failing, often because something's interfered with the mechanism without the climber realizing, or because the thing was incorrectly set up in the very first place. In soloing, there's no other person or persons there with you to verify that a climbing protection system is correctly implemented.

Advertisement

Systems that were thought to be safely redundant are unraveled by the unforeseen thing and its own unforeseen web of connections

One device is particularly notorious: the Petzl "Mini Traxion." This is what I have and it has a potentially disastrous downfall (pun intended): it pops open sometimes. It's meant to do that, but if a climber were to be using the Mini for a purpose it was not meant for, like soloing, they might not immediately realize the danger (which is easily mitigated).

Petzl has a very stern warning to this effect, advising that climbers using it for soling should use a backup device of a different variety with a different underlying mechanism. Which is a general suggestion, actually: use two devices that operate differently so that if one device fails for a particular reason, the other device will not fail for that same reason.

At the moment, I use a really pain-in-the-ass redundancy system in which I attach myself to loops tied into a second rope in addition to the ascender device on the primary rope. If the device fails, I'll take what's likely to be a more severe fall, but it will still be much better than the alternative and somewhat assuredly non-fatal.

The analog to this in, say, a large fly-by-wire airliner would probably be manual backup systems, in which hydraulic pressure can be regained by hand pumping and landing gear can be lowered and locked using only gravity. The crucial feature of redundancy is system independence. A failure in one system can cascade throughout that system, but if a backup is not connected to that system, it's safe. A low-tech backup (rope, hand pump) is a natural candidate.

Advertisement

Roped soloing is especially interesting in this respect because, unlike most everything else in rock climbing and life itself, there are many different ways of accomplishing system redundancy. It's like a puzzle of interconnected yet independent simple mechanical devices.

While there may be many different methods of implementing redundancy in my climbing system, they're not always obvious, or if they are obvious, it may not be obvious that they're really connected in some way that's not readily apparent (probably because it hasn't happened yet). The deeper explanation for my redundancy problem is the ease with which connections can be hidden. This is a simple feature of combinatorics and how the number of possible combinations of events and gear grows really fast and, as such, it becomes a lot easier to miss things. Math and experience go together really well here.

In the words of sociologist Charles Perrow, we live in a "normal accident environment." A single unforeseen combination can cause an entire cascade of failures through a system. Systems that were thought to be safely redundant are unraveled by the unforeseen thing and its own unforeseen web of connections. Systems within systems within systems. A NASA presentation on Perrow's normal accident theory explains:

System "unravelings" have an intelligence of their own: they expose hidden connections, neutralize redundancies, bypass firewalls, and exploit chance circumstances for which no engineer could reasonably plan.

So, how about my loop-based system? It seems cut and dried. I fall, the ascender fails, but I'm caught by a lanyard secured to a second rope. Whiplash aside, I'm fine.

This is the other face of redundancy, however, and what makes it such a surprisingly delicate art. It's not enough to stack independent systems on top of each other because redundancy itself is complexity. The loop scheme actually has an obvious failing: At regular intervals the climber is introducing a new situation in which the second rope must be accessed. It could be simple, but it might also involve some reaching and detouring, which introduces additional opportunities for falls, particularly falls that might occur in the gap between unclipping from one loop and clipping into the next loop. Would I be safer without this particular redundancy system? Probably not, but it illustrates how such a situation could occur.

This is all a novel problem of technology. Constantly, we're being enabled in new ways and, constantly, we're being placed within new systems of normal failure.

As for climbing, I'll amend the system proposed above in a simple and obvious way, which is suggested in a Petzl risk analysis. Rather than having a single lanyard attached to my backup loops, I have two lanyards. So, as I unclip one and move it to the next highest loop, the other lanyard stays attached. It's an obvious fix once you see it, but that's the fundamental downfall of all complex technological systems: once you see it. And once you see it, the mistake has probably already been made.