Considering proximate causes versus ultimate causes can help give us a sense of perspective. When it comes time to assign blame, it often isn’t as cut and dry as we’d like. Getting this sense of perspective might help us consider and formulate more constructive solutions rather than pointing fingers.
With the oil spill all over the news in the last weeks, its easy to see this finger point. What’s the proximate cause? Perhaps a faulty valve is to blame. If we look at larger causes of that, we see that regulation, oversight and checks and balances were missing. But why drill in such a hazardous extreme environment 1 mile down?
Ultimately the cause is of demand for oil. As a supply of easy oil dwindles, we search ever more remote locations for it. So in essence each of us plays a part in this demand for oil. Since complex technology and systems are inherently unstable, accidents are inevitable. They can be mitigated, but there always remains a degree of error. One might even argue that ultimate cause was an ever growing human population which needs energy to make food, which in turn is an inherent quality of living creatures. So says Jared Diamond in his book collapse, discussing the rise, and fall of some past societies.
Diamond’s book is also a great example of constructive thinking on this topic, as he provides examples of some societies which pulled back from that edge, and how they did it.
With the recent financial meltdown, a lot of fingers have been pointed too. Everyone is looking to find the bad guy, and upon whom to assign blame. But individuals or large financial firms have always taken risks. Their whole raison d’etre is to find more and creative ways of making money, off of fees, leverage, or so-called arbitrage. So that impulse is obviously nothing new. Lack of regulation, perhaps the government is to blame? True regulatory changes did usher in a new wave of gambling. Many would argue not enforcing capital requirements allowed firms like Lehman to get so overleveraged in the first place.
But economists argue that ultimately capitalism is prone to cycles of boom and bust. It also seems that those cycles are shortening. And ultimately the internet and computerization of trading has played a huge huge role in speeding up the global economic machine in ways that we have yet to understand or unravel.
Just as with these other complex systems we’ve described, computing systems have inherent complexity built into them. Many web-facing applications have a fleet of database servers, webservers, load balancers, and many many interlocking software components interacting in a myriad of ways.
When a site outage happens they typically coincide with a huge spike in traffic, which means some spike in popularity or attention or focus to your services and business. So it often happens to be the worst time for an outage. So the pressure mounts when such an outage occurs to find a root cause. Now obviously root cause analysis is important to help mitigate future problems, but we must be careful not to also assign blame too quickly along with that. It is a very human knee-jerk reaction, but may miss the bigger picture.
An infinite amount and variety of QA and testing can be performed, and still not model what will happen in the real world. Perhaps your users all started coming to your site because they wanted some specific piece of content that was suddenly indexed by google making your site the best result? Or maybe a new feature was rolled out, but enough testing was not done to find one particular bug. Or a developer added a new table without the proper index, making a heavily used part of the site an even heavier burden for the database.
Again as with the other scenarios we’ve described there are indeed root causes, but we also must consider the ultimate causes of complexity, and inherent failure built into such systems.
Proximate causes are in the details, they point to the specific event that triggered an outage, avalanche or other disaster. But they do not provide the entire picture. Only by considering the ultimate causes of and the complexity of the system as a whole along with proximate causes, can we can get perspective. It is from that vantage point that we may build more constructive solutions to not eliminate all risk, but at least to mitigate it.
BOOK REVIEW – Seth Godin – The Big Moo
Seth Godin edits this little book of insights, including essays and new ideas from such luminaries as Malcolm Gladwell, Mark Cuban, Tom Peters, and Amit Gupta.