All iHeavy Newsletter

Newsletter 74 – Design For Failure

It may sound like a pessimistic view of computing systems, but the fact is all of the components that make up the modern Internet stack have a certain failure rate. So looking at that realistically, planning for a break-down so you can manage it better, is essential.

Failures in traditional datacenters

In your own datacenter, or that of your managed hosting provider sit racks and racks of servers. Typically an proactive system administrator will keep a lot of spare parts around, hard drives, switches, additional servers etc. Although you don’t need them now, you don’t want to be in a position to have to order new equipment when it fails.  That would increase your recovery time dramatically.

Besides keeping extra components lying around, you also typically want to avoid the so-called single point of failure. Dual power systems, switches, database servers, webservers etc. We also see RAID as sort of standard now in all modern servers as a loss of commodity sata drive is so common. Yet this redundancy makes it a non-event. We are expecting it and so design for it.

And while we are prudent enough to perform backups regularly and document the layout of systems, rarely is the environment in a traditional datacenter completely scripted. Although attempts to test backups, and restore the database may be common, a full fire drill to rebuild everything is rarer.

Failure in the Cloud

In the last decade we saw Linux on commodity take over as the internet platform of choice because of the huge cost differential as compared to traditional hardware such as Sun or HP.   The hardware was more likely to fail, but being 1/10th the price meant you could build redundancy in to cover yourself and still save money.

The latest wave of cloud providers are bringing the same types of costs savings. But cloud hosted servers, for instance in Amazon EC2 are much less reliable than typical rack mounted servers you might have in your datacenter.

Planning for disaster recovery we agree is a really good idea, but sometimes it gets pushed aside by other priorities. In the cloud it moves to front and center as an absolute necessity. This forces a new, more robust approach to rebuilding your environment with scripts documenting and formalizing your processes.

This is all a good thing as hardware failure then becomes an expected occurrence. Failures are a given, it’s how quickly you recover that makes the difference.

Book Review:

Cloud Application Architectures by George Reese
Originally picked up this book expecting a very hands on guide to cloud deployments, especially on EC2. That is not what this book is though. It’s actually a very good CTO targeted book, covering difficult questions like cost comparisons between cloud and traditional datacenter hosting, security implications, disaster recovery, performance and service levels. The book is very readable, and not overly technical.

All Business iHeavy Newsletter

iHeavy Insights 73 – It’s Easy

In the business of technology consulting, there are times when I’ve heard this statement.  It’s Easy!  Perhaps the single biggest thing I’ve learned through a decade and a half of consulting is, people use this phrase when they are feeling overly confident.

What do I mean by that?  Well it turns out in psychology there are all sorts things we communicate with our spoken language & body language.  Some of those things we aren’t even conscious of.  In the case of the statement “It’s easy” your first thought may be about all the intricate details that have yet to be ironed out, all the hiccups that may happen along the way.  Or you may just simply be thinking of Murphy’s Law that always seems to rear it’s ugly head at the worst time.

Truth is when you hear this statement, you may also be inclined to think of it as a statement of fact.  The person saying that they have reviewed all the facts and ascertained that task x is in fact a trivial one.

Of course one doesn’t want to be the naysayer either, but you can raise concerns while still acknowledging both sides.  My tack is first to repeat what the person said in more detailed language.  By reiterating all of the details, it can sometimes illustrate right there some hidden complexity and weaken the sense of triviality to the task at hand.

A Software Developer

A few years back I had subcontractor developer working on a project.  We went over some details about what changes needed to happen.  A web-based analytical tool needed some additional search functionality.  We went over how that search would index documents in the site.  The developer explained to me “That’s easy.  No problem”.  I was suspicious.

As development unfolded we hit a bump in the road.  Besides the database indexing, additional xml documents needed to be indexed in order for the search function to work properly.  That added quite a bit of additional complexity because the search solution developer had envisioned couldn’t deal with that xml data.

A Business Prospect

I was recently reviewing a contract with a prospect, and going over items and deliverables.   They explained that for the database portion we’ll just use Amazon RDS, instead of installing MySQL and configuring the server manually.  “This piece will be easy”.  Unfortunately using Amazon’s solution is still not  push-button  in any event.   These types of oversimplifications are fine if you’re working on a time and materials basis because the complexity of the project will unfold organically, and the process will educate everyone involved.  But if you are trying to do a fixed fee project, these can be a harbinger of trouble later on.


When you hear people say “That’s Easy” understand that they are only expressing their confidence, despite their assurances.  If you are not equally confident,  you’ll both need to discuss details until you reach a middle point.  If it scares you to hear someone say something is EASY, think of it as a warning flag that you are not both on the same page.  Then remedy the situation with ample communication.

BOOK REVIEW:  Satyajit Das – Traders, Guns & Money

Financial Times has very high standards, and with an endorsement by Gillian Tett on the back, you know you are on the right track to some excellent material.  Das’ expose explores the inside world of derivatives, the so-called WMD of the financial world.   Along the way you’ll enjoy wacky stories of rogue traders, $70,000 meals, LIBOR numbers, delusional thinking, and even more about financial risk.  It illustrates exactly why Warren Buffet said “You only find out who is swimming naked when the tide goes out.”