AirBNB didn't have to fail

Today part of Amazon Web Services failed, taking down with it a slew of startups that all run on Amazon’s Cloud infrastructure. AirBNB was one of the biggest, but also Heroku, Reddit, Minecraft, Flipboard & Coursera down with it. Its not the first time. What the heck happened, and why should we care?

1. Root Cause

The AWS service allows companies like AirBNB to build web applications, and host them on servers owned and managed by Amazon. The so-called raw iron of this army of compute power sits in datacenters. Each datacenter is a zone, and there are many in each of their service regions including US East (Northern Virginia), US West (Oregon), US West (Northern California), EU (Ireland), Asia Pacific (Singapore), Asia Pacific (Tokyo), South America (Sao Paulo), and AWS GovCloud.

Today one of those datacenters in the Northern Virginia region had a failure. What does this mean? Essentially firms like AirBNB that hosted their applications ONLY in Northern Virginia experienced outages.

As it turns out, Amazon has a service level agreement of 99.95% availability. We’ve long since said goodbye to the five nines. HA is overrated.

2. Use Redundancy

Although there are lots of pieces and components to a web infrastructure, two big ones are webservers and database servers. Turns out AirBNB could make both of these tiers redundant. How do we do it?

On the database side, you can use Amazon’s multi-az or alternately read-replicas. Each have different service characteristics so you’ll have to evaluate your application to figure out what will work for you.

Then there is the option to host mysql or Percona directly on Amazon servers yourself and use replication.

[quote]Using redundant components like placing webservers and databases in multiple regions, AirBNB could avoid an Amazon outage like Monday’s that affected only Northern Virginia.[/quote]
When do I want RDS versus mysql? Here are some use cases for RDS versus roll your own MySQL.

Now that you’re using multiple zones and regions for your database the hard work is completed. Webservers can be hosted in different regions easily, and don’t require complicated replication to do it.

3. Have a browsing only mode

Another step AirBNB can take to be resilient is to build a browsing only mode into their application. Often we hear about this option for performing maintenance without downtime. But it’s even more valuable during a situation like this. In a real outage you don’t have control over how long it lasts or WHEN it happens. So a browsing only mode can provide real insurance.

For a site like AirBNB this would mean the entire website was up and operating. Customers could browse and view listings, only when they went to book a room would the encounter an error. This would be a very small segment of their customers, and a much less painful PR problem.

Facebook has experience intermittent outages of it’s service. People hardly notice because they’ll often only see a message when they are trying to comment on someone’s wall post, send a message or upload a photo. The site is still operating, but not allowing changes. That’s what a browsing only mode affords you.

[quote]A browsing only mode can make a big difference, keeping most of the site up even when transactions or publish are blocked.
[/quote]

Drupal, an open source CMS system that powers sites like Adweek.com, TheHollywoodReporter.com, and Economist.com uses this technology. It supports a browsing only mode out of the box. An amazon outage like this one would only stop editors from publishing new stories temporarily. A huge win to sites that get 50 to 100 million with-an-m pageviews per month.

4. Web Applications need Feature Flags

Feature flags give you an on/off switch. Build them into heavy duty parts of your site, and you can disable those in an emergency. Host components multiple availability zones for extra peace of mind.

One of our all time most popular posts 5 Things Toxic to Scalability included some indepth discussion of feature flags.

5. Consider Netflix’s Simian Army

Netflix takes a very progressive approach to availability. They bake redundancy and automation right into all of their infrastructure. Then they run an app called the Chaos Monkey which essentially causes outages, randomly. If resilience from constantly falling and getting back up can’t make you stronger, I don’t know what can!

Take a look at the Netflix blog for details on intentional load & stress testing.

6. Use multiple cloud providers

If all of the above isn’t enough for you, taking it further you’d do as George Reese of enstratus recommends and use multiple cloud providers. Not being beholden to one company could help in more situations than just these type of service disruptions too.

Basic EC2 Best Practices mean building redundancy into your infrastructure. Multiple cloud providers simply take that one step further.

Read this far? Grab our newsletter on scalability and startups!

31 Essential Blogs for Startups & Scalability

So many blogs, so little time! Here’s our list of the best we’ve found. Currently our favorite reader is Pulse pictured left. Starting to play around with flipboard too.

Nuts & Bolts Technical

Slashdot
One of the original tech blogs, that still covers lots of breaking news, and difficult topics. Very technical, with probing commentary. Beware the actual comments though, as they’re often full of immature and childish rants.

Planet Mysql
An aggregator of many MySQL blogs, it hits on topics from benchmarking, and advanced tuning, to new technologies on the horizon. Drupal and LAMP topics are often also covered.

mysql performance blog
Percona’s technical blog never disappoints. There are endless posts about a myriad of topics related to deploying, tuning and optimizing MySQL and all it’s variants.

Hacker News
You may not like Paul Graham, he’s easy not to. But his YCombinator News site is an awesome collection of always surprising technical topics that are sure to keep you busy.

Netflix Tech Blog
You might have read about Chaos Monkey before, that

Our very own Scalable Startups
You’re already reading us regularly of course! Why not grab our newsletter?

Programmable Web
Mashups & APIs. What more do you want? Very cool stuff here.

Also take a look at our best of compilation.

Business & Economics

HBR Blog
If you’ve ever read Harvard Business Review, you know how in depth and on point the material is. More thorough discussions than many other blogs, and excellent discussions in the comments.

Marginal Revolution
Tyler Cowen’s endlessly interesting and provocative take on the world through the eyes of economics. Like using science to analyze and solve the worlds ails, this blog always has a reasoned take on things.

NPR Planet Money
I’ve been listening to this podcast religiously since the financial crisis of 2008. It continues to intrigue and educate me in ways that college finance never did. You’ll learn a lot.

Bloomberg Businessweek
BBW despite it’s name is like Wired back in the 90’s before it got taken over by Conde, and the cutting edge writers and risk takers left. That’s right this magazine is full of analysis, creativity, and color. It’s what you’re looking for in a print magazine. One of my favorites.

Inc. Magazine
Real articles for your small business needs today. Thoughtful and topical.

Forbes Magazine
Banking, finance, politics, news.

You might also check out our Scalable Startups newsletter archives.

Venture

A VC
Fred Wilson’s iconic blog is always on the cusp, with a thoughtful and participating audience of readers.

Infochachkie
John Greathouse is a VC with a very readable blog on startups and investing.

Chris Dixon
This guy invested in tons of great startups that are household names now. With a very readable blog to match, he’s a man with ideas that we all benefit from.

Springwise
As they call it, your “Essential Fix of Entrepreneurial Ideas”.

Feld Thoughts
Brad Feld is another big VC with an excellent blog on topics relevant to Venture Capital & Startups.

Social

Andrew Chen
Consumer internet, metrics, and user growth. Brilliant idea guy. I learn from this guy’s blog everytime I check it.

Problogger
The smarties behind the book of the same name, this is essential reading for bloggers who wanna make a dent in the world.

Blog Tyrant
How to build successful blogs that make real money. Learn from Ramsay Taplin who’s done it already. Whether your blog sells products, widgets or services, there’s stuff for you here.

Kissmetrics Marketing Blog
Very good stuff on marketing, twitter, facebook and all the other good social topics.

Mixergy Blog
Business tips & startup advice with a bent towards marketing and social.

Mark Schaefer Marketing
Mark’s the brains behind the great book Return on Influence which we reviewed. His Businesses Grow blog is full of helpful ideas and insights.

Figaro Speech
You may have read my review of Word Hero and seen the earlier review of Thank You For Arguing. His blog is a real gem, extending on the wonders and lessons of word hero, you’ll be writing witty and memorable one-liners and titles that will go viral tomorrow!

Industry

Gigaom
Om Malik started out writing about the bandwidth boom and bust of the 2000’s. His blog has grown wildly to cover the industry as a whole, and contrary to the stuff you get on business insider, this is quality journalism.

AllThingsD
Another industry site with a great selection of journalists writing on the internet & startup industries.

ReadWriteWeb
Another excellent industry blog with slightly overlapping coverage to gigaom and allthingsd, but worth scanning each of them for different perspectives.

Venturebeat
Possibly a bit more venture and investment oriented than the others, but still mainly an industry coverage blog site.

Entrepreneur
Slightly more focus on business, and entrepreneurs, but also internet & startup industry topics.

Adweek
Trying to broaden my horizons by adding this one into the mix. Some very interesting topics, and plenty of overlap with internet industry and startups.

If you read this far, grab our newsletter!

The Art of Resistance

Sometimes, you have to be the bad guy. Be resistant to change. Here’s a story about how stubbornness pays off. As we’ve written about before A 4 letter word divides Dev & Ops.

I had one experience working as the primary MySQL DBA for an internet startup. Turns out they had Oracle for some applications too. And another DBA just to handle the Oracle stuff.

So it came time for Oracle guy to go on vacation. Suddenly these Oracle systems landed on my shoulders. We reviewed everything in advance, then he bid his goodbyes.

Almost as soon as he was out the door I started getting requests to change things.
“Oh we have to add this field”, or “oh we’d like to make this table change”.

I resisted enough to hold off development for a week.

“We’re now getting more heat from the CTO. Apparently certain pages on the site don’t save some very important content properly. It’s costing the business a lot of money. We need to make this change.”

My response –

[quote]No I can’t sanction this. If you want to do it, understand that it could well break Oracle’s replication.[/quote]

Oracle’s multi-master replication notoriously requires a lot of baby sitting. We held off for a few more days, and the Oracle DBA returned from his much needed vacation.

When discussing it afterward he said…

[quote]Am very glad you didn’t change those fields in the database. It would indeed have broken replication and caused problems with the backups![/quote]

Moral of the story…

  • apply the brakes around tight turns
  • don’t be afraid to say, we’re not going to do this before testing
  • you may have to say – we’re not going to do this at all…

The four-letter-word dividing Dev and Ops

devops divide

What’s that word?  RISK

Operations teams are tasked with stability and uptime. That means working against change, limiting or slowing it down where possible.

Developers are tasked with features and delivering business solutions. For that an ORM layer seems appealing for example. It speeds up & simplifies coding. At the same time it eliminates database drudgery.

For ops who are tasked with uptime, an ORM is a force against scalability. I’ve outlined five things toxic to scalability. They work against performance.

The question remains – do devops folks solve the problem?

Consider the banking crisis

Bankers are tasked with making money for their shareholders. To do this they innovate with financial products. Though you may argue they are unscrupulous at times, capitalism and shareholder value drive them to find profit.

Meanwhile the government’s job is to provide a level playing field.  They enact rules, regulate and provide oversight and auditing. As with operations, this is a conservative role, that avoids risk, and seeks stability, growth and avoidance of recessions and depressions.

These tradeoffs exist in many disciplines. The trick is how we find the balance.

There is an equally interesting question of decoupling in internet architectures. I’ll write a future piece on similar parallels I see in the economy at large.

Business lessons from brokering a half million dollar deal

I’ve had my Manhattan loft in the market for some six months now. After dropping the asking price a couple of times, buyers are finally circling, and I’m close to finalizing a deal.  In that time however, I’ve learned some big lessons.  Real Estate, is the epitome of cutthroat capitalism and as such has some very potent lessons to teach about business.

Understand Interests – Two Sides to Every Argument

Brokers have an interest – to make their fee;  buyers have an interest – to lower the selling price, sellers have an interest to get the highest selling price.  On the surface this may seem self evident.  But dig a little deeper and you’ll learn more. Brokers are somewhat aligned with buyer and seller, but like a cab driver preferring lots of short trips, their incentive is to do more deals. Fewer deals with higher margins would actually cost them. A 1%-2% higher selling price may be significant to your bottom line but meaningless to theirs.

[quote]Business Lesson 1 – Pay close attention to each party’s interests and you’ll understand the negotiating dynamic more clearly.[/quote]

In brokering, like war, no one plays fair

Each industry has it’s own ugly side. No one calls back. No one emails back. There are no pleasantries. In some businesses where credibility and reputation are built over time, people do care–little bit. But in brokering a large deal they don’t.  Silence means “Hey, thanks for going out of your way to give us an hour of your time even though we were late.  We don’t care.  We no longer need you & we’re not gonna respond.”  Got it, heard you loud and clear!

[quote]Business Lesson 2 – Be prepared for selfishness and deception. In some cases watch out for cheaters and even fraud.  Don’t let this color all your interactions, but be aware that some parties may not have the same scruples as you do and some may even break the law. [/quote]

Markets are bigger than you are

You may think your house or apartment has some intrinsic value, based on renovations or the time you’ve held it, or some other more personal metric. However the market is ultimately what sets the price of the asset. Market trends can swing wildly and irrationally so be prepared to “get real”, as they say in the business.

[quote]Business Lesson 3 – Market forces set the price of a good or service. The door swings both ways so when there is a greater supply be prepared for the price to go down, and when location, asset or skills are in great demand, be prepared for the price to go up.[/quote]

Buyers move on emotion; facts get you only so far

Salesmen tell stories. It’s their job to engage, stir passions, get people to move. Sometimes these yarns are taller than you or I might like, but if people knew exactly what they wanted, brokers wouldn’t exist. This occurs across the spectrum in sales.

[quote]Business Lesson 4 – Pay less attention to the facts and arguments a salesmen presents to you. Only the underlying interest of each party will tell a clear story of motivations. Weigh that along with the market appetite and you should have a good idea of the right price.[/quote]

Protect your interests and time

Over the months I’ve been on the market, I’ve had countless brokers try to sell me their services. An exclusive seller agreement is valuable to a broker as it allows them to control the process and control the fee distribution. One broker made an appointment to come by with their buyer over a holiday weekend. Following up at the appointed time, they explained that they were running late though frankly the broker sounded hung over. Upon arrival they brought no buyer. Seeing my irritation, they quickly fabricated a story about a phantom buyer and their specific requirements. After wasting a few hours of my time and breaking up my holiday weekend, I never heard from them again. Not fun.

[quote]Business Lesson 5 – Qualify your prospect before you grant them your time. There are many parties who can and will waste your time if you’re not careful. Be diligent about weeding them out before clearing your schedule for them.[/quote]

Business plans are bunk

I meet a lot of people in NYC. Friends, colleagues, small business owners, people with a great idea, artists, musicians, people with a dream, people who hear the startup story, namely the Google, Zynga, Apple, Facebook, or Amazon story.

Sometimes I’ll hear:

“I’m working on a business plan.”

“Why?” I ask.

“I want to make my business grow and get investment money.”

Then I think about this a little, and consider if a business plan would have ever helped me.  Would it help a deli owner run a deli? Or a laundry run a laundry?

[quote]The essence of running a large business is the day-to-day of running a small business.[/quote]

Let’s consider a less savory business – drug dealers. Their business may not be one we respect, but let’s consider the economics behind the scenes.

  • They sell people a product they want
  • They sell at a price people are willing to pay
  • If supply is low they get more to meet demand

That’s a business plan in a nutshell. If you’re not selling any product, that means people don’t want it. It may not be as valuable to the world as it is to you. If it’s valuable, people will buy. That’s the measure of good ideas. When people aren’t buying, they’re voting with their wallets.

Entrepreneur and blogger, Andrew Chen says:

Business models are a commodity. The answers are all obvious.

Sometimes all this talk of business plans sounds like therapy for people who don’t have a successful business.

The hard truth is if people aren’t buying they don’t want the product. Change it or tweak it until they do. A business plan won’t help with that.

Why the Android ecosystem is broken

Six months ago I got this crazy idea. Why not leave the mothership? Give up on iPhone and try Android. This is what being tech-agnostic is about, I thought––not being wedded to a single platform. Besides, the iPhone’s digital keypad just wasn’t working for me.

I got a monthly Boost Mobile plan, which uses Sprint. Service was stellar and I mean really good. I could call anywhere and always had a signal, even inside all these pre-war buildings you find in downtown Manhattan. How is this possible I thought? Service is one thing, but thats where the fun ends. A few months into Android hyperspace and I find myself grappling with a system that just doesn’t seem to understand what users want.

Shock and Awe

On Android – first Samsung Transform Ultra then Sidekick 4G I found the app store was like wild, wild west. Buggy apps sat along well tested ones, and only a very discerning eye and mobile guru might know the difference. Syncing was absolutely horrible. The whole platform assumes you want to sync up with Google accounts. I went with Missing Sync from Mark/Space. My addressbook started getting corrupted, duplicate records appeared, syncs would fail halfway through.

What’s more there were tons of started services I didn’t even use like Smart Navigation, Group Texting, and so on. These services seemed to run in the background, hog & bleed memory, and slow down my phone til it started crashing. I actually had to download an app called Easy Task Killer. Apparently a very popular app on the Android phones, I wonder why?

Later on I found out that T-mobile was no longer supporting the Sidekick. No wonder it was so buggy. I can’t believe they’d ship something like this.

My full list of beefs:

  • corrupted data
  • slow to non-functional syncing
  • dangerous apps
  • sharing of private data
  • ineffective calendaring
  • no support for addressbook groups
  • apps not remembering context & position
  • no good email app
  • weaker less feature rich apps
  • kludgy interface

I’ve long since quelled my desire for a physical keyboard. I was struggling with every other thing I would do with the device. Sometimes I’d just give up.

I really wanted to get along with my Android phone but my experience with it only gave me an insight into three crucial areas where it falls short.

  1. An Iron Fist
  2. People complain about Apple’s iron fist in app store approval. You can’t have it both ways. Android completely lacks discipline and users suffer hugely because of it. That weakens the platform.

  3. A manageable set of devices
  4. Developers building for Android must test on a huge spectrum of hardware. Smaller shops are likely to choose a few of the biggest ones only. Phones with a smaller user base likely have a lot of bugs just in the Android version they run. All this bodes badly as users just see buggy software, they don’t know why or how. This perception further weakens the platform.

  5. Affordable development

Building apps costs businesses money. Businesses must balance the costs of building features, test, debug, troubleshoot & release. That’s cheaper on the iphone because you have one device that is much more mature. This has a pile on affect as it strengthens the platform, users spend more on their devices, more users pile onto the iphone platform, so more money can be made building an iphone apps.  On Android higher costs, lower margins further weakens the platform.

A new love for Apple

This whole experience has brought me back to the iPhone 4S, and I have a whole new love and appreciation for the platform and the device.

  • calendar reminders make me more productive
  • data is always right, and where I need it
  • I don’t fumble with menus – I’m faster & less frustrated
  • cross-platform apps are more feature rich on the iphone
  • complex features are hidden in plain view – superb interface
  • pulse, hootsuite, yelp, notes & calendar are all integrated & productive

I also learned that sometimes less is more… much more!

Best of Guide – Highlights of Our Popular Content

We cherry pick the top 5 most popular posts of various topics we’ve covered in recent months.

Beware the client

Recently I was searching Google for “startup consultant” to see what I might find. I came across this article by John Greathouse titled Beware the Consultant. No self-respecting, hard-working consultant would sit back and take hits without defense, so in true poetry slam style I thought I’d write a quick retort.

Beware the Client…

  • The one that asks for the world.
  • The one that wants everything done yesterday.
  • The one that drags their feet paying bills or loses your invoices.
  • The one that moves the goal posts halfway through a project.
  • The one whose left hand doesn’t know what the right hand is doing.

 

 

Truth is often stranger than fiction

Of course in the real world things are much more nuanced than this black and white picture. Certainly there are probably some consultants that fit John’s description, but there are others that are loved by their clients.

  • There are tons of “little guys”.
  • There are freelancer’s that do a thankless job.
  • Consultants provide much needed plugin solutions for firms.
  • They work as long or as short as necessary.
  • They’re available, when you can’t find anyone!
  • They’re often self-taught – maintaining skills on their own dime.
  • BYO laptop, they’re plug-n-play.
  • They’re communicators – or they wouldn’t succeed in consulting.
  • They are small business owners – so they relate to your real business pain.

 

As far as performance based compensation goes, I would say two things:

  1. It’s always performance based because in my experience the startups I work with are usually bigger than I am. So if they decide not to pay being unsatisfied with the work, there’s little I can do about it.
  2. I do believe in being open to different types of arrangements, sometimes day or week based on time and materials, and other times project based. However the devil is very often in the details with regards to “quantified, clearly understood results”.

 

Although Mr Greathouse may describe *some* real-world or big-corp consultants, they are by no means representative of the lot of us.

“My startup is too cool for your business school”

An article I read on Tech Crunch recently got me thinking about startup culture. In Are You Building A Company, Or Just Your Credentials Geoff Lewis, expressed his distaste for a friend’s plan to get on Y Combinator’s ‘no idea’ startup incubation program. In this experimental approach, groups or individuals with a desire to be part of a startup but who have no product or business idea to begin with, can apply.

The thinking I believe, is that since brilliant ideas aren’t the only factor for startup success (many other factors like organisation skills, business savvy and tenacity matter too) YC will dig into their vault of ideas and match one that’s most suitable to these idea-starved groups.

Firstly, reading this I could sympathize with the discontent.  Venture capitalists exist for people who have an idea and want to realise it. There are already programs for people who don’t have an idea but want to achieve some success. They are called “careers”.

However, if you look at it from an incubator’s perspective, it’s a pretty clever and measured approach. YC knows what sort of group dynamic in startups have a higher chance of success and it is casting its net wide to find them.

But Lewis took issue with the fact that his MBA-qualified, credential-seeking buddy was signing up with YC presumably just to add to his blue ribbon collection.

With such articles what’s usually more interesting is the comments they elicit. Many who reacted responded with the same feelings of contempt; calling paper-chasers ‘hucksters’ and scoffing at their lack of passion and sincerity.

The tone among some carried this notion that startups were a special breed of entrepreneurs burdened with some sort of higher calling to liberate the world; money and honour being an afterthought.

Not being directly part of the startup circle, so-to-speak, I found the reaction amusing and frankly, rather foolish. Are people involved in startups turning into a sort of in-group? Do they really think of themselves as some kind of mutant-strain of businesses that are different from ‘regular’ enterprises?

If we look at the richest Internet companies today, once startups themselves, they are no less motivated by avarice and the bottomline, so why be so judgmental of Mr MBA treating the YC program as a way to gain credentials? I bet many talented individuals are going to have a go at it for the same reason if not a variety of reasons. As the program draws out, I suspect the number of participants will peter out by attrition anyway.

And who’s to say not coming up with a disruptive idea makes you less enterprising? Think of startups as team sports. There are some teams that innovate with the most creative gameplay that catch their opponents off-guard. There are teams that win by consistency and endurance, doing the same thing well over many years to pull ahead of the pack.

To cast aspersions on the motivations of others and to let silly prejudices limit participation seems discordant with the spirit of the Internet itself, where challenging convention is the order of the day and everyone is entitled to a shot at success.