January 2013 – Scalable Startups

Why your cloud is speeding for a scalability cliff

Join 14,000 and follow Sean Hull on twitter.

Don’t believe me that you’re headed for the cliff?

A startup scales up to no avail

Towards the end of 2012 I worked with an internet startup in the online education space. Their web application was not unusual, built in PHP and using Linux, Apache & Mysql all running on Amazon web services. They had three webservers in the mix and were seeing 1000 simultaneous users during peak traffic.

All this sounds normal except they were hitting major stalls, and app slowdowns. Before I was brought in they had scaled their MySQL server from a large to extra large instance, but were still seeing slow downs. What can we do, they asked?

I dug in and took at look at the server variables. They seemed to have substantial memory allocated to the server and Innodb. I then dug into the slow query log. This is a great facility in MySQL which sifts through activity happening against your database, and logs those which take a long time. In this case we had it set to ½ second and found tons of activity.

What was happening? Turns out there were lots of missing indexes, and badly written SQL queries.

A related popular piece AirBNB didn’t have to fail despite Amazon’s outage.

Also: Why generalists are better at scaling the web

How can we resolve these problems?

The customer asked me to explain the situation. I asked them to imagine finding a friend’s apartment in NYC without an address. Not easy right? You have to visit all of it’s 8 million residents until you locate your friend’s home.

Also check out: Real Disaster Recovery Lessons from Sandy.

This is what you’re asking the database to do without indexes. It’s very serious. It’s even compounded when you have hundreds or thousands of other users hitting different pages all with the same problems. Your whole dataset can fit in memory you tell me? So-called logical I/Os still cost, and can indeed cost dearly. What’s more sorting, joining, and grouping all compound the amount of memory your dataset can require.

Why didn’t a bigger server help?

Modern computers are fast and EC2 extra large instances have a lot of memory. But with thousands or tens of thousands of users hitting pages simultaneously, you can take down even the largest servers.

[quote]Throwing hardware at the problem is like kicking the can down the road. Ultimately you have to pay your debt and optimize your code.[/quote]

Read: Why Twitter made a shocking admission about their data centers in the IPO

High performance code isn’t automatic

We have automation, we have agile processes, we can scale web, cache and search servers with ease. The danger is in thinking that deploying in the cloud will magically deliver scalability. Another danger is thinking that ORMs like ActiveRecord in Ruby or Hibernate in Java will solve these problems. Yes they are great tools to speed up prototyping, but we become dependent on them, and they are difficult to rip out later.

Want more, check out our 5 Things Toxic To Scalability.

Also: Why startups are trying to do without techops and failing

Fred Wilson says Speed is an essential Feature

Fred Wilson recently gave a talk on his top 10 golden principals to successful web applications. He says speed is the most important feature. Enough said!

The 10 Golden Principles of Successful Web Apps from Carsonified on Vimeo.

Hiring a MySQL DBA? Check out our DBA Hiring Guide with advice and hints for candidates and CTOs as well!

Read this: Why a four letter word still divides dev and ops

Want more? Grab our Scalable Startups monthly for more tips and special content. Here’s a sample

A master isn't born but made

A review of Mastery by Robert Greene.

Also find Sean Hull’s ramblings on twitter @hullsean.

Robert Greene’s 48 Laws of Power was a great read, offering endless lessons for business and personal dealings. When I saw he just published a new book, I was quick to grab a copy.

What I like about his writing is that he’s replete with counterintuitive bits of wisdom, that really offer new perspectives on old topics.

[quote]
Many people might find the notion of an apprenticeship and skill acquisition as quaint relics of bygone eras when work meant making things. After all, we have entered the information and computer age, in which technology makes it so we can di without the kinds of menial tasks that require practice and repetition; so many things have become virtual in our lives making the craftsman model obsolete. Or so the argument goes.
[/quote]

You might also enjoy our 3 part consulting 101 guide and our very popular DBA hiring guide.

He goes on to elaborate on this idea…

[quote]
In truth, however, this idea of the nature of the times we are living in is completely incorrect, even dangerous. The era we have entered is not one in which technology will make everything easier, but rather a time of increased complexity that affects every field. In business, competition has become globalized and more intense. A business person must have a command of a much larger picture than in the past, which means more knowledge and skills. The future in science does not lie in specialization but rather in combining and cross-fertilization of knowledge in various fields. In the arts, tastes and styles are changing at an accelerated rate. An artist must be on top of this and capable of creating new forms, always remaining ahead of the curve. This often requires having more than just a specialized knowledge of that particular art form — it requires knowing other arts, even the sciences, and what is happening in the world.
[/quote]

I couldn’t agree more. We wrote a piece a while back called Why generalists are better at scaling the web and that aligns nicely with what Greene is getting at here.

He begins with insight on finding ones life task, then apprenticeship & mentoring then working through the social challenges that are always present and finally ways to stimulate the creative-active impulses.

I really like that he emphasizes it as a process and one of life-long hard work. This resonates a lot for me, as that’s how I’ve found success doing independent consulting over the years. There have been a lot of ups and downs, wrong turns, and missteps, but tenacity wins out in the end. He even dispells the myth of the naturally gifted, such as Mozart or Einstein, arguing that in fact they did put in the requisite 10,000 hours of study and were not born with mastery as such.

Greene’s lastest book is a pleasure to read, and full of insight for startups, programmers, designers and business people alike. I highly recommend it.