Category Archives: iHeavy Newsletter

The Needle in Big Data Noise

Join 5500 others and follow Sean Hull on twitter @hullsean.

[mytweetlinks]

Also take a look at: I hacked Disqus Digests to discover new blogs

Who the heck is Bayes

Thomas Bayes was a scientist & thinker, Fellow of the Royal Society, and back in 1763 author of “An Essay toward Solving a Problem in the Doctrine of Chances”. His method advocated learning by approximation, to get closer and closer to the truth by gathering more information, and factoring that into probabilities & predictions.

[quote]
What isn’t acceptable under Bayes’s theorem is to pretend that you don’t have any prior beliefs. You should work to reduce your biases, but to say you have none is a sign that you have many.
[/quote]

Why should you care?

From hurricane & earthquake warnings, to financial storms or terrorism, prediction is more important than ever. Epidemiologists can make use of Bayesian techniques to protect populations, gamblers can use it in sport, and investors for markets.

See also Amp up your blog traffic by improving your pagerank

Why Nate Silver is different

Nate is famous for predicting the 2012 presidential election with uncanny accuracy. So the book is an in depth look at how he thinks, and how he works with data. He talks of Hedgehogs – those who believe in big ideas and work from large principals, versus Foxes who see the world as messy, often inconsistent and unpredictable, but who nevertheless tend to present better though less definitive predictions. The philosophy is less of modeling, and more of testing, and adjusting along the way to get closer to the truth.

See also Sales sucks, but a bear market offers hard lessons

For engineers & startups

Nate interviewed John Sanders of a scout for the LA Dodgers. He identified five abilities and characteristics that predict success in baseball. Looking at them together, I think they can well predict success in Startup land too.

1. Preparedness & work ethic
2. Concentration & focus
3. Competitiveness & self-confidence
4. Stress management & humility
5. Adaptiveness & learning ability

The book is a bit technical and sometimes long winded. But it is choc full of real insight, and wisdom that we can all put to use in our careers and businesses.

Also: AirBNB didn’t have to fail – AWS outages be damned!

Get a whole lot more! Scalable Startups. Exclusive. Monthly.. We share special content. Here’s a sample

Sales sucks, but then I learned

Are you a developer or startup entrepreneur? Have you ever been frustrated with some of the claims made by the sales team or lacked the patience or ability to communicate across departments?

Join 4000 others and follow Sean Hull on twitter @hullsean.

Just out of college

Just out of college I got a job as a Macintosh Software Developer for a small firm outside of University at Buffalo. It was a ten person company, and half of us were on the technology side of the house. I was doing C++ & Graphical Interface design & coding.

Why is it so hard to find operations & devops talent? Enter the Mythical MySQL DBA!.

Sales is “ahead” of engineering

Besides coding, I also fielded support calls from customers which brought me perspective on both what they wanted, and where they struggled with the software. Our app helped consumers and nutritionists track diet & exercise.

[quote]
The sales team made promises of technology the company wasn’t capable of delivering. Meanwhile the engineering team was sent scrambling to answer to those promises.
[/quote]

Soon I was fielding questions from customers asking when the new heart rate monitoring would be available. I followed up by talking with the team lead & chief architect. He had no plans of building such a feature, nor did we even know how it would be possible!

[mytweetlinks]

Searching for a database expert? MySQL DBA Interview Guide.

We checked in at our weekly meetings, and the CEO explained that the sales team was simply “ahead” of engineering. Years ahead apparently even of the technology that was possible at the time!

Fast forward 5 years to professional services

A half decade later I’m doing independent consultanting for dot-coms. Much of my business came from word of mouth. Helping a firm out of a pinch, speeding their site so they can handle 10x customers on the same servers and suddenly everyone is your friend!

Too many customers is a good problem to have right? For hyper growth companies there are 5 Things Toxic To Scalability .

But all is not smooth sailing in the freelance consulting world. The dot-com crash comes along and budgets are squeezed tighter. Business spend is reduced and every dollar is scrutinized. I learned to speak to prospects about savings and personalized service, advantages of lower overhead, and real return I could provide. At the end of the day if they’re not buying, your services aren’t worth their cost!

[quote]
The sales process should inform the business about what customers really want. In a successful startup there is communication back and forth with engineering and business units so all are working in harmony.
[/quote]

Full Circle

Now coming full circle I have a wide perspective on business. I understand the engineering fundamentals, and the limitations of technology. I also have a grasp of product, and how business units must manage the bottom line, and deliver to customers or else perish in the marketplace.

Looking for a top flight cloud engineer? Grab our Amazon EC2 Interview Guide.

For the two to achieve a happy marriage, you must bring a balance of execution & technical debt, with satisfying a real customer need in the marketplace. And therein lies the innovation & startup sweet spot!

You might also like our piece Why generalists are better at scaling the web

Like our content? We publish an exclusive monthly Scalable Startups where we share more tips and special content. Here’s a sample

Dinner, dollars & devlishly creative thinking

Efficiency at Dinner?

I just finished reading Tyler Cowen’s opus, An Economist Gets Lunch. I have to admit I’m already a fan of his writing, getting a daily dose on from his blog Marginal Revolution.

Join 4000 others and follow Sean Hull on twitter @hullsean.

What I like about this book is that it is unconventional by definition. Further economists like scalability engineers like to think about efficiency. How can I squeeze out more from less? Like the business question how do I get better ROI or more bang for my buck? Questions spring to mind like – What does an economist know about food? Or – What does food eating have to do with economics? Well on both points you’ll get some surprising answers.

Hiring or job seeking? Check out our MySQL DBA Interview questions which is useful to managers, candidates and human resources alike.

To the former question, Cowen has some really good insight because he brings the fresh perspective of an economist. His sort of mantra throughout the book is:

[quote]
Food is a product of economic supply and demand, so try to figure out where the supplies are fresh, the suppliers are creative and the demanders are informed.
[/quote]

Economists & engineers talk shop

What about the second question, how is the food we eat related to economics? Further does it have an impact on environmental and energy consumption questions? As it turns out in a rather big way yes it does. Let Tyler say it in his own words…

[quote]
When it comes to relieving climate change problems, there are two approaches. The first to put it squarely is to have everyone memorize facts about boats & bananas, and update that analysis as often as is necessary. The second approach is to rely on the price system, specifically to modify prices so that they reflect more information about the value of the environment. That’s the economically smart way to address climate change. The first method is wielding a pea shooter and the second is more like a bazooka.
[/quote]

Interested in web speed? Why generalists are better at scaling the web.

What he advocates more specifically is taxing the things we want to reduce. Biggest on the list are fossil fuels he says and next up meat production which through methane emissions contribute to climate change problems. These taxes will naturally curb our use, cause us to take fewer trips, be more efficient with our use, and tighten the wallet naturally.

Applying an economists eye to food & environment yields some excellent insights. For those of us in the startups & internet these fresh takes may well give us some insight in business too. If nothing else it’ll help us find the best meal for dinner!

Want more? Grab our Scalable Startups monthly for more tips and special content. Here’s a sample

When You Have to Take the Fall

Also find Sean Hull’s ramblings on twitter @hullsean.

One of the biggest jobs in operations is monitoring. There are so many servers, databases, webservers, search servers, backup servers. Each has lots of moving parts, lots that can go wrong. Typically if you have monitoring, and react to that monitoring, you’ll head off bigger problems later.

A problem is brewing

We, myself & the operations team started receiving alerts for one server. Space was filling up. Anyone can relate to this problem. You fill up your dropbox, or the drive on your laptop and all sorts of problems will quickly bubble to the surface.

Also check out – Why generalists are better at scaling the web.

As we investigated over the coming days, a complicated chain of processes and backups were using space on this server. Space that didn’t belong to them.

Dinner boils over

What happened next was inevitable. The weekly batch jobs kicked off and failed for lack of space. Those processes were not being monitored. Business units then discovered missing data in their reports and a firestorm of emails ensued.

Hiring? Get our MySQL DBA Interview Guide for managers, recruiters and candidates alike.

Why weren’t these services being monitored, they wanted to know.

Time to shoot the messenger

Having recently seen a changing of the guard, and a couple of key positions left vacant, it was clear that the root problem was communication.

Looking for talent? Why is it so hard to find a mythical MySQL DBA or devops expert these days?

I followed up the group emails, explaining in polite tone that we do in fact have monitoring in place, but that it seemed a clear chain of command was missing, and this process fell through the cracks.

I quickly received a response from the CTO requesting that I not send “these types of emails” to the team and to direct issues directly to him.

You might also like: A CTO Must Never Do This

A consultants job

As the sands continued to shift, a lead architect did emerge, one who took ownership of the products overall. Acting as a sort of life guard with a higher perch from which to watch, we were able to escalate important issues & he would then prioritize the team accordingly.

Are you a startup grappling with scalability? Keep in mind these 5 things toxic to scalability

Sometimes things have to break a little first.

What’s more a consultants job isn’t necessarily to lead the pack, nor to force management to act. A consultant’s job is to provide the best advice possible & to raise issues to the decision makers. And yes sometimes it means being a bit of a fall guy.

Those are the breaks of the game.

Want more? Grab our Scalable Startups monthly for more tips and special content. Here’s a sample

Ken Auletta Gets Us Googled

Also find Sean Hull’s ramblings on twitter @hullsean.

The title sounds vaguely fatalistic, the end of the world is nigh, that kind of thing. It turns out though that Auletta is a journalist having reported over the years a lot on old media. So when he says “as we know it” he’s speaking as much to old media as he is to the tech vanguard.

You’ll also want to check out The Big Switch – Rewiring the World From Edison to Google by Nicholas Carr.

But what makes his book superb isn’t just his phenomenal journalistic skills, in digging up all the facts and serving up a fair and accurate presentation of things. I think it’s important that he’s not a cheerleader at all, and approaches the topic with a critical eye as much to old media who ignored many of the warning signs in early 2000′s as to google who he emphasizes has been hubristic, at times arrogant, and has struggled with issues of privacy and copyright as they’ve built their technology.

Also check Jeff Jarvis’ What Would Google Do.

What makes this book even more important though is to step back and think of it as a case study in how the internet has become such a disruptive force. And in that light, google is a business which has rode that wave as much as it has defined it. Interestingly Google was not afraid to bring him to Mountain View to speak in their AtGoogleTalks series, and that video is now up on YouTube.

Ken Auletta on At Google Talks

Want more? Grab our Scalable Startups monthly for more tips and special content. Here’s a sample

When You're Hired to Solve a People Problem

Also find Sean Hull’s ramblings on twitter @hullsean.

A good five years ago I worked for a firm in online education. Among various products they provided through their website, they were struggling with a process to get content churned out more quickly. The bottleneck was slowing down their business, and limiting the new products they could offer.

Help Us Publish, Please…

Among a number of things I was asked to look at, one particularly vexing problem surrounded publishing. Adding new products had become a cumbersome & difficult process. It took days sometimes weeks. For obvious reasons the stake holders wanted to wrestle this process out of the hands of engineering, and place it were it arguably belonged in the hands of the business units.

[quote]
When you’re hired to solve specific technical problems it only figures that you go looking for software solutions. But sometimes the problems turn out to involve the people and processes of an organization. Getting them unstuck is one of the biggest challenges an professional services consultant can face. But it is also one of the most rewarding to solve.
[/quote]

Bumping into Fear, Uncertainty & Doubt

As I dug into the meat of the problem I began to work closely with the database administrator. He was a very smart gentleman & friendly in his own way. But he also spoke with a very thick accent and brusqueness about his manner that proved difficult at times. After working together for some awhile, however I began to win him over, and he started to trust me.

Looking for a top-flight database administrator? Here’s our interview guide for recruiters, managers and candidates alike

It became apparent that he was rather resistant to handing over the keys to the publish process to non-technical folks in other departments. Having handled his share of outages, and bungling screw ups, which sometimes fall on operations during some of the least hospitable hours on the dial, I could understand his concern. What’s more he knew the code which had grown unwieldy.

If I were to use a polite euphemism I would call it spaghetti code.

Management, Managers & Trouble Brewing

Around then the CTO decided to send a manager to sniff around. Unfortunately the manager in question was a very hands off type. His edict was simply to get this done in two weeks, and proceeded to go on vacation. Upon his return when things were still hitting snags, things started to go south.

Despite AWS failures firms like AirBNB and Reddit didn’t have to go down.

Though some of the process had been automated, I refused to move the changes into full push-button automation without first testing on dev environments. Of course those requests had fallen on deaf ears.

Problem comes to a head

Next the hands off manager escalated things upstream, of course adding his own spin on the situation. Shortly thereafter I’m called into the CTOs office only to get royally chewed out. A serious smack down which I’ll admit came almost out of nowhere.

A related article which readers also found quite popular: A CTO Must Never Do This

Oh, honestly I’m not complaining. On some level this is the job of the consultant. To act as the third party, wise or unbiased second opinion, and even punching bag at times.

Once things calmed down, I explained the situation from top to bottom. Yes there was messy code, and yes the process was complex, but it could of course be automated. What really stood in the way was a very resistant engineer who currently owned the process.

As much of the Sandy recovery continues, Devops can learn real lessons from the hurricane.

The CTO for his part concurred, having had trouble communicating with the engineer himself, and really not liking him much. He then appointed a proper project manager to oversee redoing the publish process from scratch.

A Plea for Cooperation

If I were to do it all again, for my part I’d sniff out the people dynamics more carefully. It’s often the case the companies have the engineering talent in house to solve a particular problem, but not the will or knowledge to put it into play.

Is your business growing? Having trouble scaling? Here’s how we do a performance review. It’s the first step on your way to hyper growth.

To managers & CTOs I’d encourage where possible to look for people, process and communication issues. Try to ferret out when something is an engineering problem, or whether it is one of people, silos and territory.

Want more? Grab our Scalable Startups monthly for more tips and special content. Here’s a sample

Switch to the cloud – shift of a century

The switch to cloud is way bigger than you think

A Review of Nicholas Carr’s book “The Big Switch”, available on Amazon here.

Also find Sean Hull’s ramblings on twitter @hullsean.

Do you work in devops or as a performance consultant? Do you manage web applications backed by databases? If so you probably love high performance beefy iron, big servers with equally fast RAID arrays that deliver lightening fast response to an entire application and ultimately your customers.

A related article, Devops can learn from Sandy, serious and very real disaster recovery lessons.

So if you’re like me you may feel a little leery about the cloud. On AWS for example, server commoditization has taken infrastructure for a sharp turn south. We struggle with unreliable disk performance & shared network bandwidth, while our applications compete with other customers in the so-called multi-tenant environment. Even the servers themselves drop like flies. Something’s got to give!

[quote]
All of them would be wise to study the examples of General Electric and Westinghouse. A hundred years ago, both these companies were making a lot of money selling electricity-production components and systems to individual companies. That business disappeared as big utilities took over electricity supply. But GE and Westinghouse were able to reinvent themselves. They became leading suppliers of generators and other equipment to the new utilities, and they also operated or invested in utilities themselves. Most important of all, they built vast new businesses supplying electric appliances to consumers — businesses that only became possible after the arrival of large-scale electric utilities. Sometimes a company can discover an even better business if it’s willing to abandon an old one.
[/quote]

Shopping for a smartphone? Find out why the Android platform is broken.

That’s why Carr’s book offers an eerie and uncanny read. What we’re seeing today in infrastructure very closely parallels what happened to electricity before it. Turns out at the turn of the century electricity production was not centralized and no electric grid was yet criss-crossing the country. Big companies actually built and managed their own power plants.

What happened?

Through the efforts of great entrepreneurs like Thomas Edison & Samuel Insull, the electricity production machinery were centralized and eventually incorporated and run by government managed utilities. All of this drove costs of electricity precipitously downward.

Looking for a database expert? Take a look at our MySQL Interview guide for candidates, hiring managers and recruiters alike.

How did we benefit? Oh can you count the ways?

Now households could afford electricity too. Next we saw consumer appliances and automation begin. Vacuum cleaners to washing machines flourished, bringing social change with it.

[quote]
What the fiber-optic internet does for computing is exactly what the alternating-current network did for electricity: it makes the location of the equipment unimportant to the user. But it does more than that. Because the internet has been designed to accomodate any type of computer and any form of digital information, it also plays the role of Insull’s rotary converter: it allows disparate and formerly incompatible machines to operate together as a single system. It creates harmony out of cacophony.
[/quote]

Cloud providers fail, components fail, datacenters fail. Find out why AirBNB & Reddit didn’t have to fail even while it’s AWS cloud went down.

The takeaway

The shift to cloud computing is way bigger than one application or one business. And the gains and momentum are way larger than we in devops may realize. With that it’s an inexorable shift, and one we would do well to embrace. Like all shifts we need to learn to adopt our technologies, as the benefits to business are incalculable.

Want more? Grab our Scalable Startups monthly for more tips and special content. Here’s a sample

When Clients Don't Pay – Consulting War Stories

Also find Sean Hull’s ramblings on twitter @hullsean.

It’s a cold cold winter out there, so as they say you need money to keep the lights on? Yep, that’s true whether you’re a small business or a consultant. Everyone has to pay their bills. Or do they?

It’s an unfortunate fact of life in business, but sometimes there are differences. Disagreements about deliverables, timelines, milestones, and deadlines. But when all the work is done, there are still sometimes differences over dollars.

Over the years most of those I’ve managed to work out with clients, but there were a few that went sour. One case was with a large entertainment firm. The music business is one that I hadn’t interacted with much before. For these guys I’d done a few days work in the past, and was paid promptly. Now they were in a bit of a jam. They called me up and asked if I could help.

Related article: A CTO Must Never Do This.

These types of emergencies often come at the worst times, and I explained that I was already juggling a few other things. They pleaded for help, and I relented. I carved out a full day of time for them, explaining the day rate and so forth. While on the phone though, I expressed caution.

[quote]I understand that your issue is urgent now, what happens if your needs change in the next 48 hours, I asked? Not a problem they replied, we can use your help anyway, so we’ll book you either way. Fast forward 48 hours, work canceled & client won’t take my call.[/quote]

Great I thought, verbal assurances. That works for me, I thought.

Fast forward 32 hours, and I receive an email saying the problem is resolved, and offering a “kill fee” of which I knew nothing, and which was never negotiated or discussed.

Along comes the day of reconning, and I call the client. They don’t take my call. Shortly there after I receive additional emails. I reply and explain we should talk on the phone. Still the client can’t find the time to pickup the call.

So I judiciously put together an invoice for the work. I email it directly to finance, and CC all parties. From there I get responses ranging from disinterest to denial. Over the coming months I periodically resend the invoice, but to no avail.

Or was it? I actually feel that this experience is to great avail.

1. Small disagreements foreshadow larger ones down the road
2. A relationship between client and vendor is a mutual one. If parties can only pickup the phone when they need something, then things are out of balance already.

Sure I lost a day & the fee associated. But I gained a lesson.

Patience and polite persistence

I firmly believe that being patient and persistent wins in the end. Sometimes clients have hiccups in payroll or budgets. Keep communication lines open.

Freelancers & consultants: Grab my Consulting 101 Guide.

Appealing to fairness

If you maintain a healthy relationship with your client, then appeals to fairness are normally heeded.

Appealing to promises

Emails are important to keep a paper trail of agreements. Communicate clearly and often so you know when you get derailed, and can refer back to what was agreed previously.

Popular: AirBNB didn’t have to failed – AWS hosting outages.

Setting precedents & expectations up front

This is an important one, that freelancers and consultants alike sometimes forget. Setting and agreeing on expectations is key. Often details are in the fine print or left out completely. So ongoing communication can iron out those differences or bring them to light.

Want to hire the best? Read our DBA Interview Guide.

Getting a protective deposit

If you haven’t worked together before, a deposit makes a lot of sense. Executing on this is more than a show of faith. It underlines that accounts payable is on board with your hiring, and you are now in the payments system.

Related: Hiring is a numbers game

Sizing each other up

Websites provide the first representation of you and your client to each other. How you carry yourself and how they feel meeting them face to face is important as well. Ideally you’ll meet each other at the client’s offices, where you may meet others on the team, and put names to faces. At the very least a skype call will go a long way as well.

Managing spend – communication along the way

Keep a close eye on invoices. If a client is getting behind, resolve it before continuing to work. Deadlines are mutual in a business relationship. Yours to complete work by a certain date, and theirs to pay by an equally agreed upon time.

Read also: Real Disaster Recovery Lessons from hurricane Sandy.

Never go to court – defer to the handshake & gentleman’s agreement

I know the lawyers out there may think I’m naive. But I’ve been in business a long time, and I believe a handshake means and says a lot. Also common sense language & contracts, in the form of emails and so forth are better than heavily legalese ones that no one but a lawyer can understand.

What’s more going to court has a huge cost in time & stress. Don’t go that route. I also think it appeals to clients knowing that you’re not the litigious sort.

Want more? Grab our Scalable Startups monthly for more tips and special content. Here’s a sample

Fear a zombie invasion? Try algorithms & bots

Automate This - How Algorithms Came to Rule Our World - Christopher Steiner

Automate This - How Algorithms Came to Rule Our World - Christopher Steiner

Also find Sean Hull’s ramblings on twitter @hullsean.

Have you read any of those tongue in cheek books on how to survive the coming zombie apocalypse? They’re funny in their own way, and reading Chris Steiner’s book it was the first image that popped into my head.

Countless industries falling to the algorithm invasion

I thought i followed automation & computer algorithms closely. Turns out this stuff is touching a *LOT* more industries than I even realized. Not only touching, but altering and changing the landscape of jobs. In many cases supporting & helping folks do their jobs better. In others they’re actually supplanting jobs.

Wall street – very high speed trading

The biggest industry that algorithms first touched was wall street. Rewind the clock back to 1980 and enter Thomas Peterffy. He went about applying mathematics to commodities options trading which to that point had been traded on gut instinct. The math won out, and he went on to become a so-called market maker on wall street.

The automation & computerization of trading continued at breakneck speed into the 90′s sucking up all the best math & science minds that the Ivy schools could churn out.

[quote]In a world where an increasing amount of power is being accumulated by those who can fashion and leverage algorithms with skill & finesse, our future looks to be one filled with bots judging us, routing us & measuring us. – Chris Steiner[/quote]

Enter high frequency trading and you know where this is going. We’ve weathered the near collapse of the financial system in 2008, and endless market shocks at least some of which can be traced back to algorithms. Today a full 60% of the world stock markets are run by high frequency computerized trading.

On a related note, check out Eli Pariser’s book The Filter Bubble which talks about search, and how algorithms are crafting and steering what we see and read.

Software & the creation of music

Our next stop is a strange one. We meet David Cope who, with a fascinating mix of music & love of writing software, built code to compose music. He then even played it at concerts. He astounded then infuriated critics who couldn’t tell what was composed by a human and what by computer. Some even loved the Cope code generated music more!

And after that it just gets stranger. He introduces us to the music industry software that analyzes music for it’s potential to be a hit. Can such software find unknown geniuses? Does it do away with the job of the A&R man?

It sure poses some tough questions. If such software can make record companies more profitable, no one will scoff.

Health care takes a lashing

Next up software touches the art of medicine, and the sacred science of diagnosis. First we’re seeing software that assists doctors, helping find cancers and analyzing symptoms & databases of diagnosis. But what of the day when such software does the job better. It’s already happening.

With rising health care costs, improvements in health care diagnostics at lower cost will be a huge boon for the industry. Spotting cancer better & faster?

Physicians beware.

Interested in management & business? Checkout A CTO Must Never Do This a war story about life in the devops trenches.

NASA & sanity in space

Before astronauts are sent off into space, they are given endless psychological evaluations. They’re asked questions, and probed on and on. Turns out there are a few different psychic groups people fit into, and if you mix the wrong ones, it can be volatile combination.

Turns out algorithms and software help NASA evaluate people, and ultimately makes those missions safer by pairing compatible and non-volatile personalities. When in the cramped confines of a spaceship it can be life or death.

Invasion of the Math refugees

As criticism, towards the end of the book I felt Steiner lost his way a bit. He spent lots of pages going on about the exodus of engineering, math & science talent to Wall Street, for what he saw as little gain. Whatever your opinions on that may be it didn’t seem to relate to the overall theme and served as a bit of a distraction.

As the financial industry has since contracted, he says that problem has subsided and talent has moved back to startups where innovation happens. As he says it’s certainly where the jobs are.

Ultimately I would have liked to hear some of his critical thoughts. He brings much to our attention, but doesn’t offer much in the way of caution or alternative ways forward.

Want more? Grab our Scalable Startups monthly for more tips and special content. Here’s a sample

Why your cloud is speeding for a scalability cliff

Join 14,000 and follow Sean Hull on twitter.

Don’t believe me that you’re headed for the cliff?

A startup scales up to no avail

Towards the end of 2012 I worked with an internet startup in the online education space. Their web application was not unusual, built in PHP and using Linux, Apache & Mysql all running on Amazon web services. They had three webservers in the mix and were seeing 1000 simultaneous users during peak traffic.

All this sounds normal except they were hitting major stalls, and app slowdowns. Before I was brought in they had scaled their MySQL server from a large to extra large instance, but were still seeing slow downs. What can we do, they asked?

I dug in and took at look at the server variables. They seemed to have substantial memory allocated to the server and Innodb. I then dug into the slow query log. This is a great facility in MySQL which sifts through activity happening against your database, and logs those which take a long time. In this case we had it set to ½ second and found tons of activity.

What was happening? Turns out there were lots of missing indexes, and badly written SQL queries.

A related popular piece AirBNB didn’t have to fail despite Amazon’s outage.

Also: Why generalists are better at scaling the web

How can we resolve these problems?

The customer asked me to explain the situation. I asked them to imagine finding a friend’s apartment in NYC without an address. Not easy right? You have to visit all of it’s 8 million residents until you locate your friend’s home.

Also check out: Real Disaster Recovery Lessons from Sandy.

This is what you’re asking the database to do without indexes. It’s very serious. It’s even compounded when you have hundreds or thousands of other users hitting different pages all with the same problems. Your whole dataset can fit in memory you tell me? So-called logical I/Os still cost, and can indeed cost dearly. What’s more sorting, joining, and grouping all compound the amount of memory your dataset can require.

Related: Why you can’t find a MySQL DBA

Why didn’t a bigger server help?

Modern computers are fast and EC2 extra large instances have a lot of memory. But with thousands or tens of thousands of users hitting pages simultaneously, you can take down even the largest servers.

[quote]Throwing hardware at the problem is like kicking the can down the road. Ultimately you have to pay your debt and optimize your code.[/quote]

Read: Why Twitter made a shocking admission about their data centers in the IPO

High performance code isn’t automatic

We have automation, we have agile processes, we can scale web, cache and search servers with ease. The danger is in thinking that deploying in the cloud will magically deliver scalability. Another danger is thinking that ORMs like ActiveRecord in Ruby or Hibernate in Java will solve these problems. Yes they are great tools to speed up prototyping, but we become dependent on them, and they are difficult to rip out later.

Want more, check out our 5 Things Toxic To Scalability.

Also: Why startups are trying to do without techops and failing

Fred Wilson says Speed is an essential Feature

Fred Wilson recently gave a talk on his top 10 golden principals to successful web applications. He says speed is the most important feature. Enough said!

The 10 Golden Principles of Successful Web Apps from Carsonified on Vimeo.

Hiring a MySQL DBA? Check out our DBA Hiring Guide with advice and hints for candidates and CTOs as well!

Read this: Why a four letter word still divides dev and ops

Want more? Grab our Scalable Startups monthly for more tips and special content. Here’s a sample