We've all seen cloud computing discussed ad nauseam on blogs, on Twitter, Quora, Stack Exchange, your mom’s Facebook page... you get the idea. The tech bloggers and performance experts often pipe in with their graphs and statistics showing clearly that dollar-for-dollar, cloud hosted virtual servers can’t compete with physical servers in performance, so why is everyone pushing them? It's just foolhardy, they say.

On the other end, management and their bean counters would simply roll their eyes saying this is why the tech guys aren't running the business.

Seriously, why the disconnect?

Quite simply, they have very different agendas. The operations team is trying to deliver application speed, reliability, and scalability. Meanwhile the management team is trying to balance cash flow, capital investments and operating expenses, all this while retaining as much flexibility in decision making as possible.

Sound familiar? Yes, the bottom line for both are the same, but they approach the problem from two different sides.

After hearing this refrain from enough customers, I realized the question we were asking in operations was the wrong question. It is true that cloud servers do not perform as well as traditional servers -- at least not yet. The question that one should ask is, given this performance constraint with virtual servers, how can I alleviate the performance limitations of virtual servers and storage but still deploy into the cloud?

A little bit of history

Anyone who's been around computing since the mid 1990’s will see where I'm heading with this. Commoditization of computing, that's where. In the late nineties there was a big shift happening. In the 1990’s as Linux was maturing, commodity servers became all the rage. A big shift began to happen which drove cheaper PC hardware into the datacenter. Where once Sun stood supreme, suddenly all these upstarts were pushing crappy commodity hardware at 1/10th of the cost.

Old-guard systems administrators  at that time would balk, telling you how the stuff wasn't reliable, performance was worse, and they just failed too often. And you know what? They were right! Nevertheless look where we are today. Linux barreled through the datacenter because it lowered costs and introduced flexibility by giving you more choices and more redundancy.

All this rings very true today with the push to the cloud. Startups are the biggest adopters of cloud computing because the future is mostly unknown to them. Investing a quarter million dollars in hardware today when you don't know where you'll be in six months - that's a very hard prospect to entertain.

But cloud computing more than just lowers cost. Don't have the money for redundant servers to support Disaster Recovery? With Amazon EC2 you simply write scripts to rebuild your infrastructure and keep them handy when such an ill-fated day arrives. Flexibility, scalability and easier cost management. That's what the cloud delivers.

In his latest book, What Would Google Do? Jeff Jarvis seems to have authored a gushing tribute to the search giant that has pledged to do no evil. He paints a very optimistic picture, and shows us over and over how Google has opened up industries, and how that same openness helps consumers like you and I.

Jarvis, if you don't know him by name, has been a journalist for some time, but gained particular cred and notoriety when he blogged with the headline "Dell lies. Dell Sucks" after his horrible experiences with Dell computers and customer service.

While digging through Googly chapters, on Real Estate, Publishing, Entertainment, Shopping, Education and even Airlines, Jarvis serves up anecdotes on how a more open approach can help these industries adapt to a new business environment brought about by the Internet. He cites interesting examples like Gary Vaynerchuk, the creator of the hilarious and insanely popular winelibrary.tv show about wines, and now a public speaker on social media and brand building; and Brazilian author Paulo Coelho pirating his own works.

Taking the cue from some of these successes Jarvis goes on to propagate the idea that sharing and dishing out services for free is the way to make money. The irony that you have to buy his book for him to tell you that deserves a chuckle, and also raises the question of whether he himself buys all of that (pun inevitable). Indeed openness is great for consumers as most of us would agree. A level playing field increases competition, drives down prices for consumers. But it also drives down profits and margins.

With Google, Jarvis loses teeth

What this book lacked though was an honest assessment of the tremendous potential for abuse that Google has acquired through the course of its growth. We now see that manifesting in the current controversy of Google favoring Google+ posts in search results.

He quotes Google's Marissa Mayer as saying "Data is apolitical".  Luckily google folks are good at coming up with these great slogans.  They can hide behind them all the while they're muscling search objectivism out of, and ‘social queues’ into search.  I really wonder whether this may backfire on them, and not just because they don't publish all those 200 variables that impact search, but because their strength has always been their algorithms, and how they aren't biased.  How I can get search based on what's out there, and let me sift it.  The more they try to *HELP* me sift, especially without my knowing how they're helping, the more I become confused at the results, or worse, suspicious of them.

I might argue Google hasn't necessarily won by openness as Jarvis posits. Rather they've won by being first to understand the Internet, and so have been first to market in so many areas that are being heavily disrupted by the new technology.  It uses openness as a strategy against incumbents, but uses muscle and monopoly as businesses always have, in areas where it leads.

Aaron Wall put it brilliantly in his SEOBook blog:  " Where Google is losing you can count on them pushing the open label in order to build momentum and destroy the asymmetrical information advantages of existing market leaders. But where Google leads non-transparency is the norm." Or to borrow a quote from a random comment: "Google is like a ‘friend’ who buys, lets you drink for free but then slips a 5 dollar bill out of your pocket when you aren’t looking."

I really felt like Jarvis was too much of a Google fanboy. His confidence in Google is pervasive throughout the book, something one would find uncharacteristic of a journalist. Why didn't he shine the Dell-Sucks laser beam light onto Google? I kept searching for that kind of incisive commentary but I couldn’t find it.

It is for this reason that I prefer Googled, The End of the World as We Know It by Ken Auletta. Auletta offers a more balanced and critical analysis and I don’t think that makes him look like a grandpa who’s afraid of new technology. Auletta’s work and prose just came across as more thoughtful and mature. and while both books have already suffered obsolesence from the day they were released, I know which one I can turn to for a better understanding of Google.

Read Jarvis like you would any article or book -- with a healthy dose of skepticism. And perhaps also keep this question in your mind: "What Wouldn't Google Do?"

Scanning Crains NY Business recently, I saw an article on 'starting up' in 54 hours.  It's the brainchild of Marc Nager, Clint Nelsen and Franck Nouyrigat called Startup Weekend. Startup bootcamps seem to be the current extra-curricular activity of choice these days. Wharton is also getting in on it with their Innovation Tournament. Then there is the 48 Hour Startup  and of course let's not forget the 3 Day Startup.

So what's my beef?  Truth be told I admire the ambition, the optimism, and the openness of these efforts.  And for sure these bootstrapping marathons do introduce entrepreneurs to future colleagues and partners, get them asking the right questions about financing, customers, revenue, competition and so forth.

My problem with these events is they frame startups as something you *can* do quickly. As if it were a Lego set or pop-up book that gives instant results and gratification. Sure startups are 21st century tech-driven business that provide innovative products in a very short development cycle but a lot of the day-to-day running of the business are still very mundane 20th century sensibilities; not unlike running a mom and pop store, a laundromat, deli or sandwich shop.

No shortcuts

You have to learn how to ask your customers what they want, learn how to sell them something they need, and be prepared to be surprised in the process. I've worked with many startups and have seen often how many have had to throw out an idea that didn't work and re-strategize. This takes courage an conviction.

One startup client was building iPhone apps for event management. After testing it on the market it turned out nobody really wanted an event management app.  But there was a market for professional and consulting services that can help build technology platforms, and architect the right solutions.  So they shifted gears and moved into that space.  They've grown tenfold since.

Another startup client's change in direction was so radical that it even warranted a name change. Buzzd, started out as a city guide for consumers based on aggregating Facebook, FourSquare and Gowalla check-ins. Last October Buzzd became LocalResponse and turned its focus onto advertisers instead.

This kind of honest self-analysis and business acumen so crucial for survival can only be acquired through experience. For example managing overheads or being aware that profitability does not always mean solvency. And then there are decision taking and setting priorities. Repair that wall or build a new display case? Use that money on product and inventory?  Attract customers to your store?  What's the right price point to settle on?  How do you avoid legal entanglements, and who are some of the predators to watch out for?

Startup crash courses are probably a great way for  aspiring entrepreneurs to network, and for VCs to spot the next killer investment. But at the same time, it feels suspiciously like a diet fad that promises to change 30+ years of behavior overnight, and get you looking like a bikini model in no time.  Fitness and health are lifestyle choices, as is building a successful business. As much as we like to think otherwise, there are simply no shortcuts.

Abbott and Fisher’s previous book, The Art of Scalability received good reviews for shifting the way we think about scalability from merely splitting databases and adding servers, to include the human factors that weigh heavily on its success. Together with the authors’ distinguished pedigree (PayPal, Amazon, and eBay between them), I picked up a copy of their second book, Scalability Rules - 50 Principles for Scaling Web Sites without a second thought.

If Art was about laying a strong foundation for a scalable organization then Rules is the reference point for when you actually tackle the growth challenges. It acts as a reminder when you come to a crossroad of decision-taking, to keep with the principles of scaling. Each guiding principle is clearly explained and illustrated with examples. It also prescribes how and when to apply the rules.

For example I found many rules focused around people and processes such as #27 "Learn Aggressively" and #30 "Discuss and Learn From Failures".  Then there were many rules centered on application design and developement such as #17 "Don't Check Your Work" and #35 "Don't Select Everything".  I found myself agreeing with rules for architectural decisions directed to the operations team such as #11 "Use Commodity Systems" and #12 "Scale Out Your Datacenters".

With my tendency to categorize things, (or in this case recategorize the chapters) I tried to fit the rules into their respective pigeon holes, although I did struggle with that. While they didn't all fit neatly, the groupings I could come up with were:

• Application Design Rules
• Architecture Design & Implementation Rules
• People and Process Rules
• Network Related Rules
• Caching Rules
• Availability Rules

Who needs Scalability Rules?

Looking at the fifty rules divided up this way, summarizing the book becomes easier  but still a bit muddled.  For example I might ask the question, Who exactly is the book’s audience?  Is it application developers?  Certainly, there is a large section of rules designed for them.  Or maybe it's for the operations team, offering advice on how to design systems, networks and data centers, and advice for optimizing and redirecting development efforts?  Or maybe it is a book for managers, as the people and process related rules certainly are targeted towards those folks.  But if that's the case then much of the book is probably too technical for such an audience.

Ok, so let's step back, when or why might you have all these crisscrossing themes and roles?  The answer is that like me, the authors are consultants and in the course of engagements a consultant is often asked to wear many different hats.  So sometimes we're looking at scalability in regards to operational decisions, while at other times we're looking at scalability through the lens of application design decisions. And still other times scalability can best be achieved by focusing on people and processes. I’d add that not only is this a handy reference for consultants, but for any bootstrapping startup as well. Coders these day are expected to be at least a little bit knowledgeable with web operations, and vice-versa.  Especially pushing towards the guiding principles of devops, this book can help engineers take a critical look at their roles within an organization.

Looked at in that light you can really see how this book is truly an opus of knowledge on scaling web sites.  Although the organization is sometimes confusing, it will surely bring insights and enlightenment to all of your teams, whether operations, devs, managers, or business units.

As is customary in New York, even for takeout there is usually a tip jar at the checkout. Many of them have a large bowl, or glass jar in which you can throw your change as tips, or if you really love the place and service, a couple of dollars.

Of late I've noticed a few have placed those small plastic boxes with a tiny slot on the top. You try to put some change in the slot, and half of the money falls on the floor. It's as frustrating as threading a needle while suffering from astigmatic vision. Now when I come to a place that has this plastic box, I don't even bother tipping. I get a headache thinking about my change falling all over the floor. All I keep thinking is, why make it so difficult to tip?

Lessons from tipping

There're business lessons here to be sure. The most obvious being, when working with clients, make it easy for them to pay you. Don't ask them to jump through hoops to settle a bill. That goes just as well for complicated costs and itemization. Keep your invoices as simple as possible, so it's clear later on what it covers; the amount and the period. Do they like to pay by ACH or Wire, make that easy as well. Would they rather pay by Paypal? Perhaps have a carrier pigeon with a check? No problem!

This lesson might also be applied to add-on features and services. In many cases a client may require additional time or service from you. Does this involve a lot of approvals or details to be hashed over? Or is it already covered in your base contract so you both know what qualifies and how it will be billed? If you are in the design business, what's the cost of an additional design version or proof? Web operations, what's the cost of adding in another on-site day or weekend block of time?

We may not realize it but sometimes we fumble in the process of acquiring new business too. Recently I was looking to hire a web developer for a website facelift project. I liked the work of one of the applicants and tried to arrange for a quick chat over Skype. To my surprise they said they couldn’t get on Skype because they didn’t have an account. Granted they did offer to call me on my mobile but I’d also wanted to set up a three-way conference call. As a prospective client, I was amused at having to make the suggestion that they register for Skype in order to arrange the call. Someone more on the ball would’ve just said OK, and sign up for Skype right away.

Ultimately the lesson is to think in your customers’ shoes make it easy for your client to reward you when they're happy with your service, and want to buy more of what you are selling.

Clay Shirky tells a great story. Here Comes Everybody begins with a case of a lost phone in a taxi cab, and the extraordinary turn of events that led to the owner retrieving it. From photos posted online, to NYPD who were uninterested in following up, to taking it all online. Through that online publicity, the story got picked up by the NY Times and CNN, which put pressure on the police to track down the taxi.  It's a great example that illustrates the nuances, both good and bad, powerful and persistent that the Internet can unleash.

Throughout the book he weaves stories about the network effect, friends and friends of friends, and how that impacts information, organization, and the spread of ideas. Citing examples such as the SCO vs Linux court case and Groklaw, flash mobs and political organization, Shirky notes how all these events were influenced and facilitated by the Internet.

Mobilising people for collective action is that much quicker, effective, and less costly, with the Internet. Shirky examines how this could have the potential to topple dictatorships, pit amateurs against experts, and even the odds for everyone. While this may all sound too familiar (wasn’t it the same with the telegraph, the printing press, etc?) Shirky makes some intelligent and astute observations about changes in the media with the case of Wikipedia. To him the online, editable encyclopedia is not just a modern substitute for Encyclopedia Britannica, it is a whole new process of developing an encyclopedia.

Shirky tries to make his analyses accessible with clear prose and strong examples. Readers who are more interested in the how-tos rather than the what-haves of social media and web 2.0, may find the analysis overly academic. That said, Shirky’s book should be required reading for any novice wanting to appreciate the astonishing impact the Internet has had, and will have on our understanding of the world.

My reckoning was confirmed by a Bloomberg news report about stalwart retailers suffering from a dearth of talented engineers. Bloomberg cited Target’s outage-prone e-commerce site as a symptom of, among other things the market’s shortage. One of the challenges old-timers like Target face is having to compete with Silicon Valley startups as a fulfilling and ultimately, financially rewarding place to work.

From the outside looking in, it's hard to say for sure why Target.com keeps crashing, but I can speculate on a few possible scenarios.

For one, the handoff from Amazon may have been less than smooth, lacking proper documentation and so forth. It could also be that the handoff went to less experienced DBAs or perhaps, those more versed in the legacy technologies of Oracle and much less in the free-wheeling open-source ones like MySQL. Other reasons could be failures in capacity planning, incomplete or incorrect systems integration, or simply misconfigurations in the load balancer, replication of database and memory settings.

If any of these scenarios had been true for Target, a sound experienced DBA and/or operations team attuned to scaling and disaster scenarios should have been able to anticipate these outages and mitigate their impact.  That is, if there were enough talented and experienced ones to go around.

From our vantage point, we think there’s room for more individuals to specialize in this area. What we do see are developers or Unix system administrators that include MySQL experience in their bag of skills but few who can actually manage a database eco-system.  Even in the Oracle space where there are a lot of career DBAs, many of them have moved over from the business side, so they lack certain computer science and engineering fundamentals and a pure science foundation.

Much of this boils down to universities not churning out enough engineers.  And the ones that do graduate are drawn to Startups; the coolest, smartest firms like Facebook and Google. If young college grads are gunning for the best job they can find, they're likely to shoot for the sexiest most cutting edge technologies.  In today's market that means programming jobs in Ruby on Rails or perhaps Node.js. Few would aspire to be in WebOps.

Not too sexy for Ops

If I were to really go out on a limb I might ask if you've ever heard of Dustin Moskovitz?  No?  Oh he's "the ops guy" from the original Facebook team and, with a net worth estimated at $3.5bln, the youngest billionaire in the world.  Did I imply that operations and database administration wasn't sexy?

Same-same but different?

The Scenario
Imagine the scene, you use replication to backup your data. Your secondary database is your peace of mind. It's the always-on clone of your crown jewels. You even perform backups off of it so you don't impact your live website. Your backups run without errors. Your slave database runs without errors.
Then the dreaded day comes when your primary database fails. You instruct your team to switchover your application to point to your live backup database. The site comes online again. But all is not right. You notice subtle differences and your team begins to question how deep the data divide could be.

The Problem
Although MySQL replication is fairly easy to setup, and even to keep running without error, you may have unseen problems. MySQL's core technology to replicate data between master and slave is primarily statement based. Various scenarios can cause what in other database platforms you might call database corruption, that is silent drifting of data from what tables and rows contain on the master. It is no fault of your own, or perhaps one might argue even of your operations team. It is a fundamental flaw in how MySQL performs replication.

The Solution
Fortunately there is a solution. Checksums, the wonderful computational tool for comparing things can be put to work nicely to compare database. The Percona Toolkit (formerly maatkit) includes just such a utility for use with MySQL. It can be used to check the integrity of your slave databases.
If you've never performed such a check, you should do so ASAP. If your database has been running for months at a stretch, chances are there could be differences lying undiscovered between the two systems.

Depending on the volume changing in your database, you can continue to use this tool periodically to confirm that all is consistent. If integrity checks fail, there is another tool in Maatkit to syncronize differences, and bring everything back to order.

Check out the full list of NYC Startups in the original article.

View NYC Startups in a larger map

Strip by Randall Munroe; xkcd.com

First things first. This is not meant to be a beef against developers. But let’s not ignore the elephant in the living room that is the divide between brilliant code writers and the risk averse operations team.

It is almost by default that developers are disruptive with their creative coding while the guys in operations, those who deploy the code, constantly cross their fingers in the hope that application changes won’t tilt the machine. And when you’re woken up at 4am to deal with an outage or your sluggish site is costing millions in losses, the blame game and finger-pointing starts.

If you manage a startup you may be faced with this problem all the time. You know your business, you know what you're trying to build but how do you find people who can help you build and execute your ideas with minimal risk?

Ideally, you want people who can bridge the mentality divide between the programmers eager to see feature changes, the business units pushing for them, and the operations team resistant to changes for the sake of stability.

DevOps - Why can’t we all live together?

The DevOps movement is an attempt to bring all these folks together. For instance by providing insight to developers about the implications of their work on performance and availability, they can better balance the onslaught of feature demands from users with the business’ need for up-time.

Operations teams can work to expose operational data to the development teams.  Metrics collection and analytics aren’t just for the business units anymore. Employing tools like Cacti, OpenNMS or Ganglia allow you to communicate with developers and other business units alike about up-time, and the impact of deployments on site availability, and ultimately the bottom-line.

Above all, business goals and customer needs should underscore everything the engineering team is doing. Bringing all three to the table makes for a more cohesive approach that will carry everyone forward.

How to spot a DevOps person - Finding the sweet spot

The DevOps person is someone with the right combination of skill, knowledge and experience that places him or her in the sweet spot where quality assurance, programming skills and operations overlap.
There are also a few distinguishing characteristics that will help identify such an ideal candidate.

Look for good writers and communicators

Imagine the beads of sweat forming when a developer tells you: “We’ve made the changes. Nothing is broken yet.”
This is like stepping on glass because it implies something will actually break. The point is savvy developers should be aware that the majority of people do not think along the same lines as they do.
Assuming your candidate has all the required technical skills, a programmer with writing skills tends to be better at articulating ideas and methods coherently. He or she would also be less resistant to documentation and be able to step back somewhat from the itty-bitty details. Communication, afterall is at the core of the DevOps culture where different sides attempt to understand each other.

Pick good listeners

Even rarer than good writers are good listeners. Being able to hear what someone else is saying, and reiterate it in their own terms is a key important quality. In our example, the good listener would probably have translated ‘nothing is broken yet’ into “the app is running smoothly. We didn’t encounter any interruptions but we’ll keep watch on things.”

Lean towards pragmatists and avoid the fanatics

We all want people who are passionate about something but when that passion morphs into fanaticism it can be unpleasant. Fanaticism suggests a lower propensity to compromise. Such characters are very difficult to negotiate with. Analogously in tech, we see people latch on to a certain standard with unquestioning loyalty that's bafflingly irrational. Someone who has had their hand in many different technologies is more likely to be technology agnostic, or rather, pragmatic. They'll also have a broader perspective, and are able to anticipate how those technologies will play together.  Furthermore a good sense of where things will run smoothly and where there will be friction is vital.

Pay attention to extra-curricular activities

Look at technology interests, areas of study, or even outside interests. Does the person have varying interests and can converse about different topics?  Do they tell stories, and make analogies from other disciplines to make a point?  Do they communicate in jargon-free language you can understand?

Sniff out those hungry for success

As with any role, finding someone who is passionate and driven is important.  Are they on-time for appointments? Did they email you the information you requested? Are they prepared and communicative?  Are they eager to get started?

Hiring usually focuses on skills and very well-crafted resumés but why do you still find some duds now and then? By emphasizing personality, work ethics, and the ability to work with others, you can sift through the deluge of candidates and separate the wheat from the chaff for qualities that will surely serve your business better in the long run.

What I found was an excellent introduction to the nebulous world of social media marketing, where you find all sorts of advice and suggestions on how to engage your target audience.  If you're feeling like an ignoramus on matters of social media, Trust Agents is a great place to start and will give you ideas of how to 'humanize' your digital connections.

The authors illustrate the Trust Agent idea with Comcast Cares for example and how they engaged customers, and what worked so well for them.  Or Gary Vaynerchuk and his game changing Wine Library TV about wine.  He also emphasizes that building relationships online is a lot like building relationships in the real world a la Keith Ferrazzi of Never Eat Alone fame.  Engage in meaningful ways with people, don't market to them. Share valuable tidbits, and the community will reward you tenfold.

A 'trust agent'  lives by six principles:

1. Make your own game - be willing to take risks and break from the crowd
2. Be 'One of Us' - be part of the community by doing your bit and contributing to it
3. The Archimedes Effect - leverage your own strengths wisely
4. Agent Zero - position yourself at the center by connecting people and groups
5. Human Artist - learn how to work with people; help others and be conscientious of etiquette
6. Build an Army - you need allies to help spread your ideas

The book is excellent.  Put it on your holiday list.

They've sexed it up a little by calling it a face-off between Windows Azure and Amazon EC2  (no prizes for guessing which side I represent) but really it's going to be a primer on the Platform-as-a service and Infrastructure-as-a-service models. I expect some lively discussions during Q&A.

I'll be covering questions such as what cloud computing is, what EC2 provides, what is datacenter automation and the differences between a standard datacenter liks Rackspace and Amazon EC2. Meanwhile you folks who've  large investments in say EXCHANGE servers will be able to pose questions to Marcus Wendt of Composite C1.

It's Amex sponsored and you'll get a ticket good for a beer or a Citrus signature drink with which you can get cozy and warm up by the fireplace while Marcus and I are beamed through a flat screen with our respective presentations. If you're in Seattle drop in. I hope to see you or at least, hear you there.

by Ronald Bradford

No Nonsense, Readable, Practical, and Compact

I like that this book is small; 150 pages means you can carry it easily.  It's also very no nonsense.  It does not dig too deeply into theory unless it directly relates to your day-to-day needs.  And those needs probably cluster heavily around optimizing SQL queries, as those pesky developers are always breaking things

Jokes aside, this new book out on Oracle Press is a very readable volume. Bradford has drawn directly from real-world experience to give you the right bite size morsels you need in your day-to-day MySQL activities.

Highlights

Chapter one, The Five Minute DBA gives you the basic methodology if you don't already know it.  Enable the slow query log, analyze it, and use the explain facility.  Then index as appropriate, or eliminate queries if you can.

Chapter two digs a little deeper into the basics, introducing explain extended, table statistics and storage engines.  You'll also learn how to use show session & global status, as well as session & global variables.  You'll also have your first look at MySQL's data dictionary - INFORMATION_SCHEMA.

Chapter three is where it starts to get meaty.  You probably know that MySQL has b-tree indexes, but did you know that it has b+tree indexes, or hash indexes?

Chapter four digs into indexes further with single & multi-column indexes using them for sorting and joining.  You'll also find out about covering indexes which are multi-column matching the where clause, but also including columns needed in the SELECT predicate.  Do you have duplicate or unused indexes?  You'll learn why they matter to performance and how to eliminate them with tools like mk-duplicate-key-checker.

Chapter five continues along the same lines, with more coverage of indexes.  Learn to identify when you are using a covering index, fulfilling the entire query by only accessing the index.  You'll also learn about partial indexes, how they can reduce the size of index storage and retrieval while still getting your data for you.

Chapter six covers configuring the server itself, hitting on the system variables such as the innodb buffer pool (innodb_buffer_pool_size) and key buffer (key_buffer_size) as well as the query cache.  You'll also learn how to set the four main session memory settings - sort buffer (sort_buffer_size) and join buffer ( join_buffer_size) as well as the lesser known read buffers (read_buffer_size and read_rnd_buffer_size).

Chapter seven is all about the process of tuning and optimizing MySQL.  Rolling all the previous sections into marching orders, and prescriptive advice, he takes you through step by step how to apply the principles.  You'll get an introduction to mk-query-digest (though strangely without attribution to Baron Schwartz), the great maatkit tool for query analysis and aggregation, as well as the microsecond precision patch, which allows your mysql shell client to display more exact timing data.  For the patch he links back to an article on his own site which seems to be not found.  The author of the high precision mysql timer patch is Stewart Smith.

I personally got the most out of Chapter eight, full of self-described hidden performance tips.  From identifying unused or duplicate indexes, to replacing inefficient data types with better ones, why it's important to use NOT NULL where possible or how to store IP addresses efficiently, this chapter has a lot of goodies.  For those still struggling with SQL statement tuning, there are a few patterns that are described, offering advice on how to rewrite a subquery as an inner join,

What you might not know

• MySQL includes Oracle's index organized tables by a different name
• Too many indexes can dramatically impact INSERT & UPDATE performance
• Many DDL operations can be done online - see oak-online-alter-table (Shlomi Noach)
• Datatypes matter - use enum, int unsigned, timestamp & not null where possible
• Covering indexes are your friend, duplicate & unused indexes are not!
• A replication slave can have different storage engines or indexes from the master. These can support different uses - such as data warehousing or non-transactional requirements.
• While a_string LIKE '%end of my sentence.' won't use an index, you can index reverse_string, then use reverse_string LIKE REVERSE '%end of my sentence.' and MySQL will use this index.  You've simulated an advanced Oracle feature, reverse key indexes!

A few small gripes

If I were to add a few complaints it would be to say that some of the examples were rather simplistic.  In many cases tuning SQL is not as simple as just adding the right index.  For instance there was no good discussion of the dreaded "using temporary, using filesort" that we see a lot in MySQL explains when sorting has to be done, but will not fit in memory.  Or what about tmpdir=/dev/shm, how will that improve things?  What about UNION versus UNION ALL where appropriate.  Why does DISTINCT do a sort?

The book was also missing a discussion of triggers, stored procedures, when or if the query cache can cause problems and so forth.  Also the article link mentioned about chapter seven isn't the only missing link.  I followed links to optimizing sql  statements and it seems to go to a generic holding page.  Also the main link effectivemysql.com/book leads to an outline of an as yet unreleased title on Backup and Recovery.

All in all, well worth your money

However, other than these few gripes the book overall is a very welcome addition to the small family of MySQL books.  Get a copy quick before they're all gone!

There's much more we can do to speed things up, if we only turn over the right stones.  Whether you're using WordPress or not, many of these principals can be applied.  However we'll use WordPress as our test case.

Test Your Website speed

There are web-based speed testing tools that will help with this step.  Take a look at Webpagetest , Pingdom tools and Google's Chrome plugin Pagetest which integrates right into your browser.  If you're using firefox, take a look at YSlow .

If you've already got the WordPress plugin W3 Total Cache installed, that also integrates with Google Speed Test through an API key.  That integrates right into your WordPress dashboard.

1. Reduce objects

The first thing you can do to improve page performance and response is to make the page simpler.  Fewer images, fewer posts, fewer plugins, fewer widgets and so forth all contribute to a speedier page.  Obviously you don't want to forfeit functionality, but if you are loading 20 posts, you might want to reduce them to five or ten.   If you are calling out to third party APIs to load badges, twitter counts or SKYPE buttons, consider each and its affect on webpage load times.

If you have access to the underlying HTML, your application should try to reduce the number of DOM objects that are created.  These are created as the HTML is parsed.  Simpler structure here means faster page load times.

2. Compress objects

Your page is full of objects.  Many of them such as images, can be compressed.  This saves server storage space, and is a smaller object to copy across the network to the end users browser.  The web page test will show you a list of objects that are good candidates for compression.

3. Employ page & object caching

An object cache will store name/value pairs in memory for your application.  Essentially the webserver tier will cache data that it reads from the database to avoid additional round trips and network overhead.   For most of us, we'll want to get memcache installed in our webserver tier.

For a page cache, take a look at Varnish.  This can also be installed by package manager.  A page cache is like a tiny hyper efficient webserver.  It can sit on the webserver itself or on it's own server if you have many webservers in your environment.

Lastly if you have not already, grab the W3 Total Cache plugin for WordPress.  This plugin integrates directly with these two types of caches.  Simply click the Performance tab, select General Settings, and scroll to the object cache and varnish sections.

4. Browser caching

Browser caching is a tricky one.  You might think this is totally in the hands of the end user.  As it turns out, however the objects your webserver sends back to clients specifies various caching information in the HTTP headers.

Since we're encouraging you to test things yourself, fire up your command line terminal, and use "curl" to take a look at the headers of a file on your webserver.  Here's an example:

$ curl -I http://www.iheavy.com/files/pros.gif

HTTP/1.1 200 OK

Date: Tue, 01 Nov 2011 04:58:55 GMT

Server: Apache/2.2.3 (CentOS)

Last-Modified: Wed, 24 Aug 2011 01:06:09 GMT

ETag: "1649b1fb-5562-4ab35eadfe240"

Accept-Ranges: bytes

Content-Length: 21858

Content-Type: image/gif

Now we'll go ahead and enable W3 Total Cache in wordpress, then rerun the same curl command again:

$ curl -I http://www.iheavy.com/files/pros.gif

HTTP/1.1 200 OK

Date: Tue, 01 Nov 2011 05:01:27 GMT

Server: Apache/2.2.3 (CentOS)

Last-Modified: Wed, 24 Aug 2011 01:06:09 GMT

ETag: "5562-4ab35eadfe240"

Accept-Ranges: bytes

Content-Length: 21858

Cache-Control: max-age=31536000, public, must-revalidate, proxy-revalidate

Expires: Wed, 31 Oct 2012 05:01:27 GMT

Vary: User-Agent

Pragma: public

X-Powered-By: W3 Total Cache/0.9.2.3

Content-Type: image/gif

The main line we're interested here is the cache-control line.  Notice that the object has an expiration of 31536000.  Turns out that is the number of seconds in one week.  This tells the browser to keep these objects in its own cache and not refetch from the webserver each time.  That's a tremendous speedup.

5. Employ a CDN

If you haven't heard of CDN before, it stands for content delivery network.  Dropbox, the filesharing service is essentially a CDN service with a handsome user interface built on top of it.  Akamai is another famous CDN solution, one of the most mature, and respected available.  If you're on Amazon you might look at CloudFront, their CDN solution.

What is a CDN exactly?  Consider your pressing need for some basic groceries.  Maybe you need juice, a sandwich, beer or cigarettes.   If you're walking around the city, you'll find a deli on every corner.  That's the quickest way to get those items, a short walk and a short line.  When you need bigger items, you might make a weekly trek to Whole Foods.  From the perspective of food and beverage manufacturers you certainly want to get your products into the delis.  They are your grocery distribution networks, if you will.

W3 Total Cache supports Amazon CloudFront, simply enter your private and secret keys, and save.  Then upload all of your content up to S3 and you'll be benefiting from a CDN in no time.

Test again

Now that you've had a chance to look at all these caching opportunities, and hopefully implemented many of them in WordPress, rerun your Google Speed Test.

Google Page Test - Score of 91

We were able to raise our score from the mid-70's to 91 after tweaking various pieces of the site.  We haven't even implemented image sprites yet.

Hopefully this time around you'll see some objects loaded via CDN, more objects compressed, and proper expirations on most if not all content on your site.

CDN & caching are closer than you think

Most internet sites can make great gains in faster response time by looking in the right places.  Application and server rearchitecting give you a lot, but usually involve more complex and ongoing effort by developers.  However the basic caching improvements we described above will make a big difference as well.  CDN and caching technology is a lot more within reach than you think it is.

Move to the Valley, make shrewd investments in other startups and become insanely rich like Sean Parker? A Bit lofty perhaps. How about try, try again and succeed. Then reinvent yourself as a guru dishing out startup wisdom through your blog and publishing a book that ends up the top of the New York Times Bestseller's list. That's essentially what Eric Ries, author of The Lean Startup did.

True entrepreneurs fail many times before they succeed and continuously find opportunities to reinvent themselves. Ries is one of them. He's taken all that he's learned from his failures, and later successes, from his college years in the 1990s right through the dotcom crash, and packaged them into a guide for startups to consult in their quest for world domination. 

Like any business or management book that wants to capture readers' attention The Lean Startup has ground-breaking ideas or rather something that challenges traditional thinking.

How to be lean and light-footed

One of  the main themes is that startups can stay lean and light-footed - and in the process avoid millions of dollars in sunk costs on failed ideas - by  discovering what customers want as they are developing the product itself. Ries advocates a Minimum Viable Product, that is when rolling out new products companies can already run and even charge, before they decide whether the blue or the red version is going to sell more. From there on the process of improving and perfecting the product is a continuous one through listening to what customers want, although not through their written feedback but rather through their actions. This favors scientific methods of decision-making over what is viewed as failure prone hunches and guess work. And it makes sense for startups who simple don't have the financial resources to take big risks.

Written in simple down-to-earth prose, The Lean Startup is an easy and quick read with sufficient real world examples. The good thing about books these days is they don't have to end up as dated publications. Since Ries is building a brand around the whole methodology, one can find additional resources and discussions it in his blog (where it all began anyway), and through Lean Startup circles and wikis.

A recent post on SEO MOZ alerted us to an interesting story where spot instances spiked to $1000 per hour. Incredible as it seems, but the psychology of bidding can be rather perverse. Still, this kilo of a quote for mere server space is pretty outrageous. With Amazon's usual price at around 50 cents per hour for these 2xlarge instances, they could have made some serious money during this short burst had a transaction actually took place. To the skeptics among us, this spike smells uncannily like a couple of algos gone crazy, incidentally also on Amazon.

But it also speaks of over reliance on one cloud provider, specifically Amazon's continued dominance in the space. As an alternative to Amazon for computing power you could also go to a service like SpotCloud which sits on top of multiple clouds, so you're less likely to experience this type of burst of demand. Then again, the spot instance volatility is baked in. Still the lesson is instructive, and will surely spell further growth for Amazon.

Help! How To Become Slightly Happier and Get a Bit More Done

I've long overcome that sheepish feeling when browsing the Self-help section at the bookstore. Sure, How to Make Friends and Influence People or the Seven Steps to World Domination in your bookcase aren't exactly the sort of titles to suggest a deep intellect but I like to keep an open mind when checking out the latest hardcover secret to happiness and prosperity. Basically I try not to diss a book just because it's got "soup" on the cover.

I will concede that publishers have gone a bit overboard with churning out the number of self-help titles in the last 20 years or so. As with anything that proliferates you're stuck with having to wade through the swamp of well, BS. HELP! How to Become Slightly Happier and Get a Bit More Done by Oliver Burkeman is ideal for those curious enough about self-improvement but too cool to buy into mind-body-soul mantras.

Since 2006 Burkeman has been writing a column in The Guardian, self-mockingly titled, This Column Will Change Your Life. It has garnered a large enough following and accumulated a substantial amount of content to warrant a book version or if you like, a best-of collection.

Approaching the subject with rational skepticism, Burkeman has made it his mission to sort through the best and the worse of self-help literature to tell us what works and what doesn't. He doesn't hesitate to call out the snake oil salesmen nor does he dismiss all self-help advice as hokum because he genuinely believes there is something we can all learn from research in human-happiness, however flawed.

I reckon Burkeman's greatest achievement is his ability to strip down what has been pedaled as new enlightenment by the self-help gurus, into realistic terms with wry humor and wit. I say realistic, because he reminds us that self improvement is a lifelong project for anyone who cares to make the effort, and it would be ridiculous to take a prescriptivist approach. From managing your inbox to procrastinating less, the topics he tackles are easy to relate to; for example, why your friends' Facebook statuses usually make their lives seem more awesome than they actually are or his explanation of the Color of the Bike Shed phenomenon, where the time spent on any item is inversely related to its cost and importance.

Although I sometimes wonder why buy a book when you can read all its contents online, Burkeman's writing has obviously been compelling enough for its publishers to re-release the title with a new so-called mass-market cover (notice the cute belly-up turtle). Even if it doesn't change your life, HELP! will definitely entertain.

Restaurant Scalability

Could pro-waitering serve up some lessons on web scalability? Observing peak hour dining at a New York restaurant gave us some insight.

I was dining at a restaurant the other day with friends. It was a warm and cozy place, nicely decorated with a long, narrow dining room.  The food was scrumptious, yet we were getting increasingly frustrated by the service as the night went along.

With some waiting experience behind me, I could immediately see the problem. The waiters, probably through lack of experience, were making the mistake of doing one thing at a time.  They would go to a table, respond to one customer's request, and go and fetch that item.  Back and forth, back and forth they would dart, but always dealing with one request at a time.

As the night got busier the demands began weighing in, like the insidious lactic acid build-up in a distance runner. Soon the incessant gesturing and requests from diners would turn the once beaming, confident servicemen into distressed drones evasive of eye-contact for fear of having to fulfill yet another request for a water top-up.

I knew how they felt because this happened to me many times when I first learned to wait tables.  Until a seasoned veteran waiter showed me how it was done I used to scamper about like a headless chicken. He explained to me that I should enter the dining room, look up, walk to each table and see if they needed anything.  Each time walking around to all the tables in my station, I would then return to fetch many requests all at once.  A fork or napkin for this customer, a glass of wine or condiments for another.  I would then put all of those items on my tray and return to satisfy all those requests in parallel.

Observing the waiters that night, it struck me that this is very much what web scalability is about.

Serialization

Serialization in web applications for example is like when the waiter fetches only one item at a time.  So-called decoupling of services is like delegating out the table cleanup to a busboy so you can concentrate more on customers.

Complex Infrastructures

Gordon Ramsay often makes his first move paring down the menu.  Complex menus can be confusing to customers, but they can also complicate things in the kitchen and thereby reduce quality.  So too can complicated infrastructures.  Years ago we had a client whose environment included two different webservers (nginx and Apache), three programming languages (Python, Ruby, and PHP), inconsistent versions across them and load balancing only across apache.  If that wasn't enough they had MySQL databases scattered randomly across machines, some of which were not backed up.   This kind of complexity breeds all sorts of problems.

Be Mindful of Scaling 

While developing your application, bake scalability into the mix.  Include multiple database handles for read and write databases.  Back to our restaurant analogy;  this is similar to opening shop in a building that has a adjacent vacant unit available.  As your business grows, you can occupy that additional space fairly easily because you put a little extra thought into your location at the beginning.

Exciting news, Oracle just announced commercial MySQL extensions that they'll be offering paid extensions to the core MySQL free product.

To be sure, this has raised waves of concern among the community, but on the whole I suspect it will be a good thing for MySQL.  This brings more commercial addons to the table, which only increases the options for customers.  Many will continue to use the core database product only, and avoid license hassles while others will surely embark on a hybrid approach if it solves their everyday business problems.

Speaking of licensing, back in May, Amazon announced that it's RDS or Relational Database Service would now offer Oracle as an option.  For those familiar with negotiating Oracle licensing, and the concomitant headache, this is certainly a welcome addition.  It also marks a dramatic break from the previous per-cpu or per-seat licensing models, and Oracle's embrace of the cloud model of instant provisioning.  Interesting all around!

For those concerned about MySQL as a whole going commercial, there are three large companies continuing to support and distribute the core database including Percona, Monty Program AB (MariaDB) and Brian Aker's fork Drizzle.  These distros have strong community support, and that strength will only grow.

Once a development or operations team gets over the hurdle of open-source, and start to feel comfortable with the way software works outside of the enterprise world, they will likely start to settle in and feel comfortable.  Best not to get too cushy though for there are more surprises hiding around the corner.  Here are a few of the biggest ones.

1. Replication Is Not Perfect

Yes, you've installed MySQL, setup a slave, and got it replicating properly.  You check the slave and it's 0 seconds behind the master.  What's more you monitor the error log file, and have a check in place to alert you if something happens there.  Job completed, good job!

Not so fast.  Unfortunately this is not the end of the story.  Many MySQL replication slaves are not consistent with their masters, but they drift apart silently.  Really?  Yes really.  But there are tools to help you.  Maatkit includes a checksum tool that will build checksums for all of your tables, and populate a table with that data.  Your checksum data then flows through replication to the slave.  From there you can perform a check on the slave to see if there are any differences.  Such a check is easy to wire into nagios or other monitoring software, to alert you as soon as a difference materializes in your slave database.

With vigilant data integrity checking in place, your MySQL slave is robust enough to perform backups off of.  But beware if you're not using tools to check and relying only on the error log.  This is unfortunately not sufficient.

2. Queries With Bad Performance

MySQL's optimizer is still fairly rudimentary.   There are some classes of queries that MySQL does not handle well.  If I were to provide a general rule it would be to stick to fairly simple queries, with at most a few joins.  Simpler lookups are sure to be fast.  Also less complex and convoluted queries are easier to understand, optimize and tune later on if they require it.

Subqueries

Although MySQL supports these, they often optimize badly, and perform terribly.  Review an explain plan to be sure you're optimal.  Replace such a query with a join where possible.

Joins

MySQL only supports the nested loops algorithm to do joins.  Merge join and hash join, two alternate and very efficient algorithms have not yet been implemented in MySQL unfortunately.  Be aware, run explain and profile your slow queries.

Using Temporary; Using Filesort

When MySQL has to do sorting, it does so in one of two ways.  If the dataset is small it'll use a quicksort, and do it in memory.  In this case you'll see just "using filesort".  However if  you see "Using Temporary; Using Filesort" in an EXPLAIN plan output, it means MySQL is using a two-pass algorithm to do a sort, creating a temporary table because it won't all fit in memory.   Sometimes a composite or multi-column index can help with these situations.  You may also affect this behavior by setting sort_buffer_size and join_buffer_size at the session level, then rerunning the explain.

Newer versions of MySQL such as 5.6 have gotten much smarter about this, as the optimizer is always being tweaked and improved.  Check your query plans in dev & test before migrating your production database.

Single Index Usage

Although MySQL supports very efficient b-tree indexes, it cannot always use multiple indexes, or merge indexes together for more efficient path to data.  Although there are some exceptions, figure at each stage of a query and on each table MySQL will likely choose only one index.  When you're not satisfied with this, you may choose to build a covering or multi-column index.  Keep in mind these will only serve that one particular query, or a query against the leading edge column.  Also keep in mind that the more indexes you have built on a table, the less performant inserts, updates and deletes will be against this table.  All of those operations will require index maintenance so you must balance read performance (indexes on more columns) with write performance (requiring index maintenance and updating).

3. Where Are The Transactional Tables?

Yes MySQL does offer row-level locking, ACID compliance, and repeatable read and other isolation levels and referential integrity.  In fact it's implementation of the buffer cache to support reading and writing blocks from disk is modeled on Oracle's own algorithm.  So why are you getting all of that?

As it turns out MySQL supports various Storage Engines, unlike other databases which have all of this rolled into the kernel, and only support transactional tables.  This can be thought of as a feature although in reality it is more of a anomaly of how MySQL has evolved, and where it's roots lie.  As such the default storage engine is MySQL which is not transactional.  All changes are effectively auto-commit, and it only supports table level locking.

If you haven't checked already, verify what table types you have as follows:

mysql> select table_schema, engine, count(*) from information_schema.tables group by table_schema, engine;

Outside of system tables, your application tables should all be InnoDB.  You can modify an existing table to be transactional as follows:

mysql> alter table mytable engine=innodb;

You should also set storage_engine variable to default to Innodb.  Finally you may also choose to set the sql_mode NO_ENGINE_SUBSTITION which will force MySQL to flag an error whenever the storage engine you choose is not available.  This prevents it from creating a table with a storage engine other than your choice.

1. Object Relational Mappers

ORMs are popular among developers but not among performance experts.  Why is that?  Primarily these two engineers experience a web application from entirely different perspectives.  One is building functionality, delivering features, and results are measured on fitting business requirements.  Performance and scalability are often low priorities at this stage.  ORMs allow developers to be much more productive, abstracting away the SQL difficulties of interacting with the backend datastore, and allowing them to concentrate on building the features and functionality.

On the performance side the picture is a bit different.  By leaving SQL query writing to an ORM, you are faced with complex queries that the database cannot optimize well.  What's more ORMs don't allow easy tweaking of queries, slowing down the tuning process further.

2. Synchronous, Serial, Coupled or Locking Processes

Locking in a web application operates something like traffic lights in the real world.  Replacing a traffic light with a traffic circle often speeds up traffic dramatically.  That's because when you're out somewhere in the country where there's very little traffic, no one is waiting idly at a traffic light for no reason.  What's more even when there's a lot of traffic, a traffic circle keeps things flowing.  If you need locking, better to use InnoDB tables as they offer granular row level locking than table level locking like MyISAM tables.

Avoid things like semi-synchronous replication that will wait for a message from another node before allowing the code to continue.  Such waits can add up in a highly transactional web application with many thousands of concurrent sessions.

Avoid any type of two-phase commit mechanism that we see in clustered databases quite often.  Multi-phase commit provides a serialization point so that multiple nodes can agree on what data looks like, but they are toxic to scalability.  Better to use technologies that employ an eventually consistent algorithm.

3. One Copy of Your Database

Without replication, you rely on only one copy of your database.  In this configuration, you limit all of your webservers to using a single backend datastore, which becomes a funnel or bottleneck.  It's like a highway that is under construction, forcing all the cars to squeeze into one lane.  It's sure to slow things down.  Better to build parallel roads to start with, and allow the application aka the drivers to choose alternate routes as their schedule and itinerary dictate.

4. Having No Metrics

Having no metrics in place is toxic to scalability because you can visualize what is happening on your systems.  Without this visual cue, it is hard to get business units, developers and operations teams all on the same bandwagon about scalability issues.  If teams are having trouble groking this, realize that these tools simple provide analytics for infrastructure.

There are tons of solutions too, that use SNMP and are non-invasive.  Consider Cacti, Munin, OpenNMS, Ganglia and Zabbix to name a few.  Metrics collections can involve business metrics like user registrations, accounts or widgets sold.  And of course they should also include low level system cpu, memory, disk & network usage as well as database level activity like buffer pool, transaction log, locking sorting, temp table and queries per second activity.

5. Lack of Feature Flags

Applications built without feature flags make it much more difficult to degrade gracefully.  If your site gets bombarded by a spike in web traffic and you aren't magically able to scale and expand capacity, having inbuilt feature flags gives the operations team a way to dial down the load on the servers without the site going down.   This can buy you time while you scale your webservers and/or database tier or even retrofit your application to allow multiple read and write databases.

Without these switches in place, you limit scalability and availability.

Want more?  Signup for our scalability newsletter.

1. Deploy with Roll Forward & Rollback Scripts

When developers check-in code that requires schema changes, that release should also require two scripts to perform database changes. One script will apply those changes, alter tables to add columns, change data types, seed data, clean data, create new tables, views, stored procedures, functions, triggers and so forth. A release should also include a rollback script, which would return tables to their previous state.

This idea of database change management is popular as Migrations in Ruby on Rails. It is a convenient way to enforce good coding, and better database change management. However, keep in mind it has it's limits. See Perform Backup section below.

2. Ban SELECT * FROM table

Whenever you select from a table, be sure to specify the column names in all of your SQL code.  That way if later features or code changes add columns to that table, there is no potential to break old code that did SELECT * and got the old table column structure.  If the table was formerly returning four columns and after your code and schema changes returns five columns, it might break code that uses SELECT *.

The same goes for update statements.  Be sure to specify columns explicitely, and have a policy to enforce this on developers.

3. Perform Backup on Affected Data

Database rollback scripts have their limits, so a full database backup on any affected tables, or to be thorough the entire database, is a fully insured way to deploy database changes.

Since the developer's script cannot account for data in a live database, a rollback script will not work on database changes that do any of the following.  Only a backup will suffice:

• drop a table
• drop a column
• delete any data

4. Build in Feature Flags

Feature flags provide the operations team with a mechanism to turn on or off a new feature.  If you're going the extra mile to build in these switches you will make rolling out new features a lot more trouble free.  Even better, include a mechanism to enable or disable a feature for a percentage or class of users.  That allows you to dial the feature on gradually or expose it only to a certain group of users.   In an emergency this will also come in handy as a way to quickly disable a new feature that has gone awry.

5. Involve DBAs in the Review & Test Process

Before deploying a new release, include a process whereby the DBA must review schema changes and scripts.  This provides an additional sanity check, and a Database Administrators perspective on those changes.  He or she may well provide insight or caution on potential gotchas that you might encounter.