Scanning Crains NY Business recently, I saw an article on 'starting up' in 54 hours.  It's the brainchild of Marc Nager, Clint Nelsen and Franck Nouyrigat called Startup Weekend. Startup bootcamps seem to be the current extra-curricular activity of choice these days. Wharton is also getting in on it with their Innovation Tournament. Then there is the 48 Hour Startup  and of course let's not forget the 3 Day Startup.

So what's my beef?  Truth be told I admire the ambition, the optimism, and the openness of these efforts.  And for sure these bootstrapping marathons do introduce entrepreneurs to future colleagues and partners, get them asking the right questions about financing, customers, revenue, competition and so forth.

My problem with these events is they frame startups as something you *can* do quickly. As if it were a Lego set or pop-up book that gives instant results and gratification. Sure startups are 21st century tech-driven business that provide innovative products in a very short development cycle but a lot of the day-to-day running of the business are still very mundane 20th century sensibilities; not unlike running a mom and pop store, a laundromat, deli or sandwich shop.

No shortcuts

You have to learn how to ask your customers what they want, learn how to sell them something they need, and be prepared to be surprised in the process. I've worked with many startups and have seen often how many have had to throw out an idea that didn't work and re-strategize. This takes courage an conviction.

One startup client was building iPhone apps for event management. After testing it on the market it turned out nobody really wanted an event management app.  But there was a market for professional and consulting services that can help build technology platforms, and architect the right solutions.  So they shifted gears and moved into that space.  They've grown tenfold since.

Another startup client's change in direction was so radical that it even warranted a name change. Buzzd, started out as a city guide for consumers based on aggregating Facebook, FourSquare and Gowalla check-ins. Last October Buzzd became LocalResponse and turned its focus onto advertisers instead.

This kind of honest self-analysis and business acumen so crucial for survival can only be acquired through experience. For example managing overheads or being aware that profitability does not always mean solvency. And then there are decision taking and setting priorities. Repair that wall or build a new display case? Use that money on product and inventory?  Attract customers to your store?  What's the right price point to settle on?  How do you avoid legal entanglements, and who are some of the predators to watch out for?

Startup crash courses are probably a great way for  aspiring entrepreneurs to network, and for VCs to spot the next killer investment. But at the same time, it feels suspiciously like a diet fad that promises to change 30+ years of behavior overnight, and get you looking like a bikini model in no time.  Fitness and health are lifestyle choices, as is building a successful business. As much as we like to think otherwise, there are simply no shortcuts.

Who needs Scalability Rules?

Looking at the fifty rules divided up this way, summarizing the book becomes easier  but still a bit muddled.  For example I might ask the question, Who exactly is the book’s audience?  Is it application developers?  Certainly, there is a large section of rules designed for them.  Or maybe it's for the operations team, offering advice on how to design systems, networks and data centers, and advice for optimizing and redirecting development efforts?  Or maybe it is a book for managers, as the people and process related rules certainly are targeted towards those folks.  But if that's the case then much of the book is probably too technical for such an audience.

Ok, so let's step back, when or why might you have all these crisscrossing themes and roles?  The answer is that like me, the authors are consultants and in the course of engagements a consultant is often asked to wear many different hats.  So sometimes we're looking at scalability in regards to operational decisions, while at other times we're looking at scalability through the lens of application design decisions. And still other times scalability can best be achieved by focusing on people and processes. I’d add that not only is this a handy reference for consultants, but for any bootstrapping startup as well. Coders these day are expected to be at least a little bit knowledgeable with web operations, and vice-versa.  Especially pushing towards the guiding principles of devops, this book can help engineers take a critical look at their roles within an organization.

Looked at in that light you can really see how this book is truly an opus of knowledge on scaling web sites.  Although the organization is sometimes confusing, it will surely bring insights and enlightenment to all of your teams, whether operations, devs, managers, or business units.

Walking around New York you find yourself stopping at plenty of different places to grab some takeout for lunch. There are Vietnamese sandwich places, pizza shops, noodle bars, taco stands, juice bars and of course your daily coffee shop. You'll find an endless variety.

As is customary in New York, even for takeout there is usually a tip jar at the checkout. Many of them have a large bowl, or glass jar in which you can throw your change as tips, or if you really love the place and service, a couple of dollars.

Of late I've noticed a few have placed those small plastic boxes with a tiny slot on the top. You try to put some change in the slot, and half of the money falls on the floor. It's as frustrating as threading a needle while suffering from astigmatic vision. Now when I come to a place that has this plastic box, I don't even bother tipping. I get a headache thinking about my change falling all over the floor. All I keep thinking is, why make it so difficult to tip?

Lessons from tipping

There're business lessons here to be sure. The most obvious being, when working with clients, make it easy for them to pay you. Don't ask them to jump through hoops to settle a bill. That goes just as well for complicated costs and itemization. Keep your invoices as simple as possible, so it's clear later on what it covers; the amount and the period. Do they like to pay by ACH or Wire, make that easy as well. Would they rather pay by Paypal? Perhaps have a carrier pigeon with a check? No problem!

This lesson might also be applied to add-on features and services. In many cases a client may require additional time or service from you. Does this involve a lot of approvals or details to be hashed over? Or is it already covered in your base contract so you both know what qualifies and how it will be billed? If you are in the design business, what's the cost of an additional design version or proof? Web operations, what's the cost of adding in another on-site day or weekend block of time?

We may not realize it but sometimes we fumble in the process of acquiring new business too. Recently I was looking to hire a web developer for a website facelift project. I liked the work of one of the applicants and tried to arrange for a quick chat over Skype. To my surprise they said they couldn’t get on Skype because they didn’t have an account. Granted they did offer to call me on my mobile but I’d also wanted to set up a three-way conference call. As a prospective client, I was amused at having to make the suggestion that they register for Skype in order to arrange the call. Someone more on the ball would’ve just said OK, and sign up for Skype right away.

Ultimately the lesson is to think in your customers’ shoes make it easy for your client to reward you when they're happy with your service, and want to buy more of what you are selling.

Business Insider posted this spectacular interactive google map of our amazing Startup Ecosystem here in New York City.  Wow!

Check out the full list of NYC Startups in the original article.

View NYC Startups in a larger map

Strip by Randall Munroe; xkcd.com

First things first. This is not meant to be a beef against developers. But let’s not ignore the elephant in the living room that is the divide between brilliant code writers and the risk averse operations team.

It is almost by default that developers are disruptive with their creative coding while the guys in operations, those who deploy the code, constantly cross their fingers in the hope that application changes won’t tilt the machine. And when you’re woken up at 4am to deal with an outage or your sluggish site is costing millions in losses, the blame game and finger-pointing starts.

If you manage a startup you may be faced with this problem all the time. You know your business, you know what you're trying to build but how do you find people who can help you build and execute your ideas with minimal risk?

Ideally, you want people who can bridge the mentality divide between the programmers eager to see feature changes, the business units pushing for them, and the operations team resistant to changes for the sake of stability.

DevOps - Why can’t we all live together?

The DevOps movement is an attempt to bring all these folks together. For instance by providing insight to developers about the implications of their work on performance and availability, they can better balance the onslaught of feature demands from users with the business’ need for up-time.

Operations teams can work to expose operational data to the development teams.  Metrics collection and analytics aren’t just for the business units anymore. Employing tools like Cacti, OpenNMS or Ganglia allow you to communicate with developers and other business units alike about up-time, and the impact of deployments on site availability, and ultimately the bottom-line.

Above all, business goals and customer needs should underscore everything the engineering team is doing. Bringing all three to the table makes for a more cohesive approach that will carry everyone forward.

How to spot a DevOps person - Finding the sweet spot

The DevOps person is someone with the right combination of skill, knowledge and experience that places him or her in the sweet spot where quality assurance, programming skills and operations overlap.
There are also a few distinguishing characteristics that will help identify such an ideal candidate.

Look for good writers and communicators

Imagine the beads of sweat forming when a developer tells you: “We’ve made the changes. Nothing is broken yet.”
This is like stepping on glass because it implies something will actually break. The point is savvy developers should be aware that the majority of people do not think along the same lines as they do.
Assuming your candidate has all the required technical skills, a programmer with writing skills tends to be better at articulating ideas and methods coherently. He or she would also be less resistant to documentation and be able to step back somewhat from the itty-bitty details. Communication, afterall is at the core of the DevOps culture where different sides attempt to understand each other.

Pick good listeners

Even rarer than good writers are good listeners. Being able to hear what someone else is saying, and reiterate it in their own terms is a key important quality. In our example, the good listener would probably have translated ‘nothing is broken yet’ into “the app is running smoothly. We didn’t encounter any interruptions but we’ll keep watch on things.”

Lean towards pragmatists and avoid the fanatics

We all want people who are passionate about something but when that passion morphs into fanaticism it can be unpleasant. Fanaticism suggests a lower propensity to compromise. Such characters are very difficult to negotiate with. Analogously in tech, we see people latch on to a certain standard with unquestioning loyalty that's bafflingly irrational. Someone who has had their hand in many different technologies is more likely to be technology agnostic, or rather, pragmatic. They'll also have a broader perspective, and are able to anticipate how those technologies will play together.  Furthermore a good sense of where things will run smoothly and where there will be friction is vital.

Pay attention to extra-curricular activities

Look at technology interests, areas of study, or even outside interests. Does the person have varying interests and can converse about different topics?  Do they tell stories, and make analogies from other disciplines to make a point?  Do they communicate in jargon-free language you can understand?

Sniff out those hungry for success

As with any role, finding someone who is passionate and driven is important.  Are they on-time for appointments? Did they email you the information you requested? Are they prepared and communicative?  Are they eager to get started?

Hiring usually focuses on skills and very well-crafted resumés but why do you still find some duds now and then? By emphasizing personality, work ethics, and the ability to work with others, you can sift through the deluge of candidates and separate the wheat from the chaff for qualities that will surely serve your business better in the long run.

Recently at Surge 2011, the annual  conference on scalability  and performance, Google's CIO Ben Fried gave an illuminating keynote address. His main insight was that generalists are the people that will lead engineering teams in successfully scaling the web.

Restaurant Scalability

Could pro-waitering serve up some lessons on web scalability? Observing peak hour dining at a New York restaurant gave us some insight.

I was dining at a restaurant the other day with friends. It was a warm and cozy place, nicely decorated with a long, narrow dining room.  The food was scrumptious, yet we were getting increasingly frustrated by the service as the night went along.

With some waiting experience behind me, I could immediately see the problem. The waiters, probably through lack of experience, were making the mistake of doing one thing at a time.  They would go to a table, respond to one customer's request, and go and fetch that item.  Back and forth, back and forth they would dart, but always dealing with one request at a time.

As the night got busier the demands began weighing in, like the insidious lactic acid build-up in a distance runner. Soon the incessant gesturing and requests from diners would turn the once beaming, confident servicemen into distressed drones evasive of eye-contact for fear of having to fulfill yet another request for a water top-up.

I knew how they felt because this happened to me many times when I first learned to wait tables.  Until a seasoned veteran waiter showed me how it was done I used to scamper about like a headless chicken. He explained to me that I should enter the dining room, look up, walk to each table and see if they needed anything.  Each time walking around to all the tables in my station, I would then return to fetch many requests all at once.  A fork or napkin for this customer, a glass of wine or condiments for another.  I would then put all of those items on my tray and return to satisfy all those requests in parallel.

Observing the waiters that night, it struck me that this is very much what web scalability is about.

Serialization

Serialization in web applications for example is like when the waiter fetches only one item at a time.  So-called decoupling of services is like delegating out the table cleanup to a busboy so you can concentrate more on customers.

Complex Infrastructures

Gordon Ramsay often makes his first move paring down the menu.  Complex menus can be confusing to customers, but they can also complicate things in the kitchen and thereby reduce quality.  So too can complicated infrastructures.  Years ago we had a client whose environment included two different webservers (nginx and Apache), three programming languages (Python, Ruby, and PHP), inconsistent versions across them and load balancing only across apache.  If that wasn't enough they had MySQL databases scattered randomly across machines, some of which were not backed up.   This kind of complexity breeds all sorts of problems.

Be Mindful of Scaling 

While developing your application, bake scalability into the mix.  Include multiple database handles for read and write databases.  Back to our restaurant analogy;  this is similar to opening shop in a building that has a adjacent vacant unit available.  As your business grows, you can occupy that additional space fairly easily because you put a little extra thought into your location at the beginning.

One of the great things about the Internet is how it has made it easier to put great ideas into practice. Whether the ideas are about improving people’s lives or a new way to sell and old-fashioned product, there’s nothing like a good little startup tale of creative disruption to deliver us from something old and tired.

When migrating to the cloud consider security and resource variability, the cultural shift for operations and the new cost model.

1. Costs Move to Operating Expenses

When computing resources move from hardware, components and infrastructure that you buy and own, to those you rent by the hour, they also change places on your balance sheet.  They move from being capital expenses that depreciate over time to operating expenses.

At the end of the day, this is a very good thing because of the idea of discounted cash flow, and cost of capital in accounting.  It's always better to spend your money later.

That said it shifts how servers are provisioned, and how business units, and managers think about those expenses.  So expect some hiccups in that regard.

2. Operations & Automation - Cultural Shift

Best practices include solid disaster recovery, backups and then fire drills to prove all your ducks are in a row.  But best practices often get sidelined for more pressing concerns, and other obstructing priorities.

With virtual cloud servers, you start with a server that is less reliable to begin with, and a lower SLA to boot.  So automation from your starting point moves to front and center.  This requires a more managed approach to rolling out servers, akin to a small software development team in the habit of rolling out code manually, being pressed to move into version control.  This new managed approach to web operations is a good thing, encouraging configuration management, and automating resource provisioning tools like Chef.  Still it is a cultural shift, and one that won't happen overnight.

3. Security

Although the Amazon environment does not provide perimeter security, the security groups model can provide equally good security if managed well.  But one also has to manage other things closely such as:

• authentication keys
• machine images - built without sensitive data in the AMI

Furthermore compliance questions are often on everyones mind.  While you cannot say for sure where your data is physically located in the AWS environment, you can control whether it is encrypted or not, and you control those encryption keys.  Nevertheless legal requirements may have other plans in this regard.

Consider also what would happen if your cloud provider were hit with a lawsuit that being overly broad, sweeps some of your servers into its net.  How will you handle such a scenario?  Use multiple cloud providers and test your disaster recovery scenarios.

4. Resource Variability

Like moving from big iron to commodity servers fifteen years ago, cloud hosted environments can take some getting used to.  In the virtualized environment, the CPU, disk and network resources you are allocated are based on computed averages.  At any time the disk I/O throughput for example can fluctuate higher, or quite a bit lower than the service level agreement for that instance type.

All of the practices and methods for promoting scalability, decoupling components, caching agressively, and scaling horizontally will reduce the impact of these resource variabilities.  In the same way the internet masses can descend on your website without warning, building a robust infrastructure to stand up to the vagaries of virtualization will go a long way towards helping you scale smoothly when traffic spikes.

Spencer Johnson is a great writer.  His business book classic was a real page turner.  He takes a page from the REWORK book and that's a good thing.

Who Moved My Cheese is a story about mice living in a maze happy and content that they have an unlimited supply of cheese.  Then one day the cheese runs out.  Some approach this new reality by sniffing, and scurrying.  Before long they find new cheese, and more opportunities than they had before.  Some other mice hem and haw, and resist the new reality, pointing fingers, getting angry, but not facing the situation.

Of course these are situations we all face everyday.  Outsourcing changes the job landscape, technologies such as the internet disrupt established industries like newspapers, music, and media.  What's more the recent upheaval in the financial industry had at least some basis in the computerization of trading, and the acceleration of global markets.

The landscape is always changing.  Johnson's book speaks great lessons that we can all understand, and benefit from everyday in business.

Find Who Moved My Cheese on Amazon.

Shoe leather cost is similar to opportunity cost.  It refers to the cost of counteracting inflation by keeping less of your assets in cash.  Your strategy would require more trips to the bank and more walking, and incur a cost in the wearing out of the leather in your shoes.

All joking aside, it's an interesting idea.  It highlights how there are all sorts of hidden costs to different strategies.  There are hidden costs to using coupons, loyalty cards, frequent flyer miles, managing assets & investments, hiring resources and in general running a business.  Let's look at a few.

Cost of R&D

Years ago I wanted to open a mutual fund and did some research.  Luckily a friend pointed me towards Vanguard, and I got on the index investing bandwagon early.  I've had a lot of opportunities to buy stocks directly, but I've seen too many surprises in the market even in areas of tech stocks and business that I understand very well.

What's more stock investing requires attention, research into fundamentals, and certainly a bit of luck.  In my case my background is engineering, so finance is not my strong suit.  In effect there is more cost hidden there, getting up to speed in a new discipline and the many hidden surprises and fine print.

By choosing an index fund strategy you track the market, are guaranteed to beat 90% of stocks and mutual funds over time, pay a very very small fee as those funds are not actively managed by a fund manager with costs and overhead.  What's more the shoe leather cost is very low.   It's a no brainer.

Cost in People Search

Whatever technologies you choose, you embark down a path with costs on all sides.  Costs of hardware & software components, costs of licenses, and costs to find people with those skill sets.  Deciding to use the latest wiz-bang technology as an early adopter that looks like it will provide you a lot of extra bang for your buck.

Consider carefully what the market adoption of that technology is, as you will inevitably have to find resources skilled in servicing those components down the line.  Lean heavily toward widely adopted technologies and you'll have an easier time finding skilled technologists in the future.

Not Your Core Business

Early on publishing our newsletter, we chose to host the newsletter software ourselves.  The open-source solution was robust and full featured.  However many of the features proved cumbersome to configure.  Since we had the basic functionality working, we left the more complex switches alone.  This solution was a free one since we already hosted a server for the website.

Eventually we switched to mailchimp.com to host the newsletter.  Suddenly many hidden costs came to light.  By hosting ourselves we were ending up in spam folders for a lot of users.  So we were reducing our audience.  Plus we could not track who was reading the newsletter, which issues were popular, or what sections.  That analytical data provide invaluable.  What's more mailchimp integrates and automates a lot of other functions that were done manually before.  Freeing up time!

To Sum Up

Whenever you are spending inordinate time on something that is not your core business, chances are the shoe leather cost is high, and outsourcing that to a provider who specializes in that become a great cost saver.

Rework is chock full of ideas

Jason Fried and David Heinemeier Hansson's new book REWORK is one of the best startup business books I've read since Alan Weiss' Million Dollar Consulting. If you're already a fan of their signal vs noise blog, you'd be familiar with their terse style. Sharp and to the point.

Which is why you can pick it up and read it in a few hours.  You'll want to because it's well written and pared down to essentials.  In fact the book reads like their workflow advice, less mass, do it yourself, cut out the fat, concentrate on essentials.  As such they are clearly practicing what they preach, which I like.

This book isn't really for large businesses and corporate success.  It's for startups, and freelancers, and small teams.  It focuses on how and why you should do something yourself before outsourcing it so you can learn what to look for.

Here are some choice quotes from the book:

1. The menus in failing restaurants offer too many dishes.
2. When you impose a deadline you gain clarity.
3. Timeliness is more important than polish
4. Reality never sticks to best-case scenarios.
5. Do less than your competitors to beat them.
6. Share information that's valuable and you'll slowly build a loyal audience.
7. Trade the dream of overnight success for slow, measured growth.
8. Getting back to people quickly is the most important thing you can do in customer service.
9. Optimize for now and worry about the future later.
10. You don't need more hours you need better hours.

In fact I started out with a list of my favorite twenty quotes from the book and cut it down to these ten.  Go get this book, you'll be glad you read it.

Find REWORK on amazon

A very nice article just appeared in Crains New York Business this past week covering interesting people to watch in Gotham's tech scene.

With such a long list of tech startups, Silicon Alley is nearly bursting at the seams!

• Huffington Post
• Foursquare
• Rent the Runway
• BankSimple
• Knewton Inc.
• GroupMe
• Kickstarter
• Tumblr
• Tremor Media Inc.
• MyCityWay
• Bitly
• Yodle
• Boxee Inc.
• Tabula Digita
• AppNexus
• Yext
• Lot18
• Tapad
• Etsy
• TheLadders.com
• SecondMarket
• GawkerMedia

I meet new people all the time.  It's a way of life in New York.  One of the first questions new people ask each other is "What do you do?".  It begins to sound like a cliche after a while, but it can also provide endless fascinating discussions as there are so many people with different professions in New York.  Some choose a titled answer "i'm an investment banker", "I'm an emcee", "I'm an executive recruiter".  I find for "Web Scalability Consultant" or "Web Operations Expert" this only leaves confused looks.

A Plumber By Another Name

The solution of course is to tell a good story.  Stories illustrate what titles and crusty vernacular cannot.  I've used analogies to surgeons or mechanics, of course they all operate on something people can related to in front of them.  People or vehicles we use everyday.  Of course with the internet, there is a huge hidden infrastructure that most people don't see everyday.  They may vaguely know it's there, but it's still hidden out of site.

That's why I think plumbing provides such an apt visual.  As it turns out the internet is built with countless data pipes both large and small, coming into your home or laying across the bottom of the transatlantic ocean.  These pipes plug into routers, high speed traffic lights and traffic cops.  Ultimately they feed into datacenters, huge rooms filled with racks of computers, holding your websites crown jewels.  Therein contains the images and status updates from your facebook profile, your banking transactions from your personal bank account or credit card, your netflix movie stream, or the email you sent via gmail.  Even your instant messaging stream, or the data from your favorite iphone app are all stored and retrieved from here.

Amazon Outage

The recent Amazon outage has been high profile enough that a lot of folks who don't follow the latest trends in web operations, devops, and datacenter automation still heard about this event.  Turns out it's had a silver lining for Amazon cause now everyone is scrutinizing how many sites actually rely on this goliath of a hosting provider.

As it turns out the root of the amazon outage was indeed a plumbing problem.  Amazon has shown rather high transparency publishing intimate details of the problem and it's resolution.  Read more.

A misconfigured network cascaded through the system creating countless failures.  If you imagine water repairs being done in a large New York City building, they often ask tenants to turn off their water, so they won't all come on at the same time when service is restored.  SImilarly intricate problems complicated the Amazon effort, slowing down attempts to restore everything after the incident.  I wrote at length about the outage if you're interested, read more.

BOOK REVIEW:  Game-Based Marketing by Zicherman & Linder

There are so many new books coming out all the time, it's tough to sift and find the good ones.  Anyone with a website as their storefront, whether they are a product company or a services company, can gain from reading this book.

From leaderboards to frequent flyer programs, badges and more this book is full of real-world examples where game-based principles are put into action.  On the internet where attention is a rarer and rarer commodity, these concepts will surely make a big difference to your business.

Amazon book link - Game Based Marketing

Now that we've had a chance to take a deep breath after last week's AWS outage, I'll offer some comments of my own.  Hopefully just enough time has passed to begin to have a broader view, and put events in perspective.
Despite what some reports may have announced, Amazon wasn't down, but rather a small part of Amazon Web Services went down.  A failure, yes.  Beyond their service level agreement of 99.95% yes also.  Survivable, yes to this last question too.

Learning From Failure

The business management conversation du jour is all about learning from failure, rather than trying to avoid it.  Harvard Business Review's April issue headlined with "The Failure Issue - How to Understand It, Learn From It, and Recover From It".  The economist's April 16th issue had some similarly interesting pieces one by Schumpeter "Fail often, fail well",
and another in April 23rd issue "Lessons from Deepwater Horizon and Fukushima".
With all this talk of failure there is surely one takeaway.  Complex systems will fail and it is in the anticipation of that failure that we gain the most.  Let's stop howling and look at how to handle these situations intelligently.

How Do You Rebuild A Website?

In the cloud you will likely need two things.  (a) scripts to rebuild all the components in your architecture, spinup servers, fetch source code, fetch software and configuration files, configure load balancers and mount your database and more importantly (b) a database backup from which you can rebuild your current dataset.

Want to stick with EC2, build out your infrastructure in an alternate availability zone or region and you're back up and running in hours.  Or better yet have an alternate cloud provider on hand to handle these rare outages.  The choice is yours.

Mitigate risk?  Yes indeed failure is more common in the cloud, but recovery is also easier.  Failure should pressure the adoption of best practices and force discipline in deployments, not make you more of a gunslinger!

Want to see an extreme example of how this can play in your favor?  Read Jeff Atwood's discussion of so-called Chaos Monkey, a component whose sole job it is to randomly kill off servers in the Netflix environment at random.  Now that type of gunslinging will surely keep everyone on their toes!  Here's a Wired article that discusses Chaos Monkey.

George Reese of enStratus discusses the recent failure at length.  The I would argue calling Amazon's outage the Cloud's Shing Moment, all of his points are wisened and this is the direction we should all be moving.

Going The Way of Commodity Hardware

Though it is still not obvious to everyone, I'll spell it out loud and clear.  Like it or not, the cloud is coming.  Look at these numbers.

Furthermore the recent outage also highlights how much and how many internet sites rely on cloud computing, and Amazon EC2.
Way back in 2001 I authored a book on O'Reilly called "Oracle and Open Source".  In it I discussed the technologies I was seeing in the real world.  Oracle on the backend and Linux, Apache, and PHP, Perl or some other language on the frontend.  These were the technologies that startups were using.  They were fast, cheap and with the right smarts reliable too.

Around that time Oracle started smelling the coffee and ported it's enterprise database to Linux.  The equation for them was simple.  Customers that were previously paying tons of money to their good friend and confidant Sun for hardware, could now spend 1/10th as much on hardware and shift a lot of that left over cash to - you guessed it Oracle!  The hardware wasn't as good, but who cares because you can get a lot more of it.

Despite a long entrenched and trusted brand like Sun being better and more reliable, guess what?  Folks still switched to commodity hardware.  Now this is so obvious, no one questions it.  But the same trend is happening with cloud computing.

Performance is variable, disk I/O can be iffy, and what's more the recent outage illustrates front and center, the servers and network can crash at any moment.  Who in their right mind would want to move to this platform?

If that's the question you're stuck on, you're still stuck on the old model.  You have not truely comprehended the power to build infrastructure with code, to provision through automation, and really embrace managing those components as software.  As the internet itself has the ability to route around political strife, and network outages, so too does cloud computing bring that power to mom & pop web shops.

Conclusions

• Have existing investments in hardware?  Slow and cautious adoption makes most sense for you.

• Have seasonal traffic variations?  An application like this is uniquely suited to the cloud.  In fact some of the gaming applications which can autoscale to 10x or 100x servers under load, are newly solveable with the advent of cloud computing.

• Are you currently paying a lot for disaster recovery systems that primarily lay idle.  Script your infrastructure for rebuilding from bare metal, and save that part of the budget for more useful projects.

Cloud Computing may not make sense for all application types.  But as with the adoption of commodity hardware and Linux over a decade ago, economic considerations will continue to pressure adoption.

** Original article -- Intro to EC2 Cloud Deployments **

What types of applications do fit well in the cloud?

1. Applications with Seasonal Traffic Patterns

1. Proof-of-concept Applications

1. Quick Temporary Dev & Test Environments

1. CPU Intensive Applications

1. On-Demand or Unknown Future Demand

Seasonal Traffic Patterns

Your recent social media campaign has gone viral.  It's what you've been dreaming about, pinning your hopes on, and all of your hard work is now coming to fruition.  Tens of thousands of internet users, hoards of them in fact, are now descending on your website.  Only one problem, it went down!!

That's a situation you want to avoid.  Luckily there are some best practices for avoiding scenarios like the one I described.  In engineering it's termed "degrade gracefully".  That is continue functioning but with the heaviest features disabled.

Browsing Only, But Still Functioning

One way to do this is for your site to have a browsing only mode.  On the database side you can still be functioning with a read-only database.  With a switch like that, your site will continue to function while pointed to any of your read-only replication slaves.  What's more you can load balance across those easily, and keep your site up and running.

Decoupling

In software development, decoupling involves breaking apart components or pieces of an application that should not depend on one another.  One way to do this is to use a queuing system such as Amazon's SQS to allow pieces of the application to queue up work to be done.  This makes those pieces asynchronous, ie they'll return right away.  Another way is to expose services internal to your site through web services.  These individual components can then be scaled out as needed.  This makes them more highly available, and reduces the need to scale your memcache, webservers or database servers - the hardest ones to scale.

Identify Features You Can Disable

Typically your application will have features that are more superfluous, or that are not part of the core functionality.  Perhaps you have star ratings, or some other components that are heavy.  Work with the development and operations teams to identify those areas of the application that are heaviest, and that would warrant disabling if the site hits heavy storms.

Once you've done all that, document how to disable and reenable those features, so other team members will be able to flip the switches if necessary.

BOOK REVIEW:  Outsmart! - by Jim Champy

This is a great business book out on FT Press.  It's chapters are organized around eight case studies of companies that grew quickly, earning huge returns for their shareholders and customers as well.

What I like most about this book is the "Get Smart" section at the end of each chapter.  There he lists the lessons learned from that case study in a succinct summary.  Following that is a "Questions to Ask Yourself" section which really points the spotlight back on you, to help you apply those lessons.

He didn't choose companies that all took the same path or solved the same problems either.  Some were new companies filling a niche noone saw like SonicBids while others were long established brands that had languished such as Smith & Wesson.

An excellent read and highly recommended book.

With traditional managed hosting solutions, we have best practices, we have business continuity plans, we have disaster recovery, we document our processes and all the moving parts in our infrastructure.  At least we pay lip service to these goals, though from time to time we admit to getting side tracked with bigger fish to fry, high priorities and the emergency of the day.  We add "firedrill" to our todo list, promising we'll test restoring our backups.  But many times we find it is in the event of an emergency that we are forced to find out if we actually have all the pieces backed up and can reassemble them properly.

** Original article -- Intro to EC2 Cloud Deployments **

Cloud Computing is different.  These goals are no longer be lofty ideals, but must be put into practice.  Here's why.

1. Virtual servers are not as reliable as physical servers
2. Amazon EC2 has a lower SLA than many managed hosting providers
3. Devops introduces new paradigm, infrastructure scripts can be version controlled
4. EC2 environment really demands scripting and repeatability
5. New flexibility and peace of mind

Unreliable Servers

EC2 virtual servers can and will die.  Your spinup scripts and infrastructure should consider this possibility not as some far off anomalous event, but a day-to-day concern.  With proper scripts and testing of various scenarios, this should become manageable.  Use snapshots to backup EBS root volumes, and build spinup scripts with AMIs that have all the components your application requires.  Then test, test and test again.

Amazon EC2's SLA - Only 99.95%

The computing industry throws around the 99.999% or five-nines uptime SLA standard around a lot.  That amounts to less than six minutes of downtime.  Amazon's 99.95% allows for 263 minutes of downtime.  Greater downtime merely gets you a credit on your account.  With that in mind, repeatable processes and scripts to bring your infrastructure back up in different availability zones or even different datacenters is a necessity.  Along with your infrastructure scripts, offsite backups also become a wise choice.  You should further take advantage of availability zones and regions to make your infrastructure more robust.  By using private IP addresses and network, you can host a MySQL database slave in a separate zone, for instance.  You can also do GDLB or Geographically Distributed Load Balancing to send customers on the west coast to that zone, and those on the east coast to one closer to them.  In the event that one region or availability zone goes out, your application is still responding, though perhaps with slightly degraded performance.

Devops - Infrastructure as Code

With traditional hosting, you either physically manage all of the components in your infrastructure, or have someone do it for you.  Either way a phone call is required to get things done.  With EC2, every piece of your infrastructure can be managed from code, so your infrastructure itself can be managed as software.  Whether you're using waterfall method, or agile as your software development lifecycle, you have the new flexibility to place all of these scripts and configuration files in version control.  This raises manageability of your environment tremendously.  It also provides a type of ongoing documentation of all of the moving parts.  In a word, it forces you to deliver on all of those best practices you've been preaching over the years.

EC2 Environment Considerations

When servers get restarted they get new IP addresses - both private and public.  This may affect configuration files from webservers to mail servers, and database replication too, for example.  Your new server may mount an external EBS volume which contains your database.  If that's the case your start scripts should check for that, and not start MySQL until it finds that volume.  To further complicate things, you may choose to use software raid over a handful of EBS volumes to get better performance.

The more special cases you have, the more you quickly realize how important it is to manage these things in software.  The more the process needs to be repeated, the more the scripts will save you time.

New Flexibility in the Cloud

Ultimately if you take into consideration less reliable virtual servers, and mitigate that with zones and regions, and automated scripts, you can then enjoy all the new benefits of the cloud.

• autoscaling
• easy test & dev environment setup
• robust load & scalability testing
• vertically scaling servers in place - in minutes!
• pause a server - incurring only storage costs for days or months as you like
• cheaper costs for applications with seasonal traffic patterns
• no huge up-front costs

The term clustering is often used loosely in the context of enterprise databases.  In relation to MySQL in the cloud you can configure:

1. Master-master active/passive
2. Sharded MySQL Database
3. NDB Cluster

Master-Master active/passive replication

Also sometimes known as circular replication.  This is used for high availability. You can perform operations on the inactive node (backups, alter tables or slow operations) then switch roles so inactive becomes active.  You would then perform the same operations on the former master.  Applications sees "zero downtime" because they are always pointing at the active master database.  In addition the inactive master can be used as a read-only slave to run SELECT queries and large reporting queries.  This is quite powerful as typical web applications tend to have 80% or more of their work performed with read-only queries such as browsing, viewing, and verifying data and information.

Sharded MySQL Database

This is similar to what in the Oracle world is called "application partitioning".   In fact before Oracle 10 most Parallel server and RAC installations required you to do this.  For example a user table might be sharded by putting names A-F on node A, G-L on node B and so forth.

You can also achieve this somewhat transparently with user_ids.  MySQL has an autoincrement column type to handle serving up unique ids.  It also has a cluster-friendly feature called auto_increment_increment.  So in an example where you had *TWO* nodes, all EVEN numbered IDs would be generated on node A and all ODD numbered IDs would be generated on node B.  They would also be replicating changes to eachother, yet avoid collisions.

Obviously all this has to be done with care, as the database is not otherwise preventing you from doing things that would break replication and your data integrity.

One further caution with sharding your database is that although it increases write throughput by horizontally scaling the master, it ultimately reduces availability.   An outage of any server in the cluster means at least a partial outage of the cluster itself.

NDB Cluster

This is actually a storage engine, and can be used in conjunction with InnoDB and MyISAM tables.  Normally you would use it sparingly for a few special tables, providing availability and read/write access to multiple masters.  This is decidedly *NOT* like Oracle RAC though many mistake it for that technology.

MySQL Clustering In The Cloud

The most common MySQL cluster configuration we see in the Amazon EC2 environment is by far the Master-Master configuration described above.  By itself it provides higher availability of the master node, and a single read-only node for which you can horizontally scale your application queries.  What's more you can add additional read-only slaves to this setup allowing you to scale out tremendously.

Migrating from MySQL to Oracle can be as complex as picking up your life and moving from the country to the city.  Things in the MySQL world are often just done differently than they are in the Oracle world.  Our guide will give you a birds eye view of the differences to help you determine what is the right path for you.

** See also: Oracle to MySQL Migration Considerations **

MySQL comes from a more open-source or DIY background.  One of Unix and Linux administrators and even developers carrying the responsibility of a DBA.

1. Installation & Administration Considerations
2. Query and Optimizer Differences
3. Security Strengths and Weaknesses
4. Replication & High Availability
5. Table Types & Storage Engines
6. Applications, Connection Pooling, Stored Procedures and More
7. Backups & Disaster Recovery
8. Community - MySQL & Oracle Differences
9. TCO, Licensing, and Cloud Considerations
10. Advanced Oracle Features - Missing in MySQL

Check back soon as we update each of these sections.

What Do Consultants Do?

Consultants bring a whole host of tools to experiences to bear on solving your business problems.  They can fill a need quickly, look in the right places, reframe the problem, communicate and get teams working together, and bring to light problems on the horizon. And they tell stories of challenges they faced at other businesses, and how they solved them.

Frame or Reframe The Problem

Oftentimes businesses see the symptoms of a larger problem, but not the cause.  Perhaps their website is sluggish at key times, causing them to lose customers.  Or perhaps it is locking up inexplicably.  Framing the problem may involve identifying the bottleneck and pointing to a particular misconfigured option in the database or webserver.  Or it may mean looking at the technical problem you've chosen to solve and asking if it meets or exceeds what the business needs.

Tell Business Stories

Clients often have a collection of technologies and components in place to meet their business needs.  But day-to-day running of a business is ultimately about bringing a product or service to your customer.  Telling stories of challenges and solutions of past customers, helps illustrate, educate, and communicate problems you're facing today.

Fill A Need Quickly

If you have an urgent problem, and your current staff is over extended, bringing in a consultant to solve a specific problem can be a net gain for everyone.  They get up to speed quickly, bring fresh perspectives, and review your current processes and operations.  What's more they can be used in a surgical way, to augment your team for a short stint.

Get Teams Communicating

I've worked at quite a number of firms over the years and tasked with solving a specific technical problem only to find the problem was a people problem to begin with.  In some cases the firm already has the knowledge and expertise to solve a problem, but some members are blocking.  This can be because some folks feel threatened by a new solution which will take away responsibilities they formerly held.  Or it can be because they feel some solution will create new problems which they will then be responsible to cleanup.  In either case bridging the gap between business needs and operations teams to solve those needs can mean communicating to each team in ways that make sense to them.  A technical detail oriented focus makes most sense when working with the engineering teams, business and bottom-line focused when communicating with the management team.

Highlight Or Bring To Light Problems On Horizon

Is our infrastructure a ticking timebomb?  Perhaps our backups haven't been tested and are missing some crucial component?  Or we've missed some security consideration, left some password unset, left the proverbial gate open to the castle.  When you deal with your operations on a day-to-day basis, little details can be easy to miss.  A fresh perspective can bring needed insight.

BOOK REVIEW - Jaron Lanier - You Are Not a Gadget

Lanier is a programmer, musician, the father of VR way back in the 90's, and wide-ranging thinker on topics in computing and the internet.

His new book is a great, if at times meandering read on technology, programming, schizophrenia, inflexible design decisions, marxism, finance transformed by cloud, obscurity & security, logical positivism, strange loops and more.

He opposes the thinking-du-jour among computer scientists, leaning in a more humanist direction summed up here:  "I believe humans are the result of billions of years of implicit, evolutionary study in the school of hard knocks."    The book is worth a look.

Call Us Daily 11-11 EST:  +1-212-533-6828

Planning and implementing a bullet proof disaster recovery strategy forges a a large piece in your business continuity plans.  We can:

• Review your entire web operations & cloud environment
• Perform a fire drills to test  backups, scripts, and processes
• Examine security of operations
• Provide feedback of current environment
• Work closely with your team

Emergency Services

• Performance related outages
• MySQL database crash
• Upgrade related problems
• Server & Hardware outages

Best practices for backups and disaster recovery aren't tremendously different in the cloud than from a managed hosting environment.  But they are more crucial since cloud servers are less reliable than physical servers.  Also the security aspect may play a heightened role in the cloud.  Here are some points to keep in mind.

** Original article -- Intro to EC2 Cloud Deployments **

1. Perform multiple types of backups
2. Keep non-proprietary backups offsite
3. Test your backups - perform firedrills
4. Encrypt backups in S3
5. Perform Replication Integrity Checks

Perform Multiple Types of Backups

Your database tier is typically your primary datastore, so it's backups are often the most crucial.  Snapshots of EBS volumes are powerful and fast ways to perform full database backups in the AWS environment.  This involves locking all tables briefly, and running the snapshot command, followed by a release of all those table locks.  Be sure to test this process to ensure that the temporary locks on the database don't create a pileup on your webservers.

You can find xtrabackup at Percona's site

Keep Non-proprietary Backups Offsite

The EC2 snapshots are great, but they only work in EC2.  So you'll also want to perform other types of backups.  Personally I like having a few different options in the event I need to restore.  Logical backups are great for restoring one table, but are slow for restoring the entire database.  Hotbackups are great and fast to restore the whole database, but take a lot of space so may not be as efficient if you just need to restore one table.  So I like to have both.  Percona's xtrabackup and the associated innobackupex script provide an open-source hotbackup solution for MySQL.  Get it!  Then intersperse those backups with mysqldumps as well.  Alternating days, for example.

Test Your Backups - Perform Firedrills

Any good disaster recovery plan must be thoroughly tested.  Set aside the time to actually run through this from start to finish.  This is where the cloud really excels to your advantage.  Spinup all the servers that makeup your entire environment, load balancer, webservers, database servers, checkout all the source code, and configuration files.  You put your configuration files in version control, right?  Then restore the database.  This firedrill tests your server spinup scripts, your version control of source code and configuration files, and your database backups.  All of these pieces must be in place for the fire drill to succeed.  Lastly running through the whole process forces you to document details, and you find out how long your disaster recovery would actually take.

Encrypt Backups in S3

S3 stores objects as private by default, however it makes sense for particularly sensitive data to also encrypt those backups.  Remember you control access to your encryption keys but not where the data is stored or where it might move around.  So it can't hurt to be extra cautious.  Here's an excellent article on the topic.  Using mk-query-digest to checksum

Perform Replication Integrity Checks

A MySQL slave or passive master database can be a great way to offload backups away from the primary database server.  This reduces impact to your customers while backups are running.  But MySQL replication is not bulletproof.  The slaves can drift out of sync with the master silently without throwing errors.  That's why it's important to use an integrity checking tool like Maatkit's mk-table-checksum.  This tool can be set in cron to perform checksums on a slice of your database periodically.

Here's an excellent article on using the tool.  Ongoing MySQL Integrity Checking with mk-table-checksum