Business Insider posted this spectacular interactive google map of our amazing Startup Ecosystem here in New York City.  Wow!

Check out the full list of NYC Startups in the original article.

View NYC Startups in a larger map

Effective MySQL: Optimizing SQL Statements

by Ronald Bradford

No Nonsense, Readable, Practical, and Compact

I like that this book is small; 150 pages means you can carry it easily.  It's also very no nonsense.  It does not dig too deeply into theory unless it directly relates to your day-to-day needs.  And those needs probably cluster heavily around optimizing SQL queries, as those pesky developers are always breaking things

Jokes aside, this new book out on Oracle Press is a very readable volume. Bradford has drawn directly from real-world experience to give you the right bite size morsels you need in your day-to-day MySQL activities.

Highlights

Chapter one, The Five Minute DBA gives you the basic methodology if you don't already know it.  Enable the slow query log, analyze it, and use the explain facility.  Then index as appropriate, or eliminate queries if you can.

Chapter two digs a little deeper into the basics, introducing explain extended, table statistics and storage engines.  You'll also learn how to use show session & global status, as well as session & global variables.  You'll also have your first look at MySQL's data dictionary - INFORMATION_SCHEMA.

Chapter three is where it starts to get meaty.  You probably know that MySQL has b-tree indexes, but did you know that it has b+tree indexes, or hash indexes?

Chapter four digs into indexes further with single & multi-column indexes using them for sorting and joining.  You'll also find out about covering indexes which are multi-column matching the where clause, but also including columns needed in the SELECT predicate.  Do you have duplicate or unused indexes?  You'll learn why they matter to performance and how to eliminate them with tools like mk-duplicate-key-checker.

Chapter five continues along the same lines, with more coverage of indexes.  Learn to identify when you are using a covering index, fulfilling the entire query by only accessing the index.  You'll also learn about partial indexes, how they can reduce the size of index storage and retrieval while still getting your data for you.

Chapter six covers configuring the server itself, hitting on the system variables such as the innodb buffer pool (innodb_buffer_pool_size) and key buffer (key_buffer_size) as well as the query cache.  You'll also learn how to set the four main session memory settings - sort buffer (sort_buffer_size) and join buffer ( join_buffer_size) as well as the lesser known read buffers (read_buffer_size and read_rnd_buffer_size).

Chapter seven is all about the process of tuning and optimizing MySQL.  Rolling all the previous sections into marching orders, and prescriptive advice, he takes you through step by step how to apply the principles.  You'll get an introduction to mk-query-digest (though strangely without attribution to Baron Schwartz), the great maatkit tool for query analysis and aggregation, as well as the microsecond precision patch, which allows your mysql shell client to display more exact timing data.  For the patch he links back to an article on his own site which seems to be not found.  The author of the high precision mysql timer patch is Stewart Smith.

I personally got the most out of Chapter eight, full of self-described hidden performance tips.  From identifying unused or duplicate indexes, to replacing inefficient data types with better ones, why it's important to use NOT NULL where possible or how to store IP addresses efficiently, this chapter has a lot of goodies.  For those still struggling with SQL statement tuning, there are a few patterns that are described, offering advice on how to rewrite a subquery as an inner join,

What you might not know

• MySQL includes Oracle's index organized tables by a different name
• Too many indexes can dramatically impact INSERT & UPDATE performance
• Many DDL operations can be done online - see oak-online-alter-table (Shlomi Noach)
• Datatypes matter - use enum, int unsigned, timestamp & not null where possible
• Covering indexes are your friend, duplicate & unused indexes are not!
• A replication slave can have different storage engines or indexes from the master. These can support different uses - such as data warehousing or non-transactional requirements.
• While a_string LIKE '%end of my sentence.' won't use an index, you can index reverse_string, then use reverse_string LIKE REVERSE '%end of my sentence.' and MySQL will use this index.  You've simulated an advanced Oracle feature, reverse key indexes!

A few small gripes

If I were to add a few complaints it would be to say that some of the examples were rather simplistic.  In many cases tuning SQL is not as simple as just adding the right index.  For instance there was no good discussion of the dreaded "using temporary, using filesort" that we see a lot in MySQL explains when sorting has to be done, but will not fit in memory.  Or what about tmpdir=/dev/shm, how will that improve things?  What about UNION versus UNION ALL where appropriate.  Why does DISTINCT do a sort?

The book was also missing a discussion of triggers, stored procedures, when or if the query cache can cause problems and so forth.  Also the article link mentioned about chapter seven isn't the only missing link.  I followed links to optimizing sql  statements and it seems to go to a generic holding page.  Also the main link effectivemysql.com/book leads to an outline of an as yet unreleased title on Backup and Recovery.

All in all, well worth your money

However, other than these few gripes the book overall is a very welcome addition to the small family of MySQL books.  Get a copy quick before they're all gone!

Often when we think about speeding up and scaling, we focus on the application layer itself.  We look at the webserver tier, and database tier, and optimize the most resource intensive pages.

There's much more we can do to speed things up, if we only turn over the right stones.  Whether you're using WordPress or not, many of these principals can be applied.  However we'll use WordPress as our test case.

Test Your Website speed

There are web-based speed testing tools that will help with this step.  Take a look at Webpagetest , Pingdom tools and Google's Chrome plugin Pagetest which integrates right into your browser.  If you're using firefox, take a look at YSlow .

If you've already got the WordPress plugin W3 Total Cache installed, that also integrates with Google Speed Test through an API key.  That integrates right into your WordPress dashboard.

1. Reduce objects

The first thing you can do to improve page performance and response is to make the page simpler.  Fewer images, fewer posts, fewer plugins, fewer widgets and so forth all contribute to a speedier page.  Obviously you don't want to forfeit functionality, but if you are loading 20 posts, you might want to reduce them to five or ten.   If you are calling out to third party APIs to load badges, twitter counts or SKYPE buttons, consider each and its affect on webpage load times.

If you have access to the underlying HTML, your application should try to reduce the number of DOM objects that are created.  These are created as the HTML is parsed.  Simpler structure here means faster page load times.

2. Compress objects

Your page is full of objects.  Many of them such as images, can be compressed.  This saves server storage space, and is a smaller object to copy across the network to the end users browser.  The web page test will show you a list of objects that are good candidates for compression.

3. Employ page & object caching

An object cache will store name/value pairs in memory for your application.  Essentially the webserver tier will cache data that it reads from the database to avoid additional round trips and network overhead.   For most of us, we'll want to get memcache installed in our webserver tier.

For a page cache, take a look at Varnish.  This can also be installed by package manager.  A page cache is like a tiny hyper efficient webserver.  It can sit on the webserver itself or on it's own server if you have many webservers in your environment.

Lastly if you have not already, grab the W3 Total Cache plugin for WordPress.  This plugin integrates directly with these two types of caches.  Simply click the Performance tab, select General Settings, and scroll to the object cache and varnish sections.

4. Browser caching

Browser caching is a tricky one.  You might think this is totally in the hands of the end user.  As it turns out, however the objects your webserver sends back to clients specifies various caching information in the HTTP headers.

Since we're encouraging you to test things yourself, fire up your command line terminal, and use "curl" to take a look at the headers of a file on your webserver.  Here's an example:

$ curl -I http://www.iheavy.com/files/pros.gif

HTTP/1.1 200 OK

Date: Tue, 01 Nov 2011 04:58:55 GMT

Server: Apache/2.2.3 (CentOS)

Last-Modified: Wed, 24 Aug 2011 01:06:09 GMT

ETag: "1649b1fb-5562-4ab35eadfe240"

Accept-Ranges: bytes

Content-Length: 21858

Content-Type: image/gif

Now we'll go ahead and enable W3 Total Cache in wordpress, then rerun the same curl command again:

$ curl -I http://www.iheavy.com/files/pros.gif

HTTP/1.1 200 OK

Date: Tue, 01 Nov 2011 05:01:27 GMT

Server: Apache/2.2.3 (CentOS)

Last-Modified: Wed, 24 Aug 2011 01:06:09 GMT

ETag: "5562-4ab35eadfe240"

Accept-Ranges: bytes

Content-Length: 21858

Cache-Control: max-age=31536000, public, must-revalidate, proxy-revalidate

Expires: Wed, 31 Oct 2012 05:01:27 GMT

Vary: User-Agent

Pragma: public

X-Powered-By: W3 Total Cache/0.9.2.3

Content-Type: image/gif

The main line we're interested here is the cache-control line.  Notice that the object has an expiration of 31536000.  Turns out that is the number of seconds in one week.  This tells the browser to keep these objects in its own cache and not refetch from the webserver each time.  That's a tremendous speedup.

5. Employ a CDN

If you haven't heard of CDN before, it stands for content delivery network.  Dropbox, the filesharing service is essentially a CDN service with a handsome user interface built on top of it.  Akamai is another famous CDN solution, one of the most mature, and respected available.  If you're on Amazon you might look at CloudFront, their CDN solution.

What is a CDN exactly?  Consider your pressing need for some basic groceries.  Maybe you need juice, a sandwich, beer or cigarettes.   If you're walking around the city, you'll find a deli on every corner.  That's the quickest way to get those items, a short walk and a short line.  When you need bigger items, you might make a weekly trek to Whole Foods.  From the perspective of food and beverage manufacturers you certainly want to get your products into the delis.  They are your grocery distribution networks, if you will.

W3 Total Cache supports Amazon CloudFront, simply enter your private and secret keys, and save.  Then upload all of your content up to S3 and you'll be benefiting from a CDN in no time.

Test again

Now that you've had a chance to look at all these caching opportunities, and hopefully implemented many of them in WordPress, rerun your Google Speed Test.

Google Page Test - Score of 91

We were able to raise our score from the mid-70's to 91 after tweaking various pieces of the site.  We haven't even implemented image sprites yet.

Hopefully this time around you'll see some objects loaded via CDN, more objects compressed, and proper expirations on most if not all content on your site.

CDN & caching are closer than you think

Most internet sites can make great gains in faster response time by looking in the right places.  Application and server rearchitecting give you a lot, but usually involve more complex and ongoing effort by developers.  However the basic caching improvements we described above will make a big difference as well.  CDN and caching technology is a lot more within reach than you think it is.

 Oracle starts charging for MySQL Add-ons

Exciting news, Oracle just announced commercial MySQL extensions that they'll be offering paid extensions to the core MySQL free product.

To be sure, this has raised waves of concern among the community, but on the whole I suspect it will be a good thing for MySQL.  This brings more commercial addons to the table, which only increases the options for customers.  Many will continue to use the core database product only, and avoid license hassles while others will surely embark on a hybrid approach if it solves their everyday business problems.

Speaking of licensing, back in May, Amazon announced that it's RDS or Relational Database Service would now offer Oracle as an option.  For those familiar with negotiating Oracle licensing, and the concomitant headache, this is certainly a welcome addition.  It also marks a dramatic break from the previous per-cpu or per-seat licensing models, and Oracle's embrace of the cloud model of instant provisioning.  Interesting all around!

For those concerned about MySQL as a whole going commercial, there are three large companies continuing to support and distribute the core database including Percona, Monty Program AB (MariaDB) and Brian Aker's fork Drizzle.  These distros have strong community support, and that strength will only grow.

3 ways your MySQL migration project can shake you up

Once a development or operations team gets over the hurdle of open-source, and start to feel comfortable with the way software works outside of the enterprise world, they will likely start to settle in and feel comfortable.  Best not to get too cushy though for there are more surprises hiding around the corner.  Here are a few of the biggest ones.

1. Replication Is Not Perfect

Yes, you've installed MySQL, setup a slave, and got it replicating properly.  You check the slave and it's 0 seconds behind the master.  What's more you monitor the error log file, and have a check in place to alert you if something happens there.  Job completed, good job!

Not so fast.  Unfortunately this is not the end of the story.  Many MySQL replication slaves are not consistent with their masters, but they drift apart silently.  Really?  Yes really.  But there are tools to help you.  Maatkit includes a checksum tool that will build checksums for all of your tables, and populate a table with that data.  Your checksum data then flows through replication to the slave.  From there you can perform a check on the slave to see if there are any differences.  Such a check is easy to wire into nagios or other monitoring software, to alert you as soon as a difference materializes in your slave database.

With vigilant data integrity checking in place, your MySQL slave is robust enough to perform backups off of.  But beware if you're not using tools to check and relying only on the error log.  This is unfortunately not sufficient.

2. Queries With Bad Performance

MySQL's optimizer is still fairly rudimentary.   There are some classes of queries that MySQL does not handle well.  If I were to provide a general rule it would be to stick to fairly simple queries, with at most a few joins.  Simpler lookups are sure to be fast.  Also less complex and convoluted queries are easier to understand, optimize and tune later on if they require it.

Subqueries

Although MySQL supports these, they often optimize badly, and perform terribly.  Review an explain plan to be sure you're optimal.  Replace such a query with a join where possible.

Joins

MySQL only supports the nested loops algorithm to do joins.  Merge join and hash join, two alternate and very efficient algorithms have not yet been implemented in MySQL unfortunately.  Be aware, run explain and profile your slow queries.

Using Temporary; Using Filesort

When MySQL has to do sorting, it does so in one of two ways.  If the dataset is small it'll use a quicksort, and do it in memory.  In this case you'll see just "using filesort".  However if  you see "Using Temporary; Using Filesort" in an EXPLAIN plan output, it means MySQL is using a two-pass algorithm to do a sort, creating a temporary table because it won't all fit in memory.   Sometimes a composite or multi-column index can help with these situations.  You may also affect this behavior by setting sort_buffer_size and join_buffer_size at the session level, then rerunning the explain.

Newer versions of MySQL such as 5.6 have gotten much smarter about this, as the optimizer is always being tweaked and improved.  Check your query plans in dev & test before migrating your production database.

Single Index Usage

Although MySQL supports very efficient b-tree indexes, it cannot always use multiple indexes, or merge indexes together for more efficient path to data.  Although there are some exceptions, figure at each stage of a query and on each table MySQL will likely choose only one index.  When you're not satisfied with this, you may choose to build a covering or multi-column index.  Keep in mind these will only serve that one particular query, or a query against the leading edge column.  Also keep in mind that the more indexes you have built on a table, the less performant inserts, updates and deletes will be against this table.  All of those operations will require index maintenance so you must balance read performance (indexes on more columns) with write performance (requiring index maintenance and updating).

3. Where Are The Transactional Tables?

Yes MySQL does offer row-level locking, ACID compliance, and repeatable read and other isolation levels and referential integrity.  In fact it's implementation of the buffer cache to support reading and writing blocks from disk is modeled on Oracle's own algorithm.  So why are you getting all of that?

As it turns out MySQL supports various Storage Engines, unlike other databases which have all of this rolled into the kernel, and only support transactional tables.  This can be thought of as a feature although in reality it is more of a anomaly of how MySQL has evolved, and where it's roots lie.  As such the default storage engine is MySQL which is not transactional.  All changes are effectively auto-commit, and it only supports table level locking.

If you haven't checked already, verify what table types you have as follows:

mysql> select table_schema, engine, count(*) from information_schema.tables group by table_schema, engine;

Outside of system tables, your application tables should all be InnoDB.  You can modify an existing table to be transactional as follows:

mysql> alter table mytable engine=innodb;

You should also set storage_engine variable to default to Innodb.  Finally you may also choose to set the sql_mode NO_ENGINE_SUBSTITION which will force MySQL to flag an error whenever the storage engine you choose is not available.  This prevents it from creating a table with a storage engine other than your choice.

Scalability is about application, architecture and infrastructure design, and careful management of server components.

1. Object Relational Mappers

ORMs are popular among developers but not among performance experts.  Why is that?  Primarily these two engineers experience a web application from entirely different perspectives.  One is building functionality, delivering features, and results are measured on fitting business requirements.  Performance and scalability are often low priorities at this stage.  ORMs allow developers to be much more productive, abstracting away the SQL difficulties of interacting with the backend datastore, and allowing them to concentrate on building the features and functionality.

On the performance side the picture is a bit different.  By leaving SQL query writing to an ORM, you are faced with complex queries that the database cannot optimize well.  What's more ORMs don't allow easy tweaking of queries, slowing down the tuning process further.

2. Synchronous, Serial, Coupled or Locking Processes

Locking in a web application operates something like traffic lights in the real world.  Replacing a traffic light with a traffic circle often speeds up traffic dramatically.  That's because when you're out somewhere in the country where there's very little traffic, no one is waiting idly at a traffic light for no reason.  What's more even when there's a lot of traffic, a traffic circle keeps things flowing.  If you need locking, better to use InnoDB tables as they offer granular row level locking than table level locking like MyISAM tables.

Avoid things like semi-synchronous replication that will wait for a message from another node before allowing the code to continue.  Such waits can add up in a highly transactional web application with many thousands of concurrent sessions.

Avoid any type of two-phase commit mechanism that we see in clustered databases quite often.  Multi-phase commit provides a serialization point so that multiple nodes can agree on what data looks like, but they are toxic to scalability.  Better to use technologies that employ an eventually consistent algorithm.

3. One Copy of Your Database

Without replication, you rely on only one copy of your database.  In this configuration, you limit all of your webservers to using a single backend datastore, which becomes a funnel or bottleneck.  It's like a highway that is under construction, forcing all the cars to squeeze into one lane.  It's sure to slow things down.  Better to build parallel roads to start with, and allow the application aka the drivers to choose alternate routes as their schedule and itinerary dictate.

4. Having No Metrics

Having no metrics in place is toxic to scalability because you can visualize what is happening on your systems.  Without this visual cue, it is hard to get business units, developers and operations teams all on the same bandwagon about scalability issues.  If teams are having trouble groking this, realize that these tools simple provide analytics for infrastructure.

There are tons of solutions too, that use SNMP and are non-invasive.  Consider Cacti, Munin, OpenNMS, Ganglia and Zabbix to name a few.  Metrics collections can involve business metrics like user registrations, accounts or widgets sold.  And of course they should also include low level system cpu, memory, disk & network usage as well as database level activity like buffer pool, transaction log, locking sorting, temp table and queries per second activity.

5. Lack of Feature Flags

Applications built without feature flags make it much more difficult to degrade gracefully.  If your site gets bombarded by a spike in web traffic and you aren't magically able to scale and expand capacity, having inbuilt feature flags gives the operations team a way to dial down the load on the servers without the site going down.   This can buy you time while you scale your webservers and/or database tier or even retrofit your application to allow multiple read and write databases.

Without these switches in place, you limit scalability and availability.

Want more?  Signup for our scalability newsletter.

Deploying new code that includes changes to your database schema doesn't have to be a process fraught with stress and burned fingers. Follow these five tips and enjoy a good nights sleep.

1. Deploy with Roll Forward & Rollback Scripts

When developers check-in code that requires schema changes, that release should also require two scripts to perform database changes. One script will apply those changes, alter tables to add columns, change data types, seed data, clean data, create new tables, views, stored procedures, functions, triggers and so forth. A release should also include a rollback script, which would return tables to their previous state.

This idea of database change management is popular as Migrations in Ruby on Rails. It is a convenient way to enforce good coding, and better database change management. However, keep in mind it has it's limits. See Perform Backup section below.

2. Ban SELECT * FROM table

Whenever you select from a table, be sure to specify the column names in all of your SQL code.  That way if later features or code changes add columns to that table, there is no potential to break old code that did SELECT * and got the old table column structure.  If the table was formerly returning four columns and after your code and schema changes returns five columns, it might break code that uses SELECT *.

The same goes for update statements.  Be sure to specify columns explicitely, and have a policy to enforce this on developers.

3. Perform Backup on Affected Data

Database rollback scripts have their limits, so a full database backup on any affected tables, or to be thorough the entire database, is a fully insured way to deploy database changes.

Since the developer's script cannot account for data in a live database, a rollback script will not work on database changes that do any of the following.  Only a backup will suffice:

• drop a table
• drop a column
• delete any data

4. Build in Feature Flags

Feature flags provide the operations team with a mechanism to turn on or off a new feature.  If you're going the extra mile to build in these switches you will make rolling out new features a lot more trouble free.  Even better, include a mechanism to enable or disable a feature for a percentage or class of users.  That allows you to dial the feature on gradually or expose it only to a certain group of users.   In an emergency this will also come in handy as a way to quickly disable a new feature that has gone awry.

5. Involve DBAs in the Review & Test Process

Before deploying a new release, include a process whereby the DBA must review schema changes and scripts.  This provides an additional sanity check, and a Database Administrators perspective on those changes.  He or she may well provide insight or caution on potential gotchas that you might encounter.

1. Object Relational Mappers

Software development has always made use of libraries, off-the-shelf components that are shared between different projects.  These allow you to stand on the shoulders of others and build bigger things.  Frameworks do the same thing, they provide a context from which to build on.  Ruby on Rails for example provides a great starting framework from which to build web applications, managing sessions in an elegant way.

Unfortunately the same process when applied to interacting with and managing database access does not work well.  In these cases you build a framework for example that might treat tables as objects.  As you interact with and build code to use those objects, the ORM dynamically constructs the SQL to handle getting stuff into and out of the database.  Secretly no one likes writing SQL, and any layer of software that will do it for me has go to be a good thing!

Unfortunately for scalability it is decidedly *not* a good thing.  From Hibernate to ActiveRecord, these type of abstraction layers build SQL that is innefficient, and worse tough to get at when you want to optimize it.  That's what frameworks do well, hide that complexity.  Beware though in regards to future scalability.

Instead isolate your queries in one place, and use a read database handle, and a write handle.

2. Serialization & Synchronous Processes

Synchronous processes usually block until a change makes it way to more than one place.  This is dangerous as your traffic grows higher - it becomes a real bottleneck.  Imagine a business where the boss has not learned to delegate tasks to underlings.  Everything must be ok'd by the big boss.  As the business grows larger, that one point - in software a serialization point - becomes the point of contention.  Can get ahold of the big boss, business slows to a standstill.

Better to delegate those tasks to direct reports - managers that you have given the authority to make certain decisions or sign checks for example.  In our infrastructure example we have components that are "eventually consistent" and can provide answers to questions about the data, or provide services that give answers that are sufficient.

3. Complex Infrastructures

Look at your network diagram.  Can you grok it quickly?  Can you explain it to new hires without struggling?  Or are their complex relationships, dependencies or assumptions built into it?  Simplify the stack and keep functional components on their own servers.   Simplicity is a discipline, like traveling cross-country without packing huge bags.  It requires you to constantly trim the fat, and eliminate unnecessary pieces or parts.

4. Single Database Assumption

Even if you're not at the point where you need to scale your database tier in the beginning, keep it in the back of your mind as you architect your application.  Don't assume you will talk to one backend datastore.  Build for a read handle and a write handle.  Then you will have already done much of the work to later scale since most applications do 80% reads and 20% writes.  You can easily load balance over multiple read-only slaves if necessary as well.

5. Throwing Hardware at a Problem

If you try to sidestep the hard stoic work of tuning your application, optimizing the work it is doing, disk I/O it is performing, or network roundtrips you are making, it will come back to bite you.  Thinking you can just buy bigger iron, and scale vertically will only get you so far.  One simple cannot compete with the other.  Buying hardware can get you 2x, 5x, maybe 10x.  Tuning SQL can get you 10,000x speedup.

When migrating to the cloud consider security and resource variability, the cultural shift for operations and the new cost model.

1. Costs Move to Operating Expenses

When computing resources move from hardware, components and infrastructure that you buy and own, to those you rent by the hour, they also change places on your balance sheet.  They move from being capital expenses that depreciate over time to operating expenses.

At the end of the day, this is a very good thing because of the idea of discounted cash flow, and cost of capital in accounting.  It's always better to spend your money later.

That said it shifts how servers are provisioned, and how business units, and managers think about those expenses.  So expect some hiccups in that regard.

2. Operations & Automation - Cultural Shift

Best practices include solid disaster recovery, backups and then fire drills to prove all your ducks are in a row.  But best practices often get sidelined for more pressing concerns, and other obstructing priorities.

With virtual cloud servers, you start with a server that is less reliable to begin with, and a lower SLA to boot.  So automation from your starting point moves to front and center.  This requires a more managed approach to rolling out servers, akin to a small software development team in the habit of rolling out code manually, being pressed to move into version control.  This new managed approach to web operations is a good thing, encouraging configuration management, and automating resource provisioning tools like Chef.  Still it is a cultural shift, and one that won't happen overnight.

3. Security

Although the Amazon environment does not provide perimeter security, the security groups model can provide equally good security if managed well.  But one also has to manage other things closely such as:

• authentication keys
• machine images - built without sensitive data in the AMI

Furthermore compliance questions are often on everyones mind.  While you cannot say for sure where your data is physically located in the AWS environment, you can control whether it is encrypted or not, and you control those encryption keys.  Nevertheless legal requirements may have other plans in this regard.

Consider also what would happen if your cloud provider were hit with a lawsuit that being overly broad, sweeps some of your servers into its net.  How will you handle such a scenario?  Use multiple cloud providers and test your disaster recovery scenarios.

4. Resource Variability

Like moving from big iron to commodity servers fifteen years ago, cloud hosted environments can take some getting used to.  In the virtualized environment, the CPU, disk and network resources you are allocated are based on computed averages.  At any time the disk I/O throughput for example can fluctuate higher, or quite a bit lower than the service level agreement for that instance type.

All of the practices and methods for promoting scalability, decoupling components, caching agressively, and scaling horizontally will reduce the impact of these resource variabilities.  In the same way the internet masses can descend on your website without warning, building a robust infrastructure to stand up to the vagaries of virtualization will go a long way towards helping you scale smoothly when traffic spikes.

Spencer Johnson is a great writer.  His business book classic was a real page turner.  He takes a page from the REWORK book and that's a good thing.

Who Moved My Cheese is a story about mice living in a maze happy and content that they have an unlimited supply of cheese.  Then one day the cheese runs out.  Some approach this new reality by sniffing, and scurrying.  Before long they find new cheese, and more opportunities than they had before.  Some other mice hem and haw, and resist the new reality, pointing fingers, getting angry, but not facing the situation.

Of course these are situations we all face everyday.  Outsourcing changes the job landscape, technologies such as the internet disrupt established industries like newspapers, music, and media.  What's more the recent upheaval in the financial industry had at least some basis in the computerization of trading, and the acceleration of global markets.

The landscape is always changing.  Johnson's book speaks great lessons that we can all understand, and benefit from everyday in business.

Find Who Moved My Cheese on Amazon.

1. This page or area of the website is very slow, why?

There are a lot of components that make up modern internet websites, and a lot of places to get stuck in the mud.  Website performance starts with the browser, what caching it is doing, their bandwidth to your server, what the webserver is doing (caching or not and how), if the webserver has sufficient memory, and then what the application code is doing and lastly how it is interacting with the backend database.

With all this complexity, it's no wonder so many sites struggle.  Typically these types of analysis start with some load testing, to stress test your setup, so you can watch for leaks.  Then some tools are applied to the webserver tier, and the database tier to see where the bottleneck lies.  It may be in the network itself, and how much data is passed back and forth.  Or it may be badly formed queries being sent to the database, asking it to do more work than it needs to.  Imagine looking up a friends phone number with only a first name.  You're going to be there all day digging through the phone book, getting nowhere.  This happens when a database isn't indexed properly.

At the end of the day we've seen these types of scenarios over and over, and know where to look to sniff out the trouble.

2. How can we make our database go faster?

Databases are tremendous workhorses, and they do a heroes job of storing a retrieving just the information we want.  But they require care and feeding.

• Monitor and tune your SQL queries
• Use the fastest disk subsystem you can - RAID 10
• Allocate all of your memory & in the right places

3. Are we covered as far as backups & high availability?

I hear this question a lot as well, and it's a good one to be concerned about.  With all your crown jewels in a central data store you want to know for sure you can restore it.  What's the best way to know that?  Actually restore your backups.  Run a fire drill.  Rebuild the whole thing!  Here are all the pieces you'll need to consider:

• All packages installed - from the ground up
• All software configurations, especially apache, memcache, varnish, MySQL
• All your own software components - from your version control system
• A dump of your database
• Database transaction logs to roll forward changes since the dump happened
• AMIs if you are in the cloud

If you're lucky enough to be in the cloud, all of this can be scripted.  Run fire drills to your hearts content, and kill the instances when you're satisfied that you're covered!

With the fast growth of virtualized data centers, and companies like Google, Amazon and Facebook, it's easy to forget how much is built on open-source components, aka commodity software.  In a very real way open-source has enabled the huge explosion of commodity hardware, the fast growth of the internet itself, and now the further acceleration through cloud services, cloud infrastructure, and virtualization of data centers.

Your typical internet stack and application now stands on the shoulders of tens of thousands of open source developers and projects.  Let's look at a few of them.

1. Operating System - Linux

The commodity hardware craze would never have happened without the help of an open-source operating system to run on it.  Linux is an old story now, nonetheless everything else stands on it's shoulders.

2. Multi-purpose Webserver - Apache

As of July of 2011, Apache stands with 66% of the webserver market, above Microsoft IIS with 17% according to Netcraft.  That speaks volumes about the popularity of the open source webserver.  All sorts of applications and languages use the apache webserver as the way to serve up content on the internet.  In effect there is a 66% chance that your browser is talking to Apache for any given site you visit.

3. Multi-purpose Relational Database - MySQL

Back in the late 90's we worked at tons of internet startups.  They all used Apache on the frontend and Oracle on the backend.  These days we rarely here of internet sites aka web-facing databases on Oracle.  That's because MySQL has matured quite a lot in the past decade, and now provides all of the basic bells and whistles that a complex internet site requires.  Transactions, crash protection, high availability features, and data integrity.

4.Content Management System - Drupal, Django, WordPress, Joomla

The growth and popularity of open-source CMS systems is also evident.  Although much of what they do can and has been built from scratch, they enable a better starting point, providing you a strong framework around which to build a full-feature and scalable site.

5. Languages and Frameworks - PHP, Perl, Ruby on Rails

Let's not forget the languages and frameworks that all of these web applications are written in.   PHP with it's very DIY evolution, has grown to fill the needs of real developers.  With tons of community development, contributions and support, it also is illustrative of what open-source has brought to the web.

One very strong case for cloud computing is that it can satisfy applications with seasonal traffic patterns.  One way to test the advantages of the cloud is through a hybrid approach.

Cloud infrastructure can be built completely through scripts.  You can spinup specific AMIs or machine images, automatically install and update packages, install your credentials, startup services, and you're running.

All of these steps can be performed in advance of your need at little cost.  Simply build and test.  When you're finished, shutdown those instances.  What you walk away with is scripts.  What do we mean?

The power here is that you carry zero costs for that burst capacity until you need it.  You've already build the automation scripts, and have them in place.  When your capacity planning warrants it, spinup additional compute power, and watch your internet application scale horizontally.  Once your busy season is over, scale back and disable your usage until you need it again.

Shoe leather cost is similar to opportunity cost.  It refers to the cost of counteracting inflation by keeping less of your assets in cash.  Your strategy would require more trips to the bank and more walking, and incur a cost in the wearing out of the leather in your shoes.

All joking aside, it's an interesting idea.  It highlights how there are all sorts of hidden costs to different strategies.  There are hidden costs to using coupons, loyalty cards, frequent flyer miles, managing assets & investments, hiring resources and in general running a business.  Let's look at a few.

Cost of R&D

Years ago I wanted to open a mutual fund and did some research.  Luckily a friend pointed me towards Vanguard, and I got on the index investing bandwagon early.  I've had a lot of opportunities to buy stocks directly, but I've seen too many surprises in the market even in areas of tech stocks and business that I understand very well.

What's more stock investing requires attention, research into fundamentals, and certainly a bit of luck.  In my case my background is engineering, so finance is not my strong suit.  In effect there is more cost hidden there, getting up to speed in a new discipline and the many hidden surprises and fine print.

By choosing an index fund strategy you track the market, are guaranteed to beat 90% of stocks and mutual funds over time, pay a very very small fee as those funds are not actively managed by a fund manager with costs and overhead.  What's more the shoe leather cost is very low.   It's a no brainer.

Cost in People Search

Whatever technologies you choose, you embark down a path with costs on all sides.  Costs of hardware & software components, costs of licenses, and costs to find people with those skill sets.  Deciding to use the latest wiz-bang technology as an early adopter that looks like it will provide you a lot of extra bang for your buck.

Consider carefully what the market adoption of that technology is, as you will inevitably have to find resources skilled in servicing those components down the line.  Lean heavily toward widely adopted technologies and you'll have an easier time finding skilled technologists in the future.

Not Your Core Business

Early on publishing our newsletter, we chose to host the newsletter software ourselves.  The open-source solution was robust and full featured.  However many of the features proved cumbersome to configure.  Since we had the basic functionality working, we left the more complex switches alone.  This solution was a free one since we already hosted a server for the website.

Eventually we switched to mailchimp.com to host the newsletter.  Suddenly many hidden costs came to light.  By hosting ourselves we were ending up in spam folders for a lot of users.  So we were reducing our audience.  Plus we could not track who was reading the newsletter, which issues were popular, or what sections.  That analytical data provide invaluable.  What's more mailchimp integrates and automates a lot of other functions that were done manually before.  Freeing up time!

To Sum Up

Whenever you are spending inordinate time on something that is not your core business, chances are the shoe leather cost is high, and outsourcing that to a provider who specializes in that become a great cost saver.

Rework is chock full of ideas

Jason Fried and David Heinemeier Hansson's new book REWORK is one of the best startup business books I've read since Alan Weiss' Million Dollar Consulting. If you're already a fan of their signal vs noise blog, you'd be familiar with their terse style. Sharp and to the point.

Which is why you can pick it up and read it in a few hours.  You'll want to because it's well written and pared down to essentials.  In fact the book reads like their workflow advice, less mass, do it yourself, cut out the fat, concentrate on essentials.  As such they are clearly practicing what they preach, which I like.

This book isn't really for large businesses and corporate success.  It's for startups, and freelancers, and small teams.  It focuses on how and why you should do something yourself before outsourcing it so you can learn what to look for.

Here are some choice quotes from the book:

1. The menus in failing restaurants offer too many dishes.
2. When you impose a deadline you gain clarity.
3. Timeliness is more important than polish
4. Reality never sticks to best-case scenarios.
5. Do less than your competitors to beat them.
6. Share information that's valuable and you'll slowly build a loyal audience.
7. Trade the dream of overnight success for slow, measured growth.
8. Getting back to people quickly is the most important thing you can do in customer service.
9. Optimize for now and worry about the future later.
10. You don't need more hours you need better hours.

In fact I started out with a list of my favorite twenty quotes from the book and cut it down to these ten.  Go get this book, you'll be glad you read it.

Find REWORK on amazon

If you're headhunting a cloud computing expert, specifically someone who knows Amazon Web Services (AWS) and EC2, you'll want to have a battery of questions to ask them to assess their knowledge.  As with any technical interview focus on concepts and big picture.  As the 37Signals folks like to say "hire for attitude, train for skill".  Absolutely!

1. Explain Elastic Block Storage?  What type of performance can you expect?  How do you back it up?  How do you improve performance?

EBS is a virtualized SAN or storage area network.  That means it is RAID storage to start with so it's redundant and fault tolerant.  If disks die in that RAID you don't lose data.  Great!  It is also virtualized, so you can provision and allocate storage, and attach it to your server with various API calls.  No calling the storage expert and asking him or her to run specialized commands from the hardware vendor.

Performance on EBS can exhibit variability.  That is it can go above the SLA performance level, then drop below it.  The SLA provides you with an average disk I/O rate you can expect.  This can frustrate some folks especially performance experts who expect reliable and consistent disk throughput on a server.  Traditional physically hosted servers behave that way.  Virtual AWS instances do not.

Backup EBS volumes by using the snapshot facility via API call or via a GUI interface like elasticfox.

Improve performance by using Linux software raid and striping across four volumes.

2. What is S3?  What is it used for?  Should encryption be used?

S3 stands for Simple Storage Service.  You can think of it like ftp storage, where you can move files to and from there, but not mount it like a filesystem.  AWS automatically puts your snapshots there, as well as AMIs there.  Encryption should be considered for sensitive data, as S3 is a proprietary technology developed by Amazon themselves, and as yet unproven vis-a-vis a security standpoint.

3. What is an AMI?  How do I build one?

AMI stands for Amazon Machine Image.  It is effectively a snapshot of the root filesystem.  Commodity hardware servers have a bios that points the the master boot record of the first block on a disk.  A disk image though can sit anywhere physically on a disk, so Linux can boot from an arbitrary location on the EBS storage network.

Build a new AMI by first spinning up and instance from a trusted AMI.  Then adding packages and components as required.  Be wary of putting sensitive data onto an AMI.  For instance your access credentials should be added to an instance after spinup.  With a database, mount an outside volume that holds your MySQL data after spinup as well.

4. Can I vertically scale an Amazon instance?  How?

Yes.  This is an incredible feature of AWS and cloud virtualization.  Spinup a new larger instance than the one you are currently running.  Pause that instance and detach the root ebs volume from this server and discard.  Then stop your live instance, detach its root volume.  Note the unique device ID and attach that root volume to your new server.   And the start it again.  Voila you have scaled vertically in-place!!

5. What is auto-scaling?  How does it work?

Autoscaling is a feature of AWS which allows you to configure and automatically provision and spinup new instances without the need for your intervention.  You do this by setting thresholds and metrics to monitor.  When those thresholds are crossed a new instance of your choosing will be spun up, configured, and rolled into the load balancer pool.  Voila you've scaled horizontally without any operator intervention!

6. What automation tools can I use to spinup servers?

The most obvious way is to roll-your-own scripts, and use the AWS API tools.  Such scripts could be written in bash, perl or other language or your choice.  Next option is to use a configuration management and provisioning tool like puppet or better it's successor Opscode Chef.  You might also look towards a tool like Scalr.  Lastly you can go with a managed solution such as Rightscale.

7. What is configuration management?  Why would I want to use it with cloud provisioning of resources?

Configuration management has been around for a long time in web operations and systems administration.  Yet the cultural popularity of it has been limited.  Most systems administrators configure machines as software was developed before version control - that is manually making changes on servers.  Each server can then and usually is slightly different.  Troubleshooting though is straightforward as you login to the box and operate on it directly.  Configuration management brings a large automation tool into the picture, managing servers like strings of a puppet.  This forces standardization, best practices, and reproducibility as all configs are versioned and managed.  It also introduces a new way of working which is the biggest hurdle to its adoption.

Enter the cloud, and configuration management becomes even more critical.  That's because virtual servers such as amazons EC2 instances are much less reliable than physical ones.  You absolutely need a mechanism to rebuild them as-is at any moment.  This pushes best practices like automation, reproducibility and disaster recovery into center stage.

8. Explain how you would simulate perimeter security using Amazon Web Services model?

Traditional perimeter security that we're already familiar with using firewalls and so forth is not supported in the Amazon EC2 world.  AWS supports security groups.  One can create a security group for a jump box with ssh access - only port 22 open.  From there a webserver group and database group are created.  The webserver group allows 80 and 443 from the world, but port 22 *only* from the jump box group.  Further the database group allows port 3306 from the webserver group and port 22 from the jump box group.  Add any machines to the webserver group and they can all hit the database.  No one from the world can, and no one can directly ssh to any of your boxes.

Want to further lock this configuration down?  Only allow ssh access from specific IP addresses on your network, or allow just your subnet.

For more help with EC2 interviews or evaluations, skype us or call our NYC offices at +1-212-533-6828

Does anyone remember 15 years ago when the dot-com boom was just starting?  A lot of companies were running on Sun.  Sun was the best hardware you could buy for the price.  It was reliable and a lot of engineers had experience with the operating system, SunOS a flavor of Unix.

Yet suddenly companies were switching to cheap crappy hardware.  The stuff failed more often, had lower quality control, and cheaper and slower buses.  Despite all of that, cutting edge firms and startups were moving to commodity hardware in droves.  Why was it so?

As more companies adopted commodity hardware and Linux, larger firms began to consider the combination too.  After all, all these success stories couldn't be wrong.

Fast forward to the present and we see a very similar pattern.  Everyone is moving to a crappy platform, aka the cloud.  Virtualization means I can't guarantee CPU power, it means my disk I/O experiences variability that I don't like.  My applications don't like it, my operations team doesn't like it.  My performance team complains.

Where's the pressure coming from? There is debate between operational expenses versus capital expenses, sure.  But in the end the advantages remain large:

• Zero Up Front Costs
• Instant Compute Resources - only an API call away
• Easily Scale Vertically & Horizontally
• Free Disaster Recovery - encapsulated into an infrastructure script

I would argue that the cloud will also push adoption of infrastructure automation and infrastructure as code as well.  This will push tools and technologies such as Puppet, cfengine and Chef for configuration management and cloud provisioning.  These tools, although a paradigm shift for systems administrators who are used to doing things manually, bring a new level of quality control and best practices to web operations.  It parallels the shift in software development to using version control, considered now to be an essential component of any professional software project.

It's so good that creative firms and technologists are spending time figuring out how to deal with the limitations of cloud infrastructure and virtualization:

• variable disk I/O
• variable CPU performance
• higher failure rate
• security challenges
• privacy concerns

MySQL databases are great work horses of the internet.  They back tons of modern websites, from blogs and checkout carts, to huge sites like Facebook.  But these technologies don't run themselves.  When you're faced with a system that is slowing down, you'll need the right tools to diagnose and troubleshoot the problem.  MySQL has a huge community following and that means scores of great tools for your toolbox. Here are 7 ways to troubleshoot MySQL.

1. Use innotop

Innotop is a great tool for MySQL which despite the name monitors MySQL generally as well as InnoDB usage.  It's fairly easy to install, just download the perl script. Be sure to include a [client] section to your local users .my.cnf file (you have one don't you?).  Inside that section, place one line with "user=xyz" and one line with "password=abc".

If you're concerned that installing something new is too complicated, use the poor man's version:

$ watch 'mysqladmin proc'

2. Analyze & Tune Queries

You've heard it before, so I hope you're already doing it.  Not sure where to start?  Enable your slow query log first.  Set the long_query_time low enough to capture some queries.

Then use the great maatkit tool called mk-query-digest to analyze the slow.log file.  The results will amaze you.  You'll quickly be able to sift for the heaviest four or five queries.

Next do an EXPLAIN for each of those queries from the command line prompt.  Isolate those lines which return a large number of rows.  Attempt to add columns in the WHERE clause to reduce these.  Already have a where clause?  Be sure that column is indexed.  Try to reduce sorting, consider how you are joining tables, and so forth.  Eliminate SELECT * and instead specify which columns you want, and which rows.

All these changes will make a huge impact on performance.

3. Verify & Troubleshoot Replication

Have you ever wondered if your replication is working properly?  Do you wonder if all the data in your slave database perfectly matches your master database?  As it turns out, the slave can silently drift out of sync with the master.  We are lucky when this happens and MySQL returns an error.

In either case you'll need a tool to help you.  There are two included in maatkit which will come to your rescue.  The first mk-table-checksum is run periodically on the master.  It creates checksums in a table, and that tables data propagates through replication to the slave.  You then run another check with the same tool to verify the slave.

What happens if I find differences?  Well then you'll turn to another wonderful Maatkit tool mk-table-sync designed specifically for that purpose.  Check the manual for details on usage.

4. Use Aspersa

Another great tool by Baron Schwartz author of Maatkit is the Aspersa toolset.  It helps you collect system information and then do performance profiling on that data.  Great diagnostic tool.

5. General Database Tuning

You may want to get a birds eye view on what is happening in your database.  How many tables are InnoDB versus MyISAM?  Am I using indexes well?  Am I doing too many joins?  Are there some memory settings that are not quite tweaked as best they can be.

To be sure much of this type of knowledge is learned with years of experience, there is a tool which can give you a healthy start.  Enter the MySQLTuner!

Grab a copy with wget.  Since they grabbed a .pl top-level domain, the command is quite simple to remember:

$ wget mysqltuner.pl

Then set execute permissions, and go!  Be sure to edit your .my.cnf file and include a [client] section so the tool can pickup your user/pass automatically!

6. Database Performance and Load Testing

As of 5.1 release of MySQL, all distributions include a really great tool for doing performance and load testing called mysqlslap.  You can hand it a query, tell it to run that query 10 times, 100 times, or in 10 threads and so forth.  It will fire consecutive queries at the database, and you can then capture the slow query log, or use innotop and the OS monitoring tools to see how the server responds.

7. Operating System Monitoring Tools

Most Unix folks are familiar with the interactive monitoring tool top.  If you're not, please fire it up and watch it for a little while.  It will give you insight into what's happening right now on your server.  Please also take a look at iostat to see what is happening as far as disk I/O goes.  Databases are generally disk hungry so keep a close eye on this output. A tool called mpstat will give insight into what your processors are doing.  Incidentally top can be configured to shop all of the individual processors as well, so check the options there for details.  Lastly sar can come to your assistance.  The acronym stands for system activity reporter, and tells you what Linux knows about the activity on the box.

1. Backup outside of the Cloud

Some of the high profile companies affected by Amazon's April 2011 outage could have recovered had they kept a backup of their entire site outside of the cloud.  With any hosting provider, managed traditional data center or cloud provider, alternate backups are always a good idea.  A MySQL logical backup and/or incremental backup can be copied regularly offsite or to an alternate cloud provider.  That's real insurance!

2. Use alternate regions and availability zones

Amazon's outage in April underscored a lot of things.  That the service is still evolving and experiencing growing pains is one, and that many many internet firms rely on them is yet another.  Amazon's own documentation and best practices outline the need for mitigating against server failure.  Amazon instances are not as reliable as instances in a traditional hosting center, just as commodity hardware was not as reliable as Sun hardware all the while it was replacing it.

By using multiple availability zones and regions in your infrastructure design, you mitigate against lower SLAs and less reliable servers.  This is not an option, but rather a requirement to build a resilient infrastructure.

3. Use alternate cloud providers

As we mentioned previously at the very least an alternate cloud provider can be used for your backups.  But to build a further fault tolerant infrastructure, build servers and provide services out of multiple cloud providers.  This type of setup can provide the highest availability and protection against outages as it becomes very unlikely that any regional or business failure could take down multiple providers in different geographic regions.

4. Design & test for failure

Fire drills are a great way to test for outage in a planned way.  Run through all the steps to rebuild all components in your site.  Document those processes carefully as you go through them.  A central repository of documentation stands in for your lead engineer so that the knowledge to rebuild your site does not rely on any one single person in your business.

Further testing can involve a more shotgun approach.  This tests for true resiliency but requires a robustness of architecture, process and automation that is at the very highest level.  That's what neflix does with their chaos monkey.

5. Include configs, code, files & database

Your backups are best verified by fire drill.  Rebuilding the entire infrastructure from bare metal involves many moving parts.  From the lowest level you have hardware, then you have AMIs with the OS and from there you install particular packages that your application requires.  Those packages need configurations hopefully managed by a configuration management systems such as cfengine, puppet or chef.

On top of all of that you need your application itself, and any files that it builds or relies on in the filesystem.  Lastly your single backend datastore, likely your relational database such as MySQL would require it's data backup to restore all the stateful data your application relies on.

Deploying in the Amazon cloud is touted as a great way to achieve high scalability while paying only for the computing power you use. How do you get the best scalability from the technology?

1. Use Auto-scaling

Auto-scaling is a unique feature of cloud computing and Amazon's EC2 offering. Setup a load balancer and a couple of webservers for your application as you normally would. Design your webserver based on a template AMI that you'll reuse over and over. Then setup auto-scaling and set thresholds based on the traffic you forecast. When a threshold is passed, AWS will spinup a new instance of your webserver, and roll it into the load balancer pool automatically. Once traffic falls below the scale back threshold, Amazon will take a server out of the pool for you.

Be sure to monitor this activity to start with, and also run some load and stress tests on it. You want to ensure that it behaves as you expect, and no anomalous cases will cause you to spinup servers unnecessarily.

2. Horizontally Scale the Database Tier

MySQL offers a number of solutions here.  You can configure MySQL in a master-master active passive cluster, also known as circular replication.  In this configuration, MySQL will send all completed transactions to the other server in the cluster.  Your passive server can also handle read traffic for your application.  At the same time as you achieve high availability, you also get scalability by sending much of the SELECT traffic off to another server.

As the load grows further, roll in an additional read-only slave into your setup.  You can then further scale out reads.  As you do this, you'll continue to send all writes to the single active master database.

As an alternative, you may wish to employ Amazon's RDS solution, which is based on the MySQL community server, but with certain locked in settings.  In this configuration you will not be able to use an alternate MySQL distribution such as Percona's high speed server.  What this configuration does offer is multi-az configuration.  Keep in mind though that any MySQL server you spinup will have a region and availability zone as part of it's configuration, so you are always free to use those features of Amazon's network to increase availability.

If the load on the single master database continues to be a problem, you can vertically scale that node in place.  Do this by spinning up a new larger EC2 instance on and EBS root volume.  Then detach that root volume, stop your old instance, detach it's root volume and move it over to the new server.  Once you attach that live root volume, your new larger EC2 instance will be your original server, scaled UP in place!

3. Use Striped EBS Volumes

EBS is a great technology, as it brings the flexibility of a storage area network to each of your EC2 instances.  It's not without it's challenges though, and at times you will experience great variability in disk I/O throughput.  This can pose a challenge for the database tier especially.  Keep in mind too, that Amazon's infrastructure virtualization layer will dial up or dial down the amount of disk I/O your instance receives based on how large the instance is.

One additional way to get better EBS performance is by using Linux's software raid technology.  Since EBS already has redundancy built in, you can simply use striping or RAID 0 across a number of EBS volumes - we recommend four.

Be careful with such configurations as each of your EBS volumes won't operate by themselves now, but require all four to be complete.  This will impact EBS snapshot backups, as well.

There are a lot of scalability challenges we see with clients over and over. The list could easily include 20, 50 or even 100 items, but we shortened it down to the biggest five issues we see.

1. Tune those queries

By far the biggest bang for your buck is query optimization. Queries can be functionally correct and meet business requirements without being stress tested for high traffic and high load. This is why we often see clients with growing pains, and scalability challenges as their site becomes more popular. This also makes sense. It wouldn't necessarily be a good use of time to tune a query for some page off in a remote corner of your site, that didn't receive real-world traffic. So some amount of reactive tuning is common and appropriate.

Enable the slow query log and watch it. Use mk-query-digest, the great tool from Maatkit to analyze the log. Also make sure the log_queries_not_using_indexes flag is set.  Once you've found a heavy resource intensive query, optimize it!  Use the EXPLAIN facility, use a profiler, look at index usage and create missing indexes, and understand how it is joining and/or sorting.

2. Employ Master-Master Replication

Master-master active-passive replication, otherwise known as circular replication, can be a boon for high availability, but also for scalability.  That's because you immediately have a read-only slave for your application to hit as well.  Many web applications exhibit an 80/20 split, where 80% of activity is read or SELECT and the remainder is INSERT and UPDATE.  Configure your application to send read traffic to the slave or rearchitect so this is possible.  This type of horizontal scalability can then be extended further, adding additional read-only slaves to the infrastructure as necessary.

3. Use Your Memory

It sounds very basic and straightforward, yet there are often details overlooked.  At minimum be sure to set these:

• innodb_buffer_pool_size
• key_buffer_size (MyISAM index caching)
• query_cache_size - though beware of issues on large SMP boxes
• thread_cache & table_cache
• innodb_log_file_size & innodb_log_buffer_size
• sort_buffer_size, join_buffer_size, read_buffer_size, read_rnd_buffer_size
• tmp_table_size & max_heap_table_size

4. RAID Your Disk I/O

What is underneath your database?  You don't know?  Well please find out!  Are you using RAID 5?  This is a big performance hit.  RAID5 is slow for inserts and updates.  It is also almost non-functional during a rebuild if you lose a disk.  Very very slow performance.  What should I use instead?  RAID 10 mirroring and striping, with as many disks as you can fit in your server or raid cabinet.  A database does a lot of disk I/O even if you have enough memory to hold the entire database.  Why?  Sorting requires rearranging rows, as does group by, joins, and so forth.  Plus the transaction log is disk I/O as well!

Are you running on EC2?  In that case EBS is already fault tolerant and redundant.  So give your performance a boost by striping-only across a number of EBS volumes using the Linux md software raid.

5. Tune Key Parameters

These additional parameters can also help a lot with performance.

innodb_flush_log_at_trx_commit=2

This speeds up inserts & updates dramatically by being a little bit lazy about flushing the innodb log buffer.  You can do more research yourself but for most environments this setting is recommended.

innodb_file_per_table

Innodb was developed like Oracle with the tablespace model for storage.  Apparently the kernel developers didn't do a very good job.  That's because the default setting to use a single tablespace turns out to be a performance bottleneck.  Contention for file descriptors and so forth.  This setting makes innodb create tablespace and underlying datafile for each table, just like MyISAM does.

Best Practices, the term we hear thrown around a lot.  But like going on that new years diet, too often ends up more talk than action.

Manage Processes

Operator error ie typing the wrong command is always a risk.  Logging into the wrong server to drop a database or typing the dump command such that you dump data into the database, these are risks that operations folks face everyday.

Accountability is important, be sure all of your systems folks login to their own accounts.  Apply the least privileges model, give permissions on an as needed basis.

Set prompts with big bold names that indicate production servers and their purpose.  Automate repetitive commands that are prone to typos.

Don't be afraid to give developers read-only accounts on production servers.

Communicate Clearly

Regular team meetings, a la the Agile stand ups are a great way to encourage folks to communicate.  Bring the developers and operations folks together.   Ask everyone in turn to voice their current todos, their concerns and risks they see.  Encourage everyone to listen with an open mind.  Consider different perspectives.

Communication is a cultural attribute.  So it comes from the top.  Encourage this as a CTO or CIO by asking questions, communicating your concerns, repeat your own requests in different words and paraphrase.  Listen to what your team is saying, repeat and rephrase those concerns, and how and when they will be addressed.

Document Processes

A culture of documenting services, and processes is healthy.  It provides a central location and knowledge base for the team.  It also prevents sliding into the situation where only one team member understands how to administer critical business components.  Were that person to be unavailable or to leave the company, you're stuck reverse engineering your infrastructure and guessing at architectural decisions.

Better Practices

Rather than think of best practices as something you need to achieve today, think of it as an ongoing day-to-day quest for improvement.

• repetitive manual processes - employ automation & script those processes where possible.
• where steps require investigation and research - document it
• where production changes are involved - communicate with business units, qa & operations
• always be improving - striving for better practices

In search of a good book on Chef itself, I picked up this new title on O'Reilly.  It's one of their new format books, small in size, only 75 pages.

There was some very good material in this book.  Mr. Nelson-Smith's writing style is good, readable, and informative.  The discussion of risks of infrastructure as code was instructive.  With the advent of APIs to build out virtual data centers, the idea of automating every aspect of systems administration, and building infrastructure itself as code is a new one.  So an honest discussion of the risks of such an approach is bold and much needed.  I also liked the introduction to Chef itself, and the discussion of installation.

Chef isn't really the main focus of this book, unfortunately.  The book spends a lot of time introducing us to Agile Development, and specifically test driven development.  While these are lofty goals, and the first time I've seen treatment of the topic in relation to provisioning cloud infrastructure, I did feel too much time was spent on that.  It could have been better spent discussing cloud topics generally, and the chef architecture.  For instance a discussion of chef-solo was missing and the reader is plainly led towards the opscode hosted chef-server, rather than explaining the architecture, pros and cons of managing it yourself, and so forth.

By far my biggest beef with this title was that they waited to discuss the meat of provisioning infrastructure, ie bootstrapping an Amazon EC2 instance with Chef, until five pages from the end of the book!  The other complaint I had is that there were some glaring typos, like the itemized list of steps to setup the opscode platform on pages 16 & 17.

Since there aren't a lot of titles on the market yet covering Chef, you still might want to pickup this one, but I'm more eagerly awaiting Mr. Nelson-Smith's next title and more general introduction to the topic.

Related links:

Risks & Implications of Infrastructure as Code

Chef Download Page - Opscode

Discussion of Extreme Programming

Stephen Nelson-Smith's great blog

Software development is composed of individual components.  As developers are building these units, they build tests to verify them for correctness.  These tests can verify the environment, they can verify data, they can verify edge cases and include test harnesses.  In essence they verify that the code meets the design specification.

There are a few key advantages to the unit testing approach:

1. Self-Documenting - The tests themselves provide a type of documentation for the system as a whole.
2. Advances Refactoring - At a later date you may need to repair, rewrite or refactor portions of code.  Previously built unit tests provide a tremendous help to make sure your changes still meet the previous design specification.
3. Simplifies Functional Testing - With unit testing as an ongoing concern, the final components will likely perform more reliably, and if not the tests & self-documentation may point to how or why they fail to meet some specification.

Sean Hull Quora Discussion - What is software unit testing?

Amazon Web Services is a division of Amazon the bookseller, but this part of the business is devoted solely to infrastructure and internet servers.  These are the building blocks of data centers, the workhorses of the internet.  AWS's offering of Cloud Computing solutions allows a business to setup or "spinup" in the jargon of cloud computing, new compute resources at will.  Need a small single cpu 32bit ubuntu server with two 20G disks attached?  One command and 30 seconds away, and you can have that!

As we discussed previously, Infrastructure Provisioning has evolved dramatically over the past fifteen years from something took time and cost a lot, to a fast automatic process that it is today with cloud computing.  This has also brought with it a dramatic culture shift in the way that systems administration is being done, from a fairly manual process of physical machines, and software configuration, one that took weeks to setup new services, to a scriptable and automateable process that can then take seconds.

This new realm of cloud computing infrastructure and provisioning is called Infrastructure as a Service or IaaS, and Amazon Web Services is one of the largest providers of such compute resources.  They're not the only ones of course.  Others include:

• Rackspace Cloud
• Joyent
• GoGrid
• Terremark
• 3Tera
• IBM
• Microsoft
• Enomaly
• AT&T

Cloud Computing is still in it's infancy, but is growing quickly.   Amazon themselves had a major data center outage in April that we discussed in detail. It sent some hot internet startups into a tailspin!

More discussion of Amazon Web Services on Quora - Sean Hull