Highly available systems build redundancy into the application and the architecture layers to mitigate against disasters. Since computing systems are made from commodity hardware and components which are prone to failure, having redundancy at every layer is key.
Redundancy of switches, network interfaces, and load balanced webservers are fairly straightforward and run-of-the-mill. But clustering your database tier is another trick entirely. With MySQL, master-master active-passive can work quite well, running circular replication to send all changes to both nodes. Both nodes are able to handle production traffic, and you pick the one that is active simply by configuring your application to point to that. Use a technology like MMM or Pacemaker to front your database cluster with a virtual IP (vip), so no application or webserver changes are required to switch which node takes on the master role.
Redundant components are important in a single datacenter, but what if that datacenter goes out or gets hit by a natural disaster? Is your whole business out? That’s where geographic redundancy comes in. Geographic redundancy and geo load balanced DNS comes in. Having redundant copies of your whole site on both the east and west coast with geo-dns provides the next level of high availability.