Category Archives: Cloud Migrations

RDS or MySQL – Ten Use Cases

Amazon’s Relational Database Service is based on MySQL under the hood.  So many colleagues and clients ask me – should I go with RDS or MySQL? As with every technology question, the answer is – it depends.

Here are some scenarios to help you decide.

  1. I’m replicating into Amazon from a physical datacenter
  2. A: This setup is common if you’re using Amazon’s VPC or Virtual Private Cloud.  With a router dropped into your datacenter, VPC allows you to extend and spinup virtual instances from Amazon as if they’re sitting in your own existing datacenter.  Great stuff, but you won’t be able to replicate from your existing master MySQL instance to cloud Amazon RDS instances.  To do that, roll your own with MySQL 5.5 or Percona 5.5.  RDS can only work with other RDS instances.  At least for right now.

  3. I’m using Amazon and replicating to another cloud provider
  4. A: Again roll your own, as RDS won’t work with this configuration.

  5. I’m developer or small startup building a new application.
  6. A: Use RDS.  It’s easier to deploy, and though there are fewer levers and dials you can control, you can always dump your data and move it to roll your own deployment later.

  7. We’re a small shop without a fulltime DBA
  8. A: Use RDS as it is easier to deploy.  You’ll have a couple of replication options at your disposal.  There are read replicas which are basic slaves using MySQL’s built-in asyncronous replication.  You can scale horizontally with these read-only copies, but cannot failover to these instances.  For that you’ll need a multi-az configuration where you deploy across availability zones.  Multi-az is said to be syncronous so it’s likely built on top of a distributed filesystem such as DRBD.

    Keep in mind though there is no free lunch. RDS lacks the slow query log, which is the primary way you’ll identify errant queries, and fix them. You’ll need to be extra vigilant about QA & Test as code deploys add new SQL to your application. There *IS* a way to log slow queries to a table, which is ok for off and on use. However there’s quite some overhead to this feature being turned on all the time.

  9. We want to do master-master for easy failover and failback
  10. A: Use MySQL 5.5 or Percona 5.5.  The two configurations for RDS don’t support this.  With AWS Read Replicas you have one master and multiple read-only slaves, but no failover.  With Multi-AZ you can’t access the inactive secondary database *until* you failover.  Once you failover that instance becomes the primary.

  11. I’m concerned about clean data. Is replication bulletproof?
  12. A: No not out of the box.  Whether you are using RDS or MySQL there are various scenarios where MySQL slaves can drift out of sync from the master, without throwing an error. This likely impacts Read Replicas though it’s hard to say if it affects Multi-AZ.

    If you’re concerned about data integrity with MySQL replication (and you should be!) take a look at our Guide to MySQL Integrity Checking with Percona Toolkit.

  13. Our use case is to scale reads with multiple instances
  14. A: RDS can do this handily with Read Replicas.  As long as your requirements are for vanilla replicas, this will work well for you.

    To learn more about this checkout our Guide to Autoscaling MySQL on Amazon.

  15. I don’t want to be beholden to Amazon for my database
  16. A: Well then your choice is simpler.  Locking in with Amazon RDS means when you hit a bug, your hands are a bit tied.  Amazon’s DNA is as a DIY infrastructure provider, and although they’ve added support contracts to the mix, they’re not a Rackspace.  At least not yet.

    So there’s a tradeoff.  Go with a roll your own solution where you have control over all the nuts and bolts of your technology or go with something more managed.  RDS is a managed database solution, so to the extent things are automated, your hands are also tied when you hit a snag.

  17. We want to use an alternate MySQL distribution
  18. A: Again if you want to go with a Percona, MariaDB or Drizzle, you’re going to be using a roll your own distribution.

  19. We want to use an exotic replication topology

A: If you’re in this camp, you probably already know RDS isn’t going to support you.  MySQL’s replication technology can support a myriad of configurations. Here are a few that might work for you:

  • Master-master or Circular Replication
  • Distribution Master
  • Tree Replication Setup
  • Multi-source Replication
  • Logging Only Slave Server
  • Slaving off of a Slave

Oracle Announces Paid MySQL Add-ons

 Oracle starts charging for MySQL Add-ons

Exciting news, Oracle just announced commercial MySQL extensions that they’ll be offering paid extensions to the core MySQL free product.

To be sure, this has raised waves of concern among the community, but on the whole I suspect it will be a good thing for MySQL.  This brings more commercial addons to the table, which only increases the options for customers.  Many will continue to use the core database product only, and avoid license hassles while others will surely embark on a hybrid approach if it solves their everyday business problems. Continue reading

4 Considerations Migrating to The Cloud

When migrating to the cloud consider security and resource variability, the cultural shift for operations and the new cost model. Continue reading

Cloud for Burst Capacity

One very strong case for cloud computing is that it can satisfy applications with seasonal traffic patterns.  One way to test the advantages of the cloud is through a hybrid approach.

Cloud infrastructure can be built completely through scripts.  You can spinup specific AMIs or machine images, automatically install and update packages, install your credentials, startup services, and you’re running.

All of these steps can be performed in advance of your need at little cost.  Simply build and test.  When you’re finished, shutdown those instances.  What you walk away with is scripts.  What do we mean?

The power here is that you carry zero costs for that burst capacity until you need it.  You’ve already build the automation scripts, and have them in place.  When your capacity planning warrants it, spinup additional compute power, and watch your internet application scale horizontally.  Once your busy season is over, scale back and disable your usage until you need it again.

The New Commodity Hardware Craze aka Cloud Computing

Does anyone remember 15 years ago when the dot-com boom was just starting?  A lot of companies were running on Sun.  Sun was the best hardware you could buy for the price.  It was reliable and a lot of engineers had experience with the operating system, SunOS a flavor of Unix.

Yet suddenly companies were switching to cheap crappy hardware.  The stuff failed more often, had lower quality control, and cheaper and slower buses.  Despite all of that, cutting edge firms and startups were moving to commodity hardware in droves.  Why was it so? Continue reading

5 Ways to Avoid EC2 Outages

1. Backup outside of the Cloud

Some of the high profile companies affected by Amazon’s April 2011 outage could have recovered had they kept a backup of their entire site outside of the cloud.  With any hosting provider, managed traditional data center or cloud provider, alternate backups are always a good idea.  A MySQL logical backup and/or incremental backup can be copied regularly offsite or to an alternate cloud provider.  That’s real insurance! Continue reading

3 Ways to Boost Cloud Scalability

Deploying in the Amazon cloud is touted as a great way to achieve high scalability while paying only for the computing power you use. How do you get the best scalability from the technology? Continue reading

Amazon Web Services – What is it and why is it important?

Amazon Web Services is a division of Amazon the bookseller, but this part of the business is devoted solely to infrastructure and internet servers.  These are the building blocks of data centers, the workhorses of the internet.  AWS’s offering of Cloud Computing solutions allows a business to setup or “spinup” in the jargon of cloud computing, new compute resources at will.  Need a small single cpu 32bit ubuntu server with two 20G disks attached?  One command and 30 seconds away, and you can have that!

As we discussed previously, Infrastructure Provisioning has evolved dramatically over the past fifteen years from something took time and cost a lot, to a fast automatic process that it is today with cloud computing.  This has also brought with it a dramatic culture shift in the way that systems administration is being done, from a fairly manual process of physical machines, and software configuration, one that took weeks to setup new services, to a scriptable and automateable process that can then take seconds.

This new realm of cloud computing infrastructure and provisioning is called Infrastructure as a Service or IaaS, and Amazon Web Services is one of the largest providers of such compute resources.  They’re not the only ones of course.  Others include:

  • Rackspace Cloud
  • Joyent
  • GoGrid
  • Terremark
  • 3Tera
  • IBM
  • Microsoft
  • Enomaly
  • AT&T

Cloud Computing is still in it’s infancy, but is growing quickly.   Amazon themselves had a major data center outage in April that we discussed in detail. It sent some hot internet startups into a tailspin!

More discussion of Amazon Web Services on Quora – Sean Hull

Migrating to the Cloud – Why and why not?

A lot of technical forums and discussions have highlighted the limitations of EC2 and how it loses  on performance when compared to physical servers of equal cost.  They argue that you can get much more hardware and bigger iron for the same money.  So it then seems foolhardy to turn to the cloud.  Why this mad rush to the cloud then?  Of course if all you’re looking at is performance, it might seem odd indeed.  But another way of looking at it is, if performance is not as good, it’s clearly not the driving factor to cloud adoption.

CIOs and CTOs are often asking questions more along the lines of, “Can we deploy in the cloud and settle with the performance limitations, and if so how do we get there?”

Another question, “Is it a good idea to deploy your database in the cloud?”  It depends!  Let’s take a look at some of the strengths and weaknesses, then you decide.

8 big strengths of the cloud

  1. Flexibility in disaster recovery – it becomes a script, no need to buy additional hardware
  2. Easier roll out of patches and upgrades
  3. Reduced operational headache – scripting and automation becomes central
  4. Uniquely suited to seasonal traffic patterns – keep online only the capacity you’re using
  5. Low initial investment
  6. Auto-scaling – set thresholds and deploy new capacity automatically
  7. Easy compromise response – take server offline and spinup a new one
  8. Easy setup of dev, qa & test environments

Some challenges with deploying in the cloud

  1. Big cultural shift in how operations is done
  2. Lower SLAs and less reliable virtual servers – mitigate with automation
  3. No perimeter security – new model for managing & locking down servers
  4. Where is my data?  — concerns over compliance and privacy
  5. Variable disk performance – can be problematic for MySQL databases
  6. New procurement process can be a hurdle

Many of these challenges can be mitigated against.  The promise of the infrastructure deployed in the cloud is huge, so digging our heels in with gradual adoption is perhaps the best option for many firms.  Mitigate the weaknesses of the cloud by:

  • Use encrypted filesystems and backups where necessary
  • Also keep offsite backups inhouse or at an alternate cloud provider
  • Mitigate against EBS performance – cache at every layer of your application stack
  • Employ configuration management & automation tools such as Puppet & Chef

Quora discussion – Why or why not to migrate to the cloud?