Category Archives: All

RDS or MySQL – Ten Use Cases

Amazon’s Relational Database Service is based on MySQL under the hood.  So many colleagues and clients ask me – should I go with RDS or MySQL? As with every technology question, the answer is – it depends.

Here are some scenarios to help you decide.

  1. I’m replicating into Amazon from a physical datacenter
  2. A: This setup is common if you’re using Amazon’s VPC or Virtual Private Cloud.  With a router dropped into your datacenter, VPC allows you to extend and spinup virtual instances from Amazon as if they’re sitting in your own existing datacenter.  Great stuff, but you won’t be able to replicate from your existing master MySQL instance to cloud Amazon RDS instances.  To do that, roll your own with MySQL 5.5 or Percona 5.5.  RDS can only work with other RDS instances.  At least for right now.

  3. I’m using Amazon and replicating to another cloud provider
  4. A: Again roll your own, as RDS won’t work with this configuration.

  5. I’m developer or small startup building a new application.
  6. A: Use RDS.  It’s easier to deploy, and though there are fewer levers and dials you can control, you can always dump your data and move it to roll your own deployment later.

  7. We’re a small shop without a fulltime DBA
  8. A: Use RDS as it is easier to deploy.  You’ll have a couple of replication options at your disposal.  There are read replicas which are basic slaves using MySQL’s built-in asyncronous replication.  You can scale horizontally with these read-only copies, but cannot failover to these instances.  For that you’ll need a multi-az configuration where you deploy across availability zones.  Multi-az is said to be syncronous so it’s likely built on top of a distributed filesystem such as DRBD.

    Keep in mind though there is no free lunch. RDS lacks the slow query log, which is the primary way you’ll identify errant queries, and fix them. You’ll need to be extra vigilant about QA & Test as code deploys add new SQL to your application. There *IS* a way to log slow queries to a table, which is ok for off and on use. However there’s quite some overhead to this feature being turned on all the time.

  9. We want to do master-master for easy failover and failback
  10. A: Use MySQL 5.5 or Percona 5.5.  The two configurations for RDS don’t support this.  With AWS Read Replicas you have one master and multiple read-only slaves, but no failover.  With Multi-AZ you can’t access the inactive secondary database *until* you failover.  Once you failover that instance becomes the primary.

  11. I’m concerned about clean data. Is replication bulletproof?
  12. A: No not out of the box.  Whether you are using RDS or MySQL there are various scenarios where MySQL slaves can drift out of sync from the master, without throwing an error. This likely impacts Read Replicas though it’s hard to say if it affects Multi-AZ.

    If you’re concerned about data integrity with MySQL replication (and you should be!) take a look at our Guide to MySQL Integrity Checking with Percona Toolkit.

  13. Our use case is to scale reads with multiple instances
  14. A: RDS can do this handily with Read Replicas.  As long as your requirements are for vanilla replicas, this will work well for you.

    To learn more about this checkout our Guide to Autoscaling MySQL on Amazon.

  15. I don’t want to be beholden to Amazon for my database
  16. A: Well then your choice is simpler.  Locking in with Amazon RDS means when you hit a bug, your hands are a bit tied.  Amazon’s DNA is as a DIY infrastructure provider, and although they’ve added support contracts to the mix, they’re not a Rackspace.  At least not yet.

    So there’s a tradeoff.  Go with a roll your own solution where you have control over all the nuts and bolts of your technology or go with something more managed.  RDS is a managed database solution, so to the extent things are automated, your hands are also tied when you hit a snag.

  17. We want to use an alternate MySQL distribution
  18. A: Again if you want to go with a Percona, MariaDB or Drizzle, you’re going to be using a roll your own distribution.

  19. We want to use an exotic replication topology

A: If you’re in this camp, you probably already know RDS isn’t going to support you.  MySQL’s replication technology can support a myriad of configurations. Here are a few that might work for you:

  • Master-master or Circular Replication
  • Distribution Master
  • Tree Replication Setup
  • Multi-source Replication
  • Logging Only Slave Server
  • Slaving off of a Slave

What Recession? – New York Tech Hiring Frenzy

startups hiringAccording to Crains, New York is digital jobs central.  ZocDoc, Thrillist, Foursquare and 10gen are just a few of the hot shots located at 568 Broadway, a tech hub in Soho.  Each of these firms is looking to double their headcount in the coming year.  Not bad considering the rough shape much of the economy is in.

Don’t Forget the Big Boys

Let’s not forget the huge tech firms that are also on a hiring binge,  Google, Facebook, Zynga, Twitter & Skype are all hiring.  Bloomberg Businessweek describes it literally as Silicon Valley setting up shop here!

This interactive map of the tech scene in New York is striking.

Population Growth

New York is also experiencing serious population growth.  The city expanded by 70,000 people in the last 15 months according to Gothamist.

According to another Crains article, what we’re seeing is the best job gains in 60 years!

Increased Competition for Talent

All this means increased competition for the best talent.  These days only the biggest or the coolest firms can snag the smartest people.

We wrote some hiring guides that might help: How to Hire a Great Software Developer and another about filling the Mythical MySQL DBA role.

A two part article – How to Hire a MySQL DBA and Part two.

All This and Fun Too!

And hey, what other city has spawned the European inspired Lunch Beat?

 

 

Ask Me Questions – Scalability, Performance, Cloud Computing

 

 

I blog a lot about various topics near and dear to me.  So I thought I’d turn the tables a bit, and offer the microphone up to readers.  Do you have questions on any of these topics?  Feel free to drop a note in the comments.

 

Business & Consulting

Are you a freelancer or independent consultant?  Struggling with some part of the business?  Or are you a CTO or Director looking to hire short term talent?  Glad to offer up advice and suggestions, just fire away!

Scalability

The goal of every hypergrowth company, from Pinterest to Facebook, Zynga to FourSquare.  How do they achieve it?  What architecture decisions make those applications grow effortlessly to meet user demand?  I’m sure you have some questions!

Performance

It’s on everyone’s mind, especially in the context of deploying in the cloud.  How best to achieve good performance?  Are you struggling with something specific?

High Availability

We want our systems to be available 24x7x 3million days a year!  Is it feasible?  What should we aim for in the real world?  Does our technology mix and hosting factor into the equation?  Ask away.

Startup Challenges

I’ve worked with a lot of startups over the years.  From the dot-com hey days to the more sober field we see before us today.  Many struggle with similar challenges.  Am glad to offer up comments and suggestions from what I’ve learned over the years.

Cloud Computing

What really works in the cloud? Is Amazon EC2 the only way to go?  What are the pros and cons of going with a Rackspace type provider that also has great service and the physical data center option?  How will I get good performance?  These and other questions are on everyone’s lips.  Feel free to comment if you have some concerns.

 

Business lessons from brokering a half million dollar deal

I’ve had my Manhattan loft in the market for some six months now. After dropping the asking price a couple of times, buyers are finally circling, and I’m close to finalizing a deal.  In that time however, I’ve learned some big lessons.  Real Estate, is the epitome of cutthroat capitalism and as such has some very potent lessons to teach about business.

Understand Interests – Two Sides to Every Argument

Brokers have an interest – to make their fee;  buyers have an interest – to lower the selling price, sellers have an interest to get the highest selling price.  On the surface this may seem self evident.  But dig a little deeper and you’ll learn more. Brokers are somewhat aligned with buyer and seller, but like a cab driver preferring lots of short trips, their incentive is to do more deals. Fewer deals with higher margins would actually cost them. A 1%-2% higher selling price may be significant to your bottom line but meaningless to theirs.

[quote]Business Lesson 1 – Pay close attention to each party’s interests and you’ll understand the negotiating dynamic more clearly.[/quote]

In brokering, like war, no one plays fair

Each industry has it’s own ugly side. No one calls back. No one emails back. There are no pleasantries. In some businesses where credibility and reputation are built over time, people do care–little bit. But in brokering a large deal they don’t.  Silence means “Hey, thanks for going out of your way to give us an hour of your time even though we were late.  We don’t care.  We no longer need you & we’re not gonna respond.”  Got it, heard you loud and clear!

[quote]Business Lesson 2 – Be prepared for selfishness and deception. In some cases watch out for cheaters and even fraud.  Don’t let this color all your interactions, but be aware that some parties may not have the same scruples as you do and some may even break the law. [/quote]

Markets are bigger than you are

You may think your house or apartment has some intrinsic value, based on renovations or the time you’ve held it, or some other more personal metric. However the market is ultimately what sets the price of the asset. Market trends can swing wildly and irrationally so be prepared to “get real”, as they say in the business.

[quote]Business Lesson 3 – Market forces set the price of a good or service. The door swings both ways so when there is a greater supply be prepared for the price to go down, and when location, asset or skills are in great demand, be prepared for the price to go up.[/quote]

Buyers move on emotion; facts get you only so far

Salesmen tell stories. It’s their job to engage, stir passions, get people to move. Sometimes these yarns are taller than you or I might like, but if people knew exactly what they wanted, brokers wouldn’t exist. This occurs across the spectrum in sales.

[quote]Business Lesson 4 – Pay less attention to the facts and arguments a salesmen presents to you. Only the underlying interest of each party will tell a clear story of motivations. Weigh that along with the market appetite and you should have a good idea of the right price.[/quote]

Protect your interests and time

Over the months I’ve been on the market, I’ve had countless brokers try to sell me their services. An exclusive seller agreement is valuable to a broker as it allows them to control the process and control the fee distribution. One broker made an appointment to come by with their buyer over a holiday weekend. Following up at the appointed time, they explained that they were running late though frankly the broker sounded hung over. Upon arrival they brought no buyer. Seeing my irritation, they quickly fabricated a story about a phantom buyer and their specific requirements. After wasting a few hours of my time and breaking up my holiday weekend, I never heard from them again. Not fun.

[quote]Business Lesson 5 – Qualify your prospect before you grant them your time. There are many parties who can and will waste your time if you’re not careful. Be diligent about weeding them out before clearing your schedule for them.[/quote]

Pro Blogging with the Pros

I picked up a copy of the Problogger book and flipped to the Blog Promotion chapter.  In it they recommended – Create compilation pages.

I tried it

I crafted a new post, selected some of my blogs most popular material, organized it with nice punchy one line summaries, and after about 20 minutes posted it.  Hey what the heck, I’ll give it a try.

It Worked!

  • interlink within your posts – this is huge!
  • highlight related posts
  • quote important points as excerpts
  • ask questions and invite comments

There are other great chapters on Social Media to get your posts going viral, and of course the ever important monetization topic is covered nicely.

Go pickup a copy of this book.  It’s worth much more than the cover price for sure!