Tag Archives: talent

Are career promotions like marriage… appealing until your first divorce?

surge pricing engineers

I was recently flipping through an interesting email list. It’s focused for tech leaders, managers & startup entrepreneurs. An HR team lead posted asking about “promotion paths” for engineers.

While I have an intuitive grasp of what engineers at those different levels look like, I’m having trouble making those concrete.

Join 32,000 others and follow Sean Hull on twitter @hullsean.

It struck me how antiquated the whole “career ladder” concept is. Work one job for 20-30 years. It feels like the fairytale of dating that leads safely to marriage. It all seems like a wonderful plan until it fizzles out, employees get jaded, they start seeing the real money being paid elsewhere, and begin looking around.

1. Talent in short supply

I’m not a CTO.  I should preface with that bit.  I’m a consultant.  That said I’ve worked in the tech industry for 20 years, so I have a bit of an opinion here.

Going to meetups, startup industry & pitch events. They’re all like a feeding frenzy. There are more companies hiring now than I remember back in 1998 & 1999. It’s just crazy.

Angel List says 18,000 companies are hiring right now. What about Made In NYC? That shows 735 jobs. And of course there’s Ycombinator who is hiring April 2016, which posts every other month. It has 720 comments as of this writing.

Also: Why I don’t work with recruiters

2. Are salary jumps always larger through external promotion?

I’ve seen a pattern repeated over & over.  An outside firm offers more money & grabs the talent, or the talent gets restless, starts looking & finds they get a bigger bump in salary by leaving, than by internal promotions.  

I don’t know why this is, but it seems almost universal that salary jumps are larger from outside firms, than internally through promotion.  

Also: Why devops talent is so hard to find

3. Building a better ladder

There are great posts on engineering ladders like this one from Neo and also this one from RTR. Also take a look at this one at Artsy. And of course somebody has to go and put theirs up on github. 🙂

All the titles & internal shuffling in the world aren’t going to hide industry pay for long.  When an employee gets wise to their career & the skills marketplace, they’ll eventually learn that title does not equal compensation.

Related: How to hire a developer that doesn’t suck?

4. Building a better culture

In a pricey city like New York, the only thing that seems a counterweight to this is phenomenal culture, chance to build something cool & be surrounded by coworkers you love.  To be sure bouncing around you get less of this. Companies like Etsy comes to mind. According to glassdoor companies like Airbnb, Hubspot & facebook also fit the bill.

Read: 8 questions to ask an aws expert

5. Surge pricing for engineers?

Alternatively to better ladders & promotions, perhaps what Uber did for taxi driving would make sense for hiring engineers too. Let the freelancing phenomenon grow even bigger!

Perhaps we need surge pricing for engineers. That way the very best really do get rewarded the most. Let the marketplace work it’s magic.

Also: When you have to take the fall

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Are top candidates evaluating your startup?

Editor & writer in friendly dialog

I work for a lot of startups. Many ask me for referrals. I play matchmaker when I can. But as the market continues to heat up, the demand for top talent is reaching a boiling point.

Join 29,000 others and follow Sean Hull on twitter @hullsean.

1. It’s a sellers market

That means folks with technical skills across the spectrum are very indemand. How in demand? Check Angellist, Made In NY or Indeed.com. From SRE’s to full stack developers, devops & automation experts to DBAs. Java, Ruby, Python, PHP, node.js, and of course design skills too.

I was speaking with a recruiter just today, and heard the same refrain…

Top candidates are evaluating us just as we are evaluating you.

That means firms must go the extra mile to stand out, and draw in the best talent.

Also: 5 Things toxic to scalability

2. Open the glassdoor

That’s right, manage your social media presence. Sites like Glass Door provide forums where employees past & present can discuss the day-to-day work environment. This gives prospects a chance to peer behind the curtain.

Other social media can be avenues too, from Facebook to Twitter. Having someone on staff that monitors online reputation can be crucial.

Related: Are SQL Databases dead?

3. Host a tech blog & meetups

A lot of top firms have great tech blogs. Truth be told many are dormant as demands of the day trump these outward facing initiatives. But they also put a face on the technical side of working for a firm. What problems are they solving? How cutting edge is their team?

Meetups are also a limitless forum. Smart minds will be mixing, your company brand will be spreading. Hosting technical discussions brings your firm front & center in multiple ways. It also brings possible new hires to your living room.

Read: Is high availability a myth?

4. Show warmth & transparency

I know everybody loves to grill candidates at interviews. But interviewees should be schooled on politeness & how to give a pleasant interview.

I remember one interview where I faced off with four other engineers at a round table. As the discussion unfolded, each aimed shots in succession, almost rapid fire at me. It was not only intimidating, but frustrating. Needless to say it made me a stronger more resilient interviewer, but it’s not a great way to welcome great talent. Buyer beware!

Also: The chaos theory of cloud scalability

5. Show me the money

I know I know, for engineers it’s not all about the money. Or is it? Truth be told compensation is always something prospects will weigh. Equity is fine, for what it is. But it’s a promise into the future.

More senior talent who have been through a few startups or even dot-com 1.0, may be a bit more dubious of abstract compensation. In the end competitive real dollars will speak volumes.

Also: Is upgrading Amazon RDS like a shit-storm that will not end?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Is there a devops talent gap?


New programming languages & services are being invented at a staggering pace. Hosting is changing, networking is changing, race to market is quickening.

But what does all of this mean to the search for talent? Who understands all these components? Who is an expert in any one?

Join 28,000 others and follow Sean Hull on twitter @hullsean.

1. That new car smell

We all remember that time. You know when you drove out of the dealership with your brand new wheels. Driving down the road, you feel on top of the world. You start dreaming of all the fun times you’ll have in your new car. For days and weeks afterward you walk out to your car, open the door & sit inside. It all feels special. You kind of hang out there for a few minutes enjoying the smell before you drive off. Right?

Let’s be vigilant to remember the same thing happens, or rather is happeing in technology all the time. As we automate our infrastructures with Ansible, Puppet & Chef, deploy continuous integration with Jenkins, Travis or Codeship, we should give pause. Each of these tools has it’s own syntax, it’s own bugs, it’s own community, it’s own speed of development & change, it’s own life.

Also: Does a four letter word divide dev & ops?

2. A lot of rushing

Google tells me the synonyms of agile are, nimble, lithe, supple & acrobatic. So in a fast moving world it’s no wonder agile is so big. Anything that allows us to respond to customers quicker & evolve our product faster is a good thing. Yes it is.

Over the years I’ve worked with a lot of clients & customers. Some right out of the gates are in a hurry. There is a sense of urgency even from the initial meeting. Although not in every case, sometimes these are the sign of the perpetually late. They end up throwing money around, throwing technology around, and all in a desperate attempt to plug a leaking ship.

In our race to automate & remain agile & nimble, we should also consider the future. Lets attempt to find a balance & consider future implications of technology decisions & choices.

Read: Is automation killing old-school operations?

3. Hosting, what’s that?

For many of the startups I work with today, they’ve never deployed on anything but Amazon. There was no rack of computers in a closet & a T1 line, circa 1997. There was no rackspace hosted servers or a colo in New Jersey circa 2005. Right from the beginning it was all on-demand computing.

This shift has surely brought a lot of benefit. But no one can argue it isn’t still very new. And with newness there is a learning curve. And bugs & surprises.

Related: Does a devop need to practice the art of resistance?

4. More complexity in troubleshooting

The wild ride really begins when you’re troubleshooting performance problems. Running your database on RDS you say? How the heck do I get to the terminal and run “top”? Can I do an iostat?

And what does iostat output really mean in multi-tenant Amazon, where your disk is an EBS volume across an unknown & unfriendly network. Who knows why it just slowed to a crawl, then sped up dramatically a few minutes later.

Even fetching the relevant logfile can be complicated. For all the problems the cloud eliminates, it sure introduces a few of it’s own. And who is the expert, and how to find them?

Read this: When fat fingers take down your database

5. More tech, fewer experts

I asked the question a few weeks back Do todays startups require assembly of a lot of parts that no one really understands?.

I’ve taken to browsing the stacks at the lovely StackShare site lately. There you can see what some of the top startups are using for their technology stacks. Docker, Yammer, Yelp, Stripe, Vine, Spotify & Stack Overflow are all there today.

There are new message queues like NSQ & programming languages like Markdown, Coffeescript & Clojure. Even Java. Are people still building web apps in Java. No please no!

While it’s wonderful to see such an explosion of innovation, I look at this from an operations perspective. In five years, when the first & second wave of developers at your startup have left, picture yourself trying to find talent in a long since out-of-fashion language like Dart or Swift. What’s more how do you untangle the mess you’ve now built?

Check this: Is the SQL database dead?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Best of hiring posts on scalable startups


Join 28,000 others and follow Sean Hull on twitter @hullsean.

Why I write about hiring

I’ve worked as a consultant for almost twenty years. Technology & professional services are pretty far removed from hiring, so why would I write about it?

As it turns out, finding projects, working with clients, and selling your skills & solutions has quite a lot in common to do with hiring.

As a services consultant, you’re more often a peer to technology directors & CTOs, while hiring for traditional roles is more of a boss employee relationship.


I’ve run into a lot of recruiters & hr folks over the years. Usually it means I’m talking to the wrong folks, as they’re gatekeepers & not decision makers. I wrote Why I don’t work with recruiters after some ups & downs.

Still they’re all a fact of life, and each of us has a role to play. So let’s play fair!


I’ve always wondered, Is Hiring a numbers game? That is does it bend more to persistence & throwing spagetti at the wall, or deliberate, precision searches?

MySQL interview

If you’re looking for a database expert, I put together
Top MySQL DBA interview questions and then another one
Advanced MySQL DBA Interview questions.

These are helpful not just to candidates, but to hiring managers, hr, recruiters & everyone in between.

Mythical talent

Since as far back as I can remember, DBAs have been in short supply. In the 90’s I was doing primarily Oracle work. There were never enough technical dbas. Many came from business backgrounds, and didn’t have operating system & hardware fundamentals.

As startups shifted to open source databases in droves during the 00’s, the situation became even worse. I wrote about
The mythical mysql dba – where can we find one?

Will NoSQL databases continue the same trend?

Hire a developer

With a little light humor, we throw some opinions into the fray around hiring devs with How to hire a developer that doesn’t suck.

As devops gains momentum, some see peace between the old-school silos of developers & operations. Some see the need for ops being supplanted by developers. We have some opinions too.

AWS Interview

Are you looking for an Amazon Web Services expert, who knows how to scale in the cloud? Devops & automation also on your mind? Check out
8 Questions to ask an amazon ec2 expert.

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

NYC Tech Firms Are Hiring – Map

Made In NY - Startups Hiring

If you haven’t noticed how much the NYC tech scene has grown recently, I’m afraid you’ve been hiding under a rock. It’s simply incredible.

Take a look at Mapped In NY a google maps mashup of the growing list popularized by the NY Tech Meetup called Made In New York.

Join 5000 others and follow Sean Hull on twitter @hullsean.


Having been around during the first dot-com boom back in the late 1990’s this is even more exciting to see. Despite the recession, New York’s economy is truly thriving!

New York’s Startup scene is truly thriving with a whopping 1263 firms, many of which are hiring.

Why is database administration talent in short supply? They are the Mythical MySQL DBAs

Also take a look at: Why Generalists are Better at Scaling the Web

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Here’s a sample

Hacking Job Search – Three Meaty Ideas

Also find the author on twitter @hullsean.

Demand for talented engineers has never been higher. It is in fact the dirty little secret of the startup industry, that there are simply not enough qualified folks to fill the positions.

What this means for you is that you have a lot of options. What it means for a hiring manager is that you will have to work even harder to find the right candidate. Just going to a recruiter isn’t enough. Use your network, go to meetups, follow Gary’s Guide daily.

Also check out our Mythical MySQL DBA piece where we talk about the shortage of DBAs and operations folks.

Further if you’ve dabbled in freelance or independent consulting, I wrote an interesting an in depth look at Why do people leave consulting. Understanding this can help avoid it in your own career, or avoid your resources leaving for better shores.

Find us on twitter @hullsean and linkedin where we share content and ideas everyday!

1. Build your reputation

As they say, your reputation precedes you. So start building it now. Fulltime or freelance, you want to be known.

Speaking, yes you can do it. Start with some small meetups, volunteer to speak on a topic. A ten person room is easier than 30, 50 or 100. Once you have a couple under your belt, fill out a CFP for Velocity, OSCon or some software developers conference. There are many.

Blog – if you’re not already doing so you should. Start with once a week. Comment on industry topics, controversial ideas, or engineering know-how. Prospects can look at this and learn a lot more than from a business card.

Write a book, yes you can. It may sound impossible, but the truth is that publishers are always looking for technical writers. Pick a topic near and dear to you. It’ll also give you endless material for your blog.

Go to meetups, you really need to be getting out there and networking. Get some Moo Business Cards and start working on your elevator pitch!

Social media – being active here helps your blog, and helps people find you. Twitter is a great place to do this. Interact with colleagues and startup founders, VCs and more. If you’re a hiring manager or CTO, you may find great programmers and devops this way.

We also wrote a more in-depth article Consulting and Freelance 101. It’s a three part guide with a lot of useful nugets.

Also take a look at our MySQL DBA Interview Guide which is as helpful to devops and DBAs as it is to managers hiring them!

Above all else, build your network & your reputation. It will put you in front of more people as a person, not a commodity or a resume in a pile of hundreds.

2. Qualify prospects

You definitely don’t want to take the first offer you get, and managers don’t want to hire the first candidate that comes along. You want two or three to choose from. Best way to do this is to have options.

If you’re a candidate, network or work through your colleagues. When you do get a lead, be sure you’re speaking to an economic buyer. If you’re not you’ll need to try to find that person who actually signs the checks. They are the ones who ultimately make the decision, so you want to sell yourself to them.

Get a Deposit – I know I know, if it’s your first freelance job, you don’t want to scare them off. Or maybe you do? The only prospects that would be scared off by this are ones who may not pay down the line. Dragging their feet with a deposit can also mean bureaucratic red tap, so be patient too.

Sara Horowitz has an excellent book Freelancers bible, we recommend you grab a copy right now!

Commodity You Are Not so don’t sell yourself as one. What do I mean? You are not an interchangeable part. You have special skills, you have personality, you have things that you’re particularly good at. These traits are what you need to focus on. The dime-a-dozen skills should sit more in the background.

You’ll also need to price and package your services. We talked about this in-depth in Consulting Essentials – Getting the Business.

We also think there is a reason Why Generalists are better at scaling the web.

3. Play the numbers game

For hiring managers this doesn’t mean working through recruiters that might be bringing subpar talent, it means networking through industry events, meetups, startup pitch and venture capital events. There are a few every single day in NYC and there’s no reason not to go to some of them.

For candidates, be eyeing a few different companies, and following up on more than one prospect. You should really think of this process as an integral and enjoyable part of your career, not a temporary in between stage. Networking doesn’t happen overnight, but from a regular process of meeting and engaging with colleagues over years and years in an industry.

At the end of the day hiring is a numbers game so you should play it as such. Keep searching, and always be watching the horizon.

Read this far? Grab our Scalable Startups for more tips and special content.

Top MySQL DBA interview questions (Part 2)

Also find Sean Hull’s ramblings on twitter @hullsean.

Continuing from our Top MySQL DBA interview questions (Part 1) here are five more questions that test a MySQL DBA’s knowledge, with two that will help suss out some personality traits.

  1. Disk I/O
  2. Disk performance should be an ever present concern to a DBA.  So although they don’t need to be a storage specialist, they should have a working knowledge.  Ask them about RAID versions, mirroring versus striping, and so forth.  Mirroring combines two disks as a unit.  Every write is duplicated on both disks.  If you lose one disk, you have an immediate copy.  Like a tandem truck that has spare tires running in parallel.  Lose one, and you don’t have to pull over immediately to replace it.  Striping spreads I/O over multiple disks so you on the one hand increase throughput linearly as you add disks.  That’s because you have more disks working for you.  At the same time you increase risk with each new disk you add, because the failure rate is then the sum total of all those disks.

    For relational databases the best RAID level is 10, which is striping over mirrored sets.  You use more disks, but disks are cheap compared to the hassle of any outage.

    If you’re deploying on Amazon, your candidate should be familiar with the Elastic Block Storage offering also known as EBS.  This is virtualized storage, so it introduces a whole world of operational flexibility.  No longer do you have to jump through hoops to attach, add or reconfigure storage on your servers.  It can all be done through command line API calls.  That said EBS suffers from variability problems as with any other shared resource.  Although Amazon guarantees your average throughput, the I/O you get at a given time can swing wildly from low to high.  Consider Linux software RAID across multiple EBS volumes to mitigate against this.

  3. How would you setup master/slave & master/master replication?
  4. A basic replication setup involves creating a full dump of the primary database, while it’s tables are locked.  The DBA should capture the master status, logfile & position at that time.  She should then copy the dump file to the secondary machine & import the full dump.  Finally the CHANGE MASTER TO statement should be run to point this database instance to it’s master.  Lastly START SLAVE should be issued.  If all goes well SHOW SLAVE STATUS should show YES for both of these status variables:

    Slave_IO_Running: Yes
    Slave_SQL_Running: Yes

    Master-master replication is similar, except one additional step.  After the above steps have run, you know that your application is not pointing at the slave database.  If you’re not sure, verify that fact first.  Now determine the logfile name & position on the slave with SHOW MASTER STATUS.  Return to the primary box, and run the CHANGE MASTER TO command to make it slave from the secondary box.  You’ve essentially asked MySQL to create a circular loop of replication.

    How does MySQL avoid getting into an infinite loop in this scenario?  The server_id variable must be set, and be unique for all MySQL instances in your replication topology.

    For extra credit, ask the candidate about replication integrity checking.  As important as this piece is to a solid reliable replication setup, many folks in the MySQL world are not aware of the necessity.  Though replication can be setup, and running properly, that does not mean it will keep your data clean and perfect.  Due to the nature of statement based replication, and non-deterministic functions and/or non-transactional tables, statements can make their way into the binary logs, without completing.  What this means is they may then complete on the slave, resulting in a different row set on the same table in master & slave instance.

    Percona’s pt-table-checksum is the preventative tool to use.  It can build checksums of all your tables, and then propagate those checksums through replication to the slave.  An additional check can then be run on the slave side to confirm consistency, or show which rows & data are different.

  5. How are Users & Grants different in MySQL than other DBs?
  6. Creating a grant in MySQL can effectively create the user as well.  MySQL users are implemented in a very rudimentary fashion.  The biggest misunderstanding in this area surrounds the idea of a user.  In most databases a username is unique by itself.  In MySQL it is the *combination* of user & hostname that must be unique.  So for example if I create user sean@localhost, sean@server2 and sean@server3, they are actually three distinct users, which can have distinct passwords, and privileges.  It can be very confusing that sean logging in from the local command line has different privileges or password than sean logging in from server2 and server3.  So that’s an important point.

  7. How might you hack a MySQL server?
  8. This is a good opportunity for the candidate to show some creativity with respect to operations and Linux servers.  There are all sorts of ways into a database server:

    a.bad, weak or unset passwords
    b.files with incorrect permissions – modifying or deleting filesystem files can take a database down or corrupt data
    c.intercepting packets – could reveal unencrypted data inside the database
    d.unpatched software – bugs often reveal vulnerabilities that allow unauthorized entry
    e.moving, disabling or interrupting the backup scripts – a possible timebomb until you need to restore
    f.DNS spoofing, could allow login as a different user
    g.generous permissions – may allow an unprivileged user access to protected data

    There are endless possibilities here.  Listening for creative thinking here, reveals how much that person will think thoroughly and effectively about protecting your systems from those same threats.

  9. Brain teasers, riddles and coding problems
  10. Google for a long time was a fan of these types of tests at interviews, but I’m not at all.  For one thing you filter for good test takers, and for another the candidate has no resources – either books or the internet at their disposal.

    Why not instead ask them to tell a story.  Storytelling conveys a lot of things.  It conveys a bit of teaching ability, which extends far beyond internalizing some multiple choice questions.  It tells you more about their personality, which as I’ve said is very important.  It shows how they solve problems, as they’ll take you through their process.  And gives them an opportunity to tell you a real world triumph they presided over.

Personality Questions

In my experience, some of the most important traits of a new hire center around personality traits, and how they might mix with your existing team.  Being punctual for an interview, for instance sets a precedent for many things.  But that door swings both ways, so if you want to hire these types of folks, don’t keep them waiting either!

Pay attention to whether or not the candidate takes some lead in the conversation at all.  This can indicate the person is a self starter.  Obviously a great candidate will also listen carefully and patiently to what you have to say, but may then take the ball and run with it somewhat.

Listen for signals that the person is active in the field, posting on forums, and attending conferences, meetups and forums on technology topics.  You might also ask them if they blog, and what topics interest them.

Top MySQL DBA interview questions (Part 1)

MySQL DBA interview questions

Also find Sean Hull’s ramblings on twitter @hullsean.

MySQL DBAs are in greater demand now than they’ve ever been. While some firms are losing the fight for talent, promising startups with a progressive bent are getting first dibs with the best applicants. Whatever the case, interviewing for a MySQL DBA is a skill in itself so I thought I’d share a guide of top MySQL DBA interview questions to help with your screening process.
It’s long and detailed with some background to give context so I will be publishing this in two parts.

The history of the DBA as a career

In the Oracle world of enterprise applications, the DBA has long been a strong career path. Companies building their sales staff required Peoplesoft or SAP, and those deploying the financial applications or e-business suite needed operations teams to manage those systems. At the heart of that operations team were database administrators or DBAs, a catchall title that included the responsibility of guarding your businesses crown jewels. Security of those data assets, backups, management and performance were all entrusted to the DBA.

In the world of web applications, things have evolved a bit differently. Many a startup are driven only by developers. In those smaller shops, Operations tasks are designated to one developer who takes on the additional responsibility of managing systems. In that scenario, Operations or DBA duties becomes a sort of secondary role to the primary one of building the application. Even in cases where the startup creates a specific operations role with one person managing systems administration, chances are they don’t also have DBA experience. Instead, these startups are more likely to manage the database as a typical Linux application.

When I grow up I (don’t) want to be a MySQL DBA

Where do they come from, and why don’t a lot of computer science folks gravitate towards operations, and DBA? This may be in part due to and the romance of certain job roles which we discussed in a past article, The Mythical MySQL DBA. This pattern appeared a lot in the Oracle world as well. Many folks who were career DBAs actually moved to that role from the business side. In fact you’d find that many didn’t have a computer science or engineering background in the first place. In my experience I saw many Linux and Unix administrators with a stronger foundation who would fit into the DBA role but were simply not interested in it. The same can be said of the MySQL side of the house. Computer science grads don’t get out of school aiming for a career in ops or as a DBA because it has never been regarded as the pinnacle. It’s typically the PROGRAMMERS who become the rockstars in a cool startup.

But as the Internet grows into a richer and more complex medium, things are changing. People talk about scalability, high availability, zero downtime and performance tuning. When brief outages cost millions in losses expectations are very high and that requires skilled, experienced DBAs.

We’ve made a list of comprised of skill questions, general questions and ‘good-to-know’ questions. Have fun grilling your candidate with them, although bear in mind that with interviews it’s not about knowing it all, rather how the person demonstrates critical thinking skills.

Skills Questions

  1. Why are SQL queries so fundamental to database performance?
  2. This is the one question which a DBA should have an answer to. If they can’t answer this question, they’re unlikely to be a good candidate.

    After a MySQL server is setup and running, with many of the switches and dials set to use memory, and play well with other services on the Linux server, queries remain an everyday challenge. Why is this?

    SQL queries are like little programs in and of themselves. They ask the database server to collect selections of records, cross tabulate them with other collections of records, then sort them, and slice and dice them. All of this requires MySQL to build temporary tables, perform resource intensive sorts and then organize the output in nice bite size chunks.

    Unfortunately there are many ways to get the syntax and the results right, yet not do so efficiently. This might sound like a moot point, but with modern websites you may have 5000 concurrent users on your site, each hitting pages that have multiple queries inside them.

    What makes this an ongoing challenge is that websites are typically a moving target, with business requirements pushing new code changes all the time. New code means new queries, which pose ongoing risks to application stability.

  3. Indexes – too many, too few; what’s the difference?
  4. Indexes are very important to the smooth functioning of a relational database. Imagine your telephone book of yore. I can look up all the people with last name of “Hull” in Manhattan because I have the proper index. But most yellow pages don’t include an index for *first* names even though they might occaisionally come in handy, for example with the names “Star” or “Persephone”.

    You can imagine that, if you had a phone book which you maintain and update, everytime you add or remove a name you also have to update the index. That’s right, and the same goes for your relational database.

    So therein lies the trade off, and it’s an important one. When you are *modifying* your data, adding, updating or removing records, you must do work to keep the index up to date. More indexes mean more work. However when you’re looking up data or *querying* in SQL speak, more indexes mean more ways of looking up data fast. One more trade off is that indexes take up more pages in your phonebook, and so too they take up more space on disk.

  5. Backup & Recovery – explain various types & scenarios for restore
  6. Backups come in a few different flavors that the DBA should be familiar with.

    Cold backups involve shutdown down the database server (mysqld) and then backing up all the data files by making a copy of them to another directory. To be really thorough, the entire datadir including binlogs, log files, /etc/my.cnf config file should also be backed up. The cold backup is a database in itself, and can be copied to an alternate server and mounted as-is.

    Logical backups involve using the mysqldump tool. This locks tables while it runs to maintain consistency of changing data, and can cause downtime. The resulting dump file contains CREATE DATABASE, CREATE TABLE & CREATE INDEX statements to rebuild the database. Note the file itself is not a database, but rather a set of instructions which can tell a MySQL server *HOW* to reconstruct the database. Important distinction here.

    Hot backups are a great addition to the mix as they allow the physical database data files to be backed up *WHILE* the server is up and running. In MySQL this can be achieved with the xtrabackup tool, available from Percona. Despite the name, it works very well with MyISAM and InnoDB tables too, so don’t worry if you’re not using xtradb tables.

    There are a few different restore scenarios, and the candidate should be able to describe how these various backups can be restored, and what the steps to do so would be. In addition they should understand what point-in-time recovery is, and how to perform that as well. After restoring one of the above three backup types, the DBA would use the mysqlbinlog utility to apply any subsequent transactions from the binary logs. So if the backup was made at 2am last night, and you restore that backup, the mysqlbinlog tool would be used to dig up transactions since 2am, and apply them to that restored database.

  7. Troubleshooting Performance
  8. Since this is an ongoing challenge with relational databases, a good grasp of it is crucial. One way to challenge the candidate would be to describe a recent performance problem you experienced with your infrastructure, and ask them how they would go about resolving it.

    If they struggle with the particulars of what you ran into, ask them to describe a big performance challenge they solved, what the cause was, and how they performed analysis.

    Typically, first steps involve mitigating the immediate problem by finding out what changed in the environment either operationally or code changes. If there is a bug that was hit, or other strange performance anomaly, the first stop is usually looking at log files. MySQL server error logs, and the slow query log are key files. From there, analyzing those files during the timeframe where problems occurred should yield some clues.

    You might also hope to hear some comment about metrics collection in this discussion. Tools such as cacti, munin, opennms, or ganglia are invaluable tools for drilling down on a past event or outage, and sifting through server stats to find trouble.

  9. Joins – describe a few kinds and how the server performs them
  10. A basic understanding of INNER JOIN and OUTER JOIN would be a great start. A simple example might be employees and departments. If you have four employees and two departments, an INNER JOIN of these tables together will give you the departments employees belong to. Add another employee without assigning her to a department, and the inner join won’t display her. Further adding a new department which doesn’t yet contain employees won’t display either. However performing an OUTER JOIN will give you those matches with null in the department field, and null in the employee field respectively.

    Thought of with another example, take a credit card company. One tables contains cardholders identity, their number, address, and other personal information. A second table contains their account activity. When they first join, they don’t have any monthly statements, so an INNER JOIN of cardholders with statements will yield no rows. However an OUTER JOIN on those two tables will yield a record, with a null for the statements columns.

Feeling like a MySQL expert yet? In Part 2 of Top MySQL DBA Interview Questions we’ll walkthrough four more questions plus a bonus.

The Mythical MySQL DBA

I’ve  been getting more than my fair share of calls from recruiters of late. Even in this depressed economic climate where jobs are rarer than a cab at rush-hour, it’s heartening to know that tech engineers are in great demand. And it’s even more heartening to think that demand for MySQL DBAs has never been better.

My reckoning was confirmed by a Bloomberg news report about stalwart retailers suffering from a dearth of talented engineers. Bloomberg cited Target’s outage-prone e-commerce site as a symptom of, among other things the market’s shortage. One of the challenges old-timers like Target face is having to compete with Silicon Valley startups as a fulfilling and ultimately, financially rewarding place to work. Continue reading The Mythical MySQL DBA