Are top candidates evaluating your startup?

Editor & writer in friendly dialog

I work for a lot of startups. Many ask me for referrals. I play matchmaker when I can. But as the market continues to heat up, the demand for top talent is reaching a boiling point.

Join 29,000 others and follow Sean Hull on twitter @hullsean.

1. It’s a sellers market

That means folks with technical skills across the spectrum are very indemand. How in demand? Check Angellist, Made In NY or Indeed.com. From SRE’s to full stack developers, devops & automation experts to DBAs. Java, Ruby, Python, PHP, node.js, and of course design skills too.

I was speaking with a recruiter just today, and heard the same refrain…

Top candidates are evaluating us just as we are evaluating you.

That means firms must go the extra mile to stand out, and draw in the best talent.

Also: 5 Things toxic to scalability

2. Open the glassdoor

That’s right, manage your social media presence. Sites like Glass Door provide forums where employees past & present can discuss the day-to-day work environment. This gives prospects a chance to peer behind the curtain.

Other social media can be avenues too, from Facebook to Twitter. Having someone on staff that monitors online reputation can be crucial.

Related: Are SQL Databases dead?

3. Host a tech blog & meetups

A lot of top firms have great tech blogs. Truth be told many are dormant as demands of the day trump these outward facing initiatives. But they also put a face on the technical side of working for a firm. What problems are they solving? How cutting edge is their team?

Meetups are also a limitless forum. Smart minds will be mixing, your company brand will be spreading. Hosting technical discussions brings your firm front & center in multiple ways. It also brings possible new hires to your living room.

Read: Is high availability a myth?

4. Show warmth & transparency

I know everybody loves to grill candidates at interviews. But interviewees should be schooled on politeness & how to give a pleasant interview.

I remember one interview where I faced off with four other engineers at a round table. As the discussion unfolded, each aimed shots in succession, almost rapid fire at me. It was not only intimidating, but frustrating. Needless to say it made me a stronger more resilient interviewer, but it’s not a great way to welcome great talent. Buyer beware!

Also: The chaos theory of cloud scalability

5. Show me the money

I know I know, for engineers it’s not all about the money. Or is it? Truth be told compensation is always something prospects will weigh. Equity is fine, for what it is. But it’s a promise into the future.

More senior talent who have been through a few startups or even dot-com 1.0, may be a bit more dubious of abstract compensation. In the end competitive real dollars will speak volumes.

Also: Is upgrading Amazon RDS like a shit-storm that will not end?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

The chaos theory of cloud scalability

The.Rohit - Flickr

The.Rohit – Flickr

Reading Benedict Evans weekly newsletter, you’re bound to bump into something new & useful. His newsletter covers Mobile, but that also means it touches on a lot of other areas of tech, innovation & startups.

This week he pointed me to A Weissman’s The Chaos Theory of Startups. He argues a VC’s job is to help a startup identify the right framework. It’s about finding the signal in the noise.

Join 29,000 others and follow Sean Hull on twitter @hullsean.

I think you can carry this idea over to technical operations today. There are a few key maxims I follow to keep you on the scalable track.

1. Degrade gracefully

You’ve heard it before, but have you done anything about it?

Build a read-only or browse only mode into your application. Do it now. You will thank me. When your database goes down unexpectedly (with RDS this might happen sooner than you think), you want to be able to use your lovely read-only slave database. Browse only mode, forces developers to add read-only support in most application functions, keeping the site up and running, without a full visible and ugly outage.

Which brings me to point two, be sure to have copies of your production database. Real live, only read-only copies. In Amazon speak, this is a read-replica, in MySQL this is a slave database. Most startups I see these days have this, but if you’re one of the ones dragging your feet, do this now.

Also: Is the difference between dev & ops a four-letter word?

2. Monitor & measure

Amazon’s cloudwatch is fine for what it is, and so is New Relic. But employing a dedicated tool just for monitoring, such a Nagios & cacti can give you much more granular intelligence about what’s happening with your infrastructure. Nagios gives you the monitoring & alerting, Cacti gives you the history. It’s like a BI reporting tool for infrastructure.

Related: Is automation killing old-school operations?

3. Keep components simple

Keep it simple stupid? Don’t adopt new technologies, languages, or versions of software, without first vetting them. Ask questions:

o Is there an existing piece of software or service that can overlap this new one, killing two birds with one stone?
o Does everybody know this new technology?
o Does this choice of technology solve any other broad problems we have?
o Is there a large community around the project?
o Are there a lot of engineers with experience in this chosen technology?

Tellingly, many startups don’t have an operations person to start with. In those, the danger is developers choose new solutions, with no push back.

I asked… Does a four letter word divide dev and ops?

Read: Do managers underestimate operational cost?

4. Don’t force database abstraction

Object Relational Modelers, aka database middleware, are great in theory. We want a library that takes database & SQL drudgery away from developers. Why reinvent the wheel?

The trouble is database independent code doesn’t work, and never has. ORMs are painfully inefficient, selecting all columns, or repeatedly reading rows from tables. This causes serious traffic jams inside your database.

They come in various guises, Cake PHP, Active Record for Ruby, Hibernate for Java, SQL Alchemy for Python.

Also: Is the difference between dev & ops a four-letter word?

5. Be asynchronous

This means don’t make your application code wait. Make asynchronous calls to APIs & check back later, use software queues so traffic backups don’t clog your components & communication.

Avoid any type of two-phase or multi-phase commit. These are common in clustered databases, forcing a serialization point so nodes can agree on what data looks like. However they’re toxic to scalability. Look for technologies that use eventually consistent algorithms.

Also: Is the difference between dev & ops a four-letter word?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Is upgrading RDS like a shit-storm that will not end?

aws logo

Join 29,000 others and follow Sean Hull on twitter @hullsean.

Can RDS worsen an outage ?? That’s another way to think about this question. In my experience, it very clearly increases outages, by tying one or both hands behind your back. Believe me when I say, that is terribly frustrating when you’re putting out fires!

1. Changing Parameters

An everyday occurance, is the need to change database parameters. Want to enable a login, great no problem. Except in RDS it becomes a problem! Ok, you’re thinking, why is that?

In regular MySQL, you login with the shell & issue SET GLOBAL parameter = value; Nice, easy, straightforward. No servers restarting, no nonsense. If the parameter requires a reboot, MySQL will tell you.

In RDS, the process is waay more complex. First you edit a parameter group. You can copy an existing one, or change the one you’re using. If that parameter group applies to many servers, be careful!

Ok, what next? Now you APPLY that new parameter group. You can do so immediately, or during the next maintenance window. Here’s the tricky part. Is Amazon going to restart my instance? That’s something your boss or manager will surely ask you. Well you might think it would only do so if the parameter in question required it. But I tried to enable the general log recently and Amazon tells me the status of “pending-reboot”. This change shouldn’t require that! I’m sitting there scared Amazon might suddenly decide to reboot a production server for no reason!

This is where you feel you’ve lost control. You can dig through docs all you want, but you can’t ever say for sure if a managed service will behave predictably. There’s already more layers of software between you and your relational database. Not what you want.

Also: Did MySQL & Mongo have a beautiful baby called Aurora?

2. How much longer?

Another question you’ll ask yourself is, how long will this maintenance take? With MySQL at the command line, you can run through test after test & time the process. When you go to perform tasks offhours, you already have a clear picture.

With RDS, things can’t be predicted. Servers are restarted when they needn’t be. Rebuilds take forever, and you have no progress bar. EBS performance has a hiccup and your snapshot time doubles. The troubles go on and on.

Related: Is automation killing old-school operations?

3. Why did Amazon just force an OS upgrade?

Here’s another surprise I ran into. Again we have a managed solution, so Amazon must take opportunities when they can. But you pay for it in unpredictability.

Going to perform a MySQL 5.1 to 5.5 upgrade, and I’d run through test after test in advance. Timed the process to about 45 minutes. Then went to do it in production. Amazon decided to throw in the OS upgrade too, adding 40 minutes of surprise time. What’s worse? No progress bar on that either.

Upgrades are nerve wracking enough, without this kind of stuff scaring the daylights out of you.

Read: Do managers underestimate operational cost?

4. What’s happening on my server?

All of the questions about progress are opaque on RDS because you lack command line. You can’t watch processes, disk I/O or any of the granular stuff. In my surgery analogy below, it’s as though you can’t touch the patient, find their pulse or guage if their skin is cold, clammy or pale.

Also: Is the difference between dev & ops a four-letter word?

5. Surgery with blunt instruments

At the end of the day, RDS feels like surgery with blunt instruments. If command line were your scalpel, windows & GUI tools may be your remote video surgery. And worse still, RDS would be like doing surgery on the Opportunity Mars rover, after it’s landed & stuck in a valley. Everything is delayed, it’s hard to tell what’s going on, and the worst environment to work in when you have an emergency with your database.

If you have any operations experience, deploy your own MySQL on an EC2 instance. You’ll thank yourself later.

Also: Is zero downtime even possible on RDS?

Upside to RDS

Is there any upside? Why do people use it? Push-button replication. Check. Push-button multi-az, check. Those are great if you have no DBA. Automated backups so you don’t shoot yourself in the foot, check.

I guess there is *something* to love.

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Did Dropbox, Samsung, BestBuy & TimeWarner have to fail in 2014?

dropbox outage

It was a blockbuster year for web outages in 2014. From retailers to cable providers, it seems everyone had their 15 minutes of fame.

Join 28,000 others and follow Sean Hull on twitter @hullsean.

How can we protect our web property from the same fate? Here’s a short list of suggestions.

1. How much traffic can my site handle?

If you’re not already asking this question, you should be.

Answer: Load test well in advance

Load testing simulates what happens when the internet hordes descent on your website like a plague of locusts. This can be a good thing if you’re having a special sale or promotion. Or possibly a bad thing if you’re the victim of a denial of service attack.

Also: Is there a devops talent gap?

2. How quickly can I bring online more capacity?

A more nuanced version of number one, once you know what your site can handle, you need to know if you’re even ready for more.

Answer: Auto-scale & test.
Answer: Scale your database tier well in advance

Autoscaling is a feature popularized by amazon web services, which couples monitoring your traffic, with thresholds that will kick off more capacity. This capacity may be deploying new webservers to sit behind your load balancer. Whatever your method, be sure to test carefully.

Related: Is zero downtime even possible on RDS?

3. Cache, cache, cache!!

Optimizing your site means

Answer: Use an object & page cache.

If you’re not already using memcache, redis or elasticache, you should be. These object caches sit in between your application & your database, holding frequently accessed data & reducing load on your central database server.

Consider a page cache too like nginx or varnish. These act as sort of tiny webservers that are very low overhead, low memory & fast response. They provide a buffer in front of your main application server, and can service simpler but repetitive web requests.

A third caching layer sits in the db itself, called a query cache. This is implemented differently on Oracle, SQL Server, MySQL and Postgres, but the concept is similar. Cache frequent queries, so they don’t need to be rerun each time.

Read: Do managers underestimate operational cost?

4. Degrade gracefully

Answer: Build a browse only mode

If you’ve ever logged into an airline website & selected your flight, only to run into trouble during checkout, you know what a browse-only mode is! Many sites build this, along with other feature flags & toggles to allow them to degrade gracefully, that is keep the site up and running, while essential services may be inoperable.

Every production site should prioritize for this. Sooner or later you’re gonna need it!

Also: Why you need a performance dashboard like StackExchange

5. Try CloudFlare

CloudFlare is a service for protection from denial of service attacks. Integrate their servers into your infrastructure & a monitoring process begins, of each packet you receive. If a denial of service occurs, your site can be throttled down to reduce the impact & keep things online. This protection can double for heavily loaded legitimate traffic, keeping your website online in a worse case scenario.

Also: Big data scientist interview questions

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Big data scientist interview questions

Screen-shot-2012-08-02-at-1.28.35-PM

Everybody wants to hire a data scientist these days. According to read write the role is overhyped and overpaid. Hype aside, what’s a good approach to interviewing these hard to find people?

Here’s Read Write’s guide. Also Hilary Mason has an interview guide. Also take a look at Chris Pearson’s data scientist hiring guide.

Join 28,000 others and follow Sean Hull on twitter @hullsean.

While you’ll surely have technical questions to ask, we figure you may already have a handle on those. They’ll vary from business to business.

What we’ve put together is a series of questions that we hope will tease out some good stories, and underscore a candidates real-world experience. These are are also great for the cross section of folks involved in the hiring process, higher level managers, HR & recruiters, plus technical folks that may have data near & dear to them.

1. What’s common?

What key metrics do you see firms repeatedly missing? Why are they important?

You’ve worked as data scientist before, and run into a lot of problems at different firms. Inevitably, some of those repeat themselves. Give an example of a metric you see over-and-over, that’s essential, but often missing attention.

Also: Is the difference between dev & ops a four-letter word?

2. What’s your favorite?

What is your favorite KPI and why?

As a data scientist, you’ve probably approached different companies, and found a couple of indicators that you particularly like. Maybe they highlight potential for growth? Or lead to other interesting discoveries about the business?

Related: Is automation killing old-school operations?

3. Let’s talk dollars

Give an example of a financial benefit you brought to a firm. How & how much?

Give an example where a measurement you made, and a business change it informed had real ROI for the business. What was that discovery? How did the business make the change? What was the financial benefit to the bottom line?

Read: Do managers underestimate operational cost?

4. Business data discovery

Give an example where you discovered data the business didn’t know it had. What & How?

Sometimes businesses have assets stored, that have been forgotten. Perhaps they’ve been archived, or a collection job has been forgotten. Perhaps it’s a corner of salesforce that hasn’t been evaluated. How did you bring the new data to light, and make use of it?

Also: Is the difference between dev & ops a four-letter word?

5. Why do you love data?

Why is data scientist your chosen career path?

This is an open ended question, but should spark some stories. Perhaps the candidate enjoys working with tech, product & biz-ops equally? Why are your skills uniquely suited to the role over other technical careers?

Also: Is the difference between dev & ops a four-letter word?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Why you need a performance dashboard like StackExchange

stackexchange

Most startups talk about performance crucial. But often with all the other pressing business demands, it can be forgotten until it becomes a real problem.

Flipping through High Scalability today, I found a post about Stack Exchange’s performance dashboard.

Join 28,000 others and follow Sean Hull on twitter @hullsean.

The dashboard for Stack Exchange performance is truly a tectonic shift. They have done a tremendous job with the design, to make this all visually appealing.

But to focus just on the visual aesthetics would be to miss many of the other impacts to the business.

1. Highlight reliability to the business

Many dashboards, from Cacti to New Relic present performance data. But they’re also quite technical and complicated to understand. This inhibits their usefulness across the business.

The dashboard at Stack Exchange boils performance down to the essentials. What customers are viewing, how quickly the site is serving them, and where bottlenecks are if any.

Also: Is the difference between dev & ops a four-letter word?

2. What’s our architecture?

Another thing their dashboard does is illustrate their infrastructure clearly.

I can’t count the number of startups I’ve worked at where there are extra services running, odd side utility boxes performing tasks, and general disorganization. In some cases engineering can’t tall you what one service or server does.

By outlining the architecture here, they create a living network diagram that everyone benefits from.

Related: Is automation killing old-school operations?

3. Because Fred Wilson says so

If you’re not convinced by what google says, consider Fred Wilson who surely should know. He says speed is an essential feature. In fact *the* essential feature.

The 10 Golden Principles of Successful Web Apps from Carsonified on Vimeo.

Read: Do managers underestimate operational cost?

4. Focus on page loading times!

If you scroll to the very bottom of the dashboard, you have two metrics. Homepage load time, and their “questions” page. The homepage is a metric everyone can look at, as many customers will arrive at your site though this portal. The questions page will be different for everyone. But there will be some essential page or business process that it highlights.

By sifting down to just these two metrics, we focus on what’s most important. All of this computing power, all these servers & networks are all working together to bring the fastest page load times possible!

Also: Is the difference between dev & ops a four-letter word?

5. Expose reliability to the customer

This performance page doesn’t just face the business. It also faces the customers. It lets them know how important speed is, and can underscore how serious the business takes it’s customers. Having an outage or a spike that’s slowing you down. Customers have some transparency into what’s happening.

Also: Is the difference between dev & ops a four-letter word?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

If you’re building a startup tech blog you need to ask yourself this question

Editor & writer in friendly dialog

Join 28,000 others and follow Sean Hull on twitter @hullsean.

I work at a lot of startups, and these days more and more are building tech blogs. With titles like labs or engineering at acme inc, these can be great ways to build your brand, and bring in strong talent.

So how do we make them succeed? It turns out many of the techniques that work for other blogs apply here, and regular attention can yield big gains.

1. Am I using snappy headlines?

Like it or not we live in a news world dominated by sites like Upworthy, Business Insider, Gawker & Huffpo. Ryan Holiday gained fame using a gonzo style as director of marketing at American Apparel. Ryan argues that old-style yellow journalism is back with a vengence.

Click bait asside, you *do* still need to write headlines that will click. What works often is for your title to be a little sound bite, encapsulating the gist of your post, but leaving enough hook that people need to click. Don’t be afraid to push the envelope a bit.

Also: Which tech do startups use most?

2. Line up those share buttons & feedburner

Of course you want to make the posts easy as hell to share. Cross posting on twitter, linkedin, facebook and whereever else your audience hangs out is a must. Use tools like hootsuite & buffer to line up a pipeline of content, and try different titles to see which are working.

You’ll also want to enable feedburner. Some folks will add your blog to feedly. Subscriber counts there can be a good indication of how it is growing in popularity too.

Related: Do today’s startups assemble software at their own risk?

3. Watch & listen to google analytics

You’re going to keep an eye on traffic by installing a beacon into your page header. There are lots of solutions, GA being the obvious one because it’s free. But how to use it?

Ask yourself questions. Who are my readers? Where are they coming from? How long do they spend on average? Do some pages spur readers to read more? Is there copy that works better for readers? Are my readers converting?

It’ll take time if you’re new to the tool, but start with questions like those.

Read: Is automation killing old-school operations?

4. Optimize your SEO a little bit

Although you don’t want to go overboard here, you do want to pay some attention. Using keyword rich titles, and < h2 > tags, along with wordpress SEO plugins that support other meta html tags means you’ll be speaking the language search engines understand. Add tags & categories that are relevant to your content.

Don’t overdo it though. Stick to a handful of tags per post. If you add zillions with lots of word order combinations & so forth, this kind of stuff may tip of the search engines in ways that work against you.

Check out: How to hire a developer that doesn’t suck

5. Search for untapped keywords

When I first started getting serious about blogging, I had an intern helping me with SEO. She did some searching with the moz keyword research tools and found some gems. These are searches that internet users are doing, but for which there still is not great content for.

For example if results showed “cool tech startups in gowanus brooklyn” had no strong results, then writing an article that covered this topic would be a winner right away.

These are big opportunities, because it means if you write directly for that search, you’ll rank highly for all those readers, and quickly grow traffic.

Read also: 5 things toxic to scalability

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Is Zero downtime even possible on RDS?

amazon rds mysql

Join 29,000 others and follow Sean Hull on twitter @hullsean.

Oh RDS, you offer such promise, but damn it if the devil isn’t always buried in the details.

Diving into a recent project, I’ve been looking at upgrading RDS MySQL. Major MySQL upgrades can be a bit messy. Since the entire engine is rebuilt, queries performance can change, syntax can break, and surely triggers & stored procedures can have problems.

That’s not even getting into it with storage engines. Still have some tables on MyISAM? Beware.

The conclusion I would make is if you want zero downtime, or even nearly zero, you’re going to want to roll your own MySQL on EC2 instances.

Read: Why high availability is so very hard to deliver

1. How long did that upgrade take?

First thing I set out to do was upgrade a test instance. One of the first questions my client asked, how long did that take? “Ummm… you know I can’t tell you clearly.” For an engineer this is the worst feeling. We live & die by finding answers. When your hands are tied, you really can’t say what’s going on behind the curtain.

While I’m sitting at the web dashboard, I feel like I’m trying to pickup a needle with thick leather gloves. Nothing to grasp here. At one point the dashboard was still spinning, and I was curious what was happening. I logged out and back in again, and found the entire upgrade step had already completed. I think that added five minutes to perceived downtime.

Sure I can look at the RDS instance log, and tell you when RDS logged various events. But when did the machine go offline, and when did it return for users? That’s a harder question to answer.

Without command line, I can’t monitor the process carefully, and minimize downtime. I can only give you a broad brush idea of what’s happening.

Also: RDS or MySQL 10 use cases

2. Did we need to restart the instance?

RDS insists on rebooting the instance itself, everytime it performs a “Modify” operations. Often restarting the MySQL process would have been enough! This is like hunting squirrels with a bazooka. Definitely overkill.

As a DBA, it’s frustrating to watch the minutes spin by while your hands are tied. At some point I’m starting to wonder… Why am I even here?

Related: Howto automate MySQL slow query analysis with Amazon RDS

3. EBS Snapshots are blunt instruments

RDS provides some protection against a failed upgrade. The process will automatically snapshot your volume before it begins. That’s great. If I spend

See also: Is Amazon RDS hard to manage

4. Even promoting a read-replica sucks

I also evaluated using a read-replica. Here you spinup a slave first. You then upgrade *THAT* box to 5.6 ahead of your master. While your master is still sending data to the slave, your downtime would in theory be very minimal. Put master in read-only mode, wait few seconds for slave to catchup and switch application to point to slave, then promote it!

All that would work well with command line, as your instances don’t restart. But with RDS, it takes over seven long minutes!

Read this: 5 Reasons to move data to Amazon Redshift

5. RDS can upgrade to MySQL 5.6!

MySQL 5.6 introduced a new timestamp datatype which allows for fractional seconds. Great feature, but it means the on-disk datastructures are different. Uh oh!

If you’re doing replication with MySQL 5.5 to 5.6 it will break because the rows will flow out in one size, and break the 5.6 formatted datafiles! Not good.

The solution requires running ALTER commands run on the master beforehand. That in turn locks up tables. So it turns out promoting a read-replica is a non-starter for 5.5 to 5.6. Doesn’t really save much.

All of this devil in the details stuff is terrible when you don’t have command line access.

Read: Are SQL databases dead?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Is there a devops talent gap?

jenkins_docker

New programming languages & services are being invented at a staggering pace. Hosting is changing, networking is changing, race to market is quickening.

But what does all of this mean to the search for talent? Who understands all these components? Who is an expert in any one?

Join 28,000 others and follow Sean Hull on twitter @hullsean.

1. That new car smell

We all remember that time. You know when you drove out of the dealership with your brand new wheels. Driving down the road, you feel on top of the world. You start dreaming of all the fun times you’ll have in your new car. For days and weeks afterward you walk out to your car, open the door & sit inside. It all feels special. You kind of hang out there for a few minutes enjoying the smell before you drive off. Right?

Let’s be vigilant to remember the same thing happens, or rather is happeing in technology all the time. As we automate our infrastructures with Ansible, Puppet & Chef, deploy continuous integration with Jenkins, Travis or Codeship, we should give pause. Each of these tools has it’s own syntax, it’s own bugs, it’s own community, it’s own speed of development & change, it’s own life.

Also: Does a four letter word divide dev & ops?

2. A lot of rushing

Google tells me the synonyms of agile are, nimble, lithe, supple & acrobatic. So in a fast moving world it’s no wonder agile is so big. Anything that allows us to respond to customers quicker & evolve our product faster is a good thing. Yes it is.

Over the years I’ve worked with a lot of clients & customers. Some right out of the gates are in a hurry. There is a sense of urgency even from the initial meeting. Although not in every case, sometimes these are the sign of the perpetually late. They end up throwing money around, throwing technology around, and all in a desperate attempt to plug a leaking ship.

In our race to automate & remain agile & nimble, we should also consider the future. Lets attempt to find a balance & consider future implications of technology decisions & choices.

Read: Is automation killing old-school operations?

3. Hosting, what’s that?

For many of the startups I work with today, they’ve never deployed on anything but Amazon. There was no rack of computers in a closet & a T1 line, circa 1997. There was no rackspace hosted servers or a colo in New Jersey circa 2005. Right from the beginning it was all on-demand computing.

This shift has surely brought a lot of benefit. But no one can argue it isn’t still very new. And with newness there is a learning curve. And bugs & surprises.

Related: Does a devop need to practice the art of resistance?

4. More complexity in troubleshooting

The wild ride really begins when you’re troubleshooting performance problems. Running your database on RDS you say? How the heck do I get to the terminal and run “top”? Can I do an iostat?

And what does iostat output really mean in multi-tenant Amazon, where your disk is an EBS volume across an unknown & unfriendly network. Who knows why it just slowed to a crawl, then sped up dramatically a few minutes later.

Even fetching the relevant logfile can be complicated. For all the problems the cloud eliminates, it sure introduces a few of it’s own. And who is the expert, and how to find them?

Read this: When fat fingers take down your database

5. More tech, fewer experts

I asked the question a few weeks back Do todays startups require assembly of a lot of parts that no one really understands?.

I’ve taken to browsing the stacks at the lovely StackShare site lately. There you can see what some of the top startups are using for their technology stacks. Docker, Yammer, Yelp, Stripe, Vine, Spotify & Stack Overflow are all there today.

There are new message queues like NSQ & programming languages like Markdown, Coffeescript & Clojure. Even Java. Are people still building web apps in Java. No please no!

While it’s wonderful to see such an explosion of innovation, I look at this from an operations perspective. In five years, when the first & second wave of developers at your startup have left, picture yourself trying to find talent in a long since out-of-fashion language like Dart or Swift. What’s more how do you untangle the mess you’ve now built?

Check this: Is the SQL database dead?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Best Howto posts on Scalable Startups

vagrant logo

Join 28,000 others and follow Sean Hull on twitter @hullsean.

MySQL slow query on RDS

If you run MySQL as your backend datastore is there one thing you can do to improve performance across the application?.

Those SQL queries are surely key. And the quickest way to find the culprits is to regularly analyze your log. I’ve put together a howto & script for doing this on Amazon RDS.

Automate mysql slow query analysis on Amazon RDS

Vagrant & Amazon EC2

Among the automation tools that are getting a lot of attention these days are Chef, puppet & ansible for devops, plus Docker & Vagrant.

Can we use vagrant to spinup EC2 instances? As it turns out yes. And it can be a great way to automate dev environments and works in conjunction with docker.

How to deploy on amazon EC2 with vagrant

Cache websites for speed

Is Fred Wilson right that speed is an essential feature? We certainly think so.

And besides tweaking & tuning the database, the next best way is caching. You cache objects at the browser, add a page cache and memcache, redis or elasticache. Here’s our howto.

5 tips to cache websites & boost speed

DB Change Management

Everyone uses version control for application code, whether it’s PHP, Ruby or Node.js. But are you using it for database changes?

DDL, those statements that create objects should also be included in version control. But how to do it properly? Database change management is one part art, but there are some helpful tools to get you on the right track.

With some luck you’ll be able to roll forward & backward to versions of your database schema just as easily as you can versions of your software.

5 tips better db change management

MySQL Scalability

MySQL is the big bad beast that still hobbles a lot of site performance. Here are some key tips, narrowed down to just the essentials.

5 ways to boost mysql scalability

Cloud Scalability

The cloud enables scalability, but is it ready out of the box? There are some key things to remember on your road to high scalability in the cloud.

3 ways to boost cloud scalability

Fortify MySQL Replication

MySQL replication is pretty awesome for what it is. Still there are gotchas & potholes. Here’s our guide to smooth sailing.

5 ways fortify mysql replication

MySQL replica with Hotbackup

If you’re building your own MySQL instances on EC2, you’ll also build your own replicas. Luckily there are some great tools that make this reliable & smooth. Install percona’s hotbackup tool & you’re off to the races.

Easier mysql replication using hotbackups

MySQL Backups

If you’ve forgotten all about backups since your cloud or managed solution does all that for you, think again! There are still things you should do in addition. At the very least run a fire drill & find out if all the parts are there for rebuild.

10 things to remember with mysql backups

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters