Category Archives: Startups

What’s the luckiest thing that’s happened in your career?

sashi [via Flickr]

I was browsing through Career Dean recently, a site that facilitates professionals to share knowledge & experience with more junior & recent college grads about the work world. It’s a great site. I saw the question What’s the luckiest thing that’s ever happened for your career?

I read in John Adam’s AMA his “million dollar piss” (www.careerdean.com/q/howd-you-get-the-job-twitter), which he sowed the seeds of his success basically during a piss. That’s a 1 in a million kind of story I know. I’d like to hear if anyone else has ever experienced anything remotely lucky in that way? =) something fun to come back and read if anyone answers.

Join 28,000 others and follow Sean Hull on twitter @hullsean.

Here’s how I responded…

I moved to NYC & worked at a tiny startup in the mid-nineties. Got to do Mac stuff, windows & Sun Solaris unix as well. Jumped on an Oracle project where I was a bit underwater. The firm hired a consultant to assist me for a few days. I watched what he did and learned like a sponge. Within a few months I dove into Oracle consulting and never looked back.

I felt this was an amazingly lucky opportunity to for a few reasons.

1. DIY

I’ve been consulting for almost twenty years now. And I get asked all the time how to get into freelance or independent consulting. For me the jumping off point was working for a really small ten person startup.

An environment like this is very different from a large corporation where you do one thing. At a tiny shop, everything is very do-it-yourself. You have to be self-serve & lean. It’s a constant challenge to teach yourself what you don’t already know. It’s a very vibrant environment as you enter your career.

Also: 5 Things toxic to scalability

2. Generalist

I also found that I had the chance to really apply everything I learned in computer science. It’s a hardware problem? It’s a software problem? These kind of silos that you experience at university don’t apply. One day you can be doing windows, mac, or Unix operating system configuration, the next you can be writing code. And on the third day you can be doing dba work.

In today’s terminology, this role was site reliability engineer or SRE, fullstack developer, tech support, evangelist, CTO, DBA, scalability & performance lead and more.

Related: Are generalists better at scaling the web?

3. Cutting edge

Startups to be sure are on the bleeding edge. They’re constrained by budgets, and through sheer will & experimentation, are cutting their teeth on the newest technologies out there.

These days that might be Cassandra & Kafka, Docker, MongoDB, hdfs, Redshift and so on.

Read: Do managers underestimate operational cost?

4. Ok to Fail

In larger enterprises, a lot of politics weigh on decisions, and exotic technologies are risky. When you’re at a startup, and by design you are entering uncharted waters, it’s sort of a given that it is ok to fail. This encourages learning, as there is less risk of failure.

Also: Is the difference between dev & ops a four-letter word?

5. Iterative & Agile

We talk about being agile, and lean at startups. At a very small place like this, you have one or two developers, and you deploy code constantly. It’s agile by default. And that’s a good thing.

Also: Is high availability overrated? The myth of five nines.

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

The chaos theory of cloud scalability

The.Rohit - Flickr

The.Rohit – Flickr

Reading Benedict Evans weekly newsletter, you’re bound to bump into something new & useful. His newsletter covers Mobile, but that also means it touches on a lot of other areas of tech, innovation & startups.

This week he pointed me to A Weissman’s The Chaos Theory of Startups. He argues a VC’s job is to help a startup identify the right framework. It’s about finding the signal in the noise.

Join 29,000 others and follow Sean Hull on twitter @hullsean.

I think you can carry this idea over to technical operations today. There are a few key maxims I follow to keep you on the scalable track.

1. Degrade gracefully

You’ve heard it before, but have you done anything about it?

Build a read-only or browse only mode into your application. Do it now. You will thank me. When your database goes down unexpectedly (with RDS this might happen sooner than you think), you want to be able to use your lovely read-only slave database. Browse only mode, forces developers to add read-only support in most application functions, keeping the site up and running, without a full visible and ugly outage.

Which brings me to point two, be sure to have copies of your production database. Real live, only read-only copies. In Amazon speak, this is a read-replica, in MySQL this is a slave database. Most startups I see these days have this, but if you’re one of the ones dragging your feet, do this now.

Also: Is the difference between dev & ops a four-letter word?

2. Monitor & measure

Amazon’s cloudwatch is fine for what it is, and so is New Relic. But employing a dedicated tool just for monitoring, such a Nagios & cacti can give you much more granular intelligence about what’s happening with your infrastructure. Nagios gives you the monitoring & alerting, Cacti gives you the history. It’s like a BI reporting tool for infrastructure.

Related: Is automation killing old-school operations?

3. Keep components simple

Keep it simple stupid? Don’t adopt new technologies, languages, or versions of software, without first vetting them. Ask questions:

o Is there an existing piece of software or service that can overlap this new one, killing two birds with one stone?
o Does everybody know this new technology?
o Does this choice of technology solve any other broad problems we have?
o Is there a large community around the project?
o Are there a lot of engineers with experience in this chosen technology?

Tellingly, many startups don’t have an operations person to start with. In those, the danger is developers choose new solutions, with no push back.

I asked… Does a four letter word divide dev and ops?

Read: Do managers underestimate operational cost?

4. Don’t force database abstraction

Object Relational Modelers, aka database middleware, are great in theory. We want a library that takes database & SQL drudgery away from developers. Why reinvent the wheel?

The trouble is database independent code doesn’t work, and never has. ORMs are painfully inefficient, selecting all columns, or repeatedly reading rows from tables. This causes serious traffic jams inside your database.

They come in various guises, Cake PHP, Active Record for Ruby, Hibernate for Java, SQL Alchemy for Python.

Also: Is the difference between dev & ops a four-letter word?

5. Be asynchronous

This means don’t make your application code wait. Make asynchronous calls to APIs & check back later, use software queues so traffic backups don’t clog your components & communication.

Avoid any type of two-phase or multi-phase commit. These are common in clustered databases, forcing a serialization point so nodes can agree on what data looks like. However they’re toxic to scalability. Look for technologies that use eventually consistent algorithms.

Also: Is the difference between dev & ops a four-letter word?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

If you’re building a startup tech blog you need to ask yourself this question

Editor & writer in friendly dialog

Join 28,000 others and follow Sean Hull on twitter @hullsean.

I work at a lot of startups, and these days more and more are building tech blogs. With titles like labs or engineering at acme inc, these can be great ways to build your brand, and bring in strong talent.

So how do we make them succeed? It turns out many of the techniques that work for other blogs apply here, and regular attention can yield big gains.

1. Am I using snappy headlines?

Like it or not we live in a news world dominated by sites like Upworthy, Business Insider, Gawker & Huffpo. Ryan Holiday gained fame using a gonzo style as director of marketing at American Apparel. Ryan argues that old-style yellow journalism is back with a vengence.

Click bait asside, you *do* still need to write headlines that will click. What works often is for your title to be a little sound bite, encapsulating the gist of your post, but leaving enough hook that people need to click. Don’t be afraid to push the envelope a bit.

Also: Which tech do startups use most?

2. Line up those share buttons & feedburner

Of course you want to make the posts easy as hell to share. Cross posting on twitter, linkedin, facebook and whereever else your audience hangs out is a must. Use tools like hootsuite & buffer to line up a pipeline of content, and try different titles to see which are working.

You’ll also want to enable feedburner. Some folks will add your blog to feedly. Subscriber counts there can be a good indication of how it is growing in popularity too.

Related: Do today’s startups assemble software at their own risk?

3. Watch & listen to google analytics

You’re going to keep an eye on traffic by installing a beacon into your page header. There are lots of solutions, GA being the obvious one because it’s free. But how to use it?

Ask yourself questions. Who are my readers? Where are they coming from? How long do they spend on average? Do some pages spur readers to read more? Is there copy that works better for readers? Are my readers converting?

It’ll take time if you’re new to the tool, but start with questions like those.

Read: Is automation killing old-school operations?

4. Optimize your SEO a little bit

Although you don’t want to go overboard here, you do want to pay some attention. Using keyword rich titles, and < h2 > tags, along with wordpress SEO plugins that support other meta html tags means you’ll be speaking the language search engines understand. Add tags & categories that are relevant to your content.

Don’t overdo it though. Stick to a handful of tags per post. If you add zillions with lots of word order combinations & so forth, this kind of stuff may tip of the search engines in ways that work against you.

Check out: How to hire a developer that doesn’t suck

5. Search for untapped keywords

When I first started getting serious about blogging, I had an intern helping me with SEO. She did some searching with the moz keyword research tools and found some gems. These are searches that internet users are doing, but for which there still is not great content for.

For example if results showed “cool tech startups in gowanus brooklyn” had no strong results, then writing an article that covered this topic would be a winner right away.

These are big opportunities, because it means if you write directly for that search, you’ll rank highly for all those readers, and quickly grow traffic.

Read also: 5 things toxic to scalability

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Best of Startup Content on Scalable Startups

strawberries

Join 28,000 others and follow Sean Hull on twitter @hullsean.

Costs

Costs of techops can involve short-term architectural, decisions, but what about the longer term affects of choices? Do cto’s underestimate operational costs?

A stack of…

These days the full stack of a internet or mobile startup involves a lot of varied components, from Chef, Puppet & Ansible, to Nginx, haproxy, redis, solr and some database like MySQL or Postgres on the relational end of the spectrum, or Mongodb, Hbase or Cassandra on the NoSQL side. What type of challenges does this pose to a team? I’m curious,
Do startups assemble at their own risk?

Most used tech

Leo Polovets ran some stats over the Angellist data of startups. He wanted to know Which tech do startups use most? and I summarized the results.

Death of ops?

These days with all the talk of automation, I’ve heard heard developers & even CTO’s argue of a diminishing need for backend administrators. Do startups still need techops?

Speed as a feature

Is Fred Wilson right to say speed is a feature? What does this mean for those migrating or already running in the cloud? How does scalability come into play?

Avoiding outages

Are many outages avoidable? Did Airbnb have to fail?

Performance Review

Reviewing architecture & site speed is a type of engagement that a lot of startups can benefit from. Here’s my Anatomy of a performance review.

Let things fail

Does it sometimes make sense to let things break a little? A tale of managed failure.

Young founders

I worked at one startup with a CTO just out of college. Although they were flush with cash & had real problems scaling, communication problems ultimately soured the engagement. Are you too young to be a boss?

80 million fix

Sometimes fixing serious performance bottlenecks can get a site back up on it’s feet. In this success story they went on to get acquired weeks after the fix. In tongue in cheek fashion I askWhere’s my 80 million dollars?

CTO’s should never do

There are times to get into the trenches. But what if it sacrifices leadership?What should a CTO never do?

Startups too cool for school

Joining YC but have no ideas? No problem. Is my startup too cool for your business school?

Instant business, just add water

Can a business be built in just a weekend? Is there a problem with startup bootcamps?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Today’s startups: assemble at your own risk

devops divide

I was talking with Todd Hoff recently over at High Scalability about a trend I’ve seen of late.

ME: I really liked this post by Zoli Kahan from Clay.io.  AWS, cloudflare, docker, haproxy, mysql, mongo, memcache, ansible.  They use just about every technology being talked about these days.  

Todd: Yah, that’s why I asked to republish it. I thought it was a good updated sampler stack.

ME: That said I defy you to find a team that actually *KNOWS* all those technologies.  

Todd: Agreed. Systems are a lot of assembly these days, which doesn’t mean we know how to build the parts being assembled.

Join 28,000 others and follow Sean Hull on twitter @hullsean.

The article I was referring to was: How Clay.io Built their 10x Arch Using AWS, Docker, HAProx & Lots More

1. Dizzying array of technologies in use

I’ve been working with startups since the mid-nineties. In those days most application stacks consisted of a PHP application running on Apache, with Oracle on the backend. Both webserver & db ran on Sun Solaris. Hardware was reliable. Most attention was focused on fitting everything in memory, and monitoring the servers for swapping, and disk failure. Boy have those days changed.

I see dozens of startups each year, so I see a lot of very cutting edge environments. Here’s a peak at what I’m seeing these days:

Database: MySQL, Postgres & Oracle, to Mongodb, Cassandra & Couchbase

Caching: Memcache or Redis

Search: Solr

Webservers: Apache, Nginx, Lighttpd

Load balancers: haproxy, Zen

Languages: PHP, Python & Ruby

Publishing: Drupal, WordPress, Joomla

Continuous Integration: Jenkins

Metrics: Cacti, collectd, NewRelic

Monitoring: Nagios, Ganglia, Munin, OpenNMS

Automation: Ancible, Chef, Puppet, Docker & Vagrant

Logs: Logstash

DDOS & CDN: Cloudflare, Ultradns

Whew… That’s a long list!! And we’re not even considering the API’s that many applications are now building on.

Also: Are generalists better at scaling the web?

2. Shortcuts abound

Startups early on, don’t have enough working capital to hire a huge engineering team. So that means everyone is stretched. With a list of technologies that is ever growing, something’s gotta give.

These may cut corners by handing the web & technical operations work to a developer who has some skills. But I continue to ask… Does a four-letter word divide dev & ops?

Read: Which tech do startups use most?

3. More things to break & master

Ownership of a software stack, such as a database means mastery of…

o features in current versions
o bugs of current versions
o vulnerabilities of various versions
o troubleshooting
o best practices
o backup & reliability

For example a lot of shops where I dig into the database, I find low hanging fruit, such as misconfigured startup settings, table layout or index usage.

I see similar things when a networking expert pours over the haproxy configuration, or runs ping tests across the network. Most of these components are setup with fairly vanilla configurations, leaving loose ends and frayed threads.

Check out: Why I can’t raise the bar at every firm

4. Many startups carrying technical debt

I’ve seen a growing reliance on ORM’s which is worrying. Build your foundation on a crutch, and it gets very hard to eliminate down the line. Here are Ward Cunningham’s warnings on technical debt.

Related: Are SQL Databases Dead?

5. Long term support & viability

At one five year old firm, I was brought in to address scalability problems. I met with the team and was asked to provide a comprehensive review. The first thing I found was all the original engineers had long since left, so the code was new for everyone. As I dug my heels in, I found multiple versions of Apache along with Nginx on some other servers. Their stack was built on a patchwork of Python, Ruby & PHP. Then digging in further, we found a complicated web of dependencies for digital assets, mounted across servers & unmonitored.

Lack of standards is common in environments like these. Without an operational or architectural lead, developers are left to make decisions with what is directly in front of them. Though a decision of what language to use may appear simple at the outset, it carries long term consequences.

Will that language or technology be supported in five years? Will the community survive? Will your firm be able to hire people with that skill set? Will engineers still be excited about it?

See also: Is high availability overrated? Is five nines a myth?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

When prospects mislead

MUHAMMAD ALI ROCKS GEORGE FOREMAN ON THE JAW

While a story is fresh in ones mind, it’s a great time to tell it. And so I set out to putting pen to paper about a recent consulting war story.

Join 28,000 others and follow Sean Hull on twitter @hullsean.

A financial services firm reached out to me, asking about services. We discussed the project plan, and the day after the call I sent along a quote. I suggested three options, a weekly fee, a monthly one, or monthly with advance payment.

They decided to go with option C, and we arranged a kickoff meeting.

1. Level setting on trust

I’ve done this kind of work for so long, and worked with so many clients over the years, that it sometimes becomes second nature. I arrived, and we chatted amicably. I asked him about his wikipedia page, which he seemed excited to talk about.

I was surprised that there wasn’t a check ready, as we had decided on advanced payment in full, but didn’t make a mention right away. He then tried to dial in his partner, but that just went to voicemail. So we continued the meeting without him.

I don’t know how important the meeting was to both team members, but they were both on the invite & emails. His partner never called back through the meeting either.

Read this: When migrating from Oracle to MySQL Prepare to Bushwack

2. Negotiations is part art & dance

Interestingly I had met up with some colleagues the night before over italian food. I mentioned I was meeting a new prospect the next day, but had reservations about whether they had really decided to hire me, or were just still prospecting.

So during the meeting I was somewhat conscious of that question. Are we already in exploratory, discovery mode? Has the project even begun? That’s a question, and from what I sensed it was still an open one.

As the meeting wore on, questions about oracle licenses, versions, and EC2 configurations came up. Furious note taking continues.

Related: Which tech do startups use most?

3. Time & mismanagement

One thing that comes up for me in these situations is questions of time management. In order to work with a new client, I must clear my schedule, and make time available. That has a value to start with. When it turns out a project isn’t actually ready yet, it becomes an awkward stumble out of the gates.

Also: Is automation killing the sysadmin job role?

4. Can you research this one thing

As I raised various concerns about Oracle, the data loader portion, and unknowns around how that software worked, the prospect asked if I could do a little research for them.

This is where things started to crack. Rather than answer the question, I made a more aggressive nod to the question on my mind: Have we really started on this project yet? I explained that I was confused, and gathered from our email this this was a kickoff meeting. The tension in the air rose noticeably.

He then explained “Well we’re still waiting to hear back from a vendor about XYZ”. From there I began to gather up my things.

Check this: What can fashion week teach Chad Dickerson about Net Neutrality?

5. Watch out for those Rothkos

As I stand up I comment on the digs. “Is this shared office space, those look like Rothkos?” I ask. “Nope this is all ours, my wife is a collector & art dealer. We have some real Warhol’s too”. “Wow…”, I respond, “tough business to be in!”. With that he says “Well it is very volatile, we can be out of business in a month.”

My take away here isn’t to be wary of all new prospects. Each person or business has their own *style* of doing business. Rather, until you’ve established trust with a new client, consider that you may not yet be working on the project at all.

And with that the dance continues. While you may wish to demonstrate and illustrate your knowledge, and the solutions you’d recommend, beware of solving the problem before you’re even hired!

Read: Are SQL Databases Dead?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

What can NYFW teach Chad Dickerson about net neutrality?

net neutrality

Here we are again discussing Net Neutrality… Chad Dickerson CEO of well renowned Etsy.com, has come out strongly in favor, and wants everyone to take action.

Join 27,000 others and follow Sean Hull on twitter @hullsean.

Honestly when I read his wired piece Etsy CEO to businesses: If Net Neutrality Perishes, We Will Too, I was struck by one statement:

The FCC proposal will threaten *ANY* business that uses the internet to reach it’s customers.

Any business? Quite a sweeping statement. Strikes fear into me that’s for sure… And if you read through the comments, the debate is equally fierce. One side says net neutrality is socialism! The other side says anyone against net neutrality is a shill for Comcast or Verizon! Battle lines drawn!

1. Are all businesses at risk?

Isn’t the idea that ETSY will perish overstated? Are they a high bandwidth company? Are they trying to stream video?
Is the entire Etsy community alarmed? Isn’t that a rather broad statement?

To be sure ending net neutrality will impact some businesses. Perhaps one reason VC’s like Fred Wilson are so concerned about Net Neutrality isn’t for the freedom of millions of internet users, but the threat to disruptive businesses, the startups that VC’s directly invest in.

Read: Which tech do startups use most?

2. Will all internet users be impacted?

Here again some of this debate seems overstated. I remember using the internet on a dialup modem. 300 baud, was about the speed at which you can type. Then along came 14.4, 28k and upward speeds climbed. All the while the internet was usable. Could I do all the things I can today, nope.

Even if these horrible Comcast’s & Verizon’s reduce speeds by 100 times, they will still be plenty fast for most internet users. Sure streaming video would be impacted, and yes streaming music would be impacted. But for end users, I would argue most would not be impacted. It is rather the disruptive startups & businesses that would be most impacted.

Also: Is automation killing old-school operations?

3. Are there anti-EDU parallels

In the mid-nineties, before the dot-com bubble, there was a huge raging debate about even having commercial entities on the internet at all. Enlightened internet cognoscenti considered it an abomination.

But the real world pushed it’s nose in, and today we take as a given.

Check this: Is Hunter Walk right about operations & startups?

4. Is google right about millisecond delays?

“Research from Google & Microsoft shows that delays of milliseconds result in fewer page views and fewer sales in both the short & long term”. Yep, that’s a fact. The research shows this. But what do we take away from that?

As a performance and scalability consultant I see a *TON* of websites that have huge delays, well over tiny millisecond ones that Google frets over. Internet startups struggle with performance every day.

What’s the irony? Slowdowns that Comcast or Verizon might introduce to end users pale in comparison with these larger systemic problems.

Also: 5 Ways startups misstep on scalability

5. Any lessons from sites of New York Fashion Week?

I like the Pingdom speed test tool. I used it to track the speed of some of the websites & blogs that are big for NYFW. Here’s what I found:

nyfw speed test results

What do you see? Take a look at the SIZE column. Notice something strange? The LARGEST sites, in terms of images, css & assets aren’t necessarily the SLOWEST! That’s a funny result if you consider net neutrality. If you think the network speed is the same for all websites, shouldn’t the smallest pages load fastest?

Not true at all. It’s a very simplistic way of viewing things. Fashionista.com for example is doing a ton of tuning behind the scenes. As you can see it is making their site far and away the fastest! Network bandwidth and net neutrality be damned!

Related: Are SQL Databases Dead?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Which tech do startups use most?

MySQL on Amazon Cloud AWS

Leo Polovets of Susa Ventures publishes an excellent blog called Coding VC. There you can find some excellent posts, such as pitches by analogy, and an algorithm for seed round valuations and analyzing product hunt data.

He recently wrote a blog post about a topic near and dear to my heart, Which Technologies do Startups Use. It’s worth a look.

One thing to keep in mind looking over the data, is that these are AngelList startups. So that’s not a cross section of all startups, nor does it cover more mature companies either.

In my experience startups can get it right by starting fresh, evaluating the spectrum of new technologies out there, balancing sheer solution power with a bit of prudence and long term thinking.

I like to ask these questions:

o Which technologies are fast & high performance?
o Which technologies have a big, vibrant & robust community?
o Which technologies can I find plenty of engineers to support?
o Which technologies have low operational overhead?
o Which technologies have low development overhead?

1. Database: MySQL

MySQL holds a slight lead according to the AngelList data. In my experience its not overly complex to setup and there are some experienced DBAs out there. That said database expertise can still be hard to find .

We hear a lot about MongoDB these days, and it is surely growing in popularity. Although it doesn’t support joins and arbitrary slicing and dicing of data, it is a very powerful database engine. If your application needs more straightforward data access, it can bring you amazing speed improvements.

Postgres is a close third. It’s a very sophisticated database engine. Although it may have a smaller community than MySQL, overall it’s a more full featured database. I’d have no reservations recommending it.

Also: Top MySQL DBA Interview questions

2. Hosting: Amazon

Amazon Web Services is obviously the giant in the room. They’re big, they’re cheap, they’re nimble. You have a lot of options for server types, they’ve fixed many of the problems around disk I/O and so forth. Although you may still experience latency around multi-tenant related problems, you’ll benefit from a truly global reach, and huge cost savings from the volume of customers they support.

Heroku is included although they’re a different type of service. In some sense their offering is one part operations team & one part automation. Yes ultimately you are getting hosting & virtualization, but some things are tied down. Amazon RDS provides some parallels here. I wrote Is Amazon RDS hard to manage?. Long term you’re likely going to switch to an AWS, Joyent or Rackspace for real scale.

I was surprised to see Azure on the list at all here, as I rarely see startups build on microsoft technologies. It may work for the desktop & office, but it’s not the right choice for the datacenter.

Read: Are generalists better at scaling the web?

3. Languages: Javascript

Javascript & Node.js are clearly very popular. They are also highly scalable.

In my experience I see a lot of PHP & of course Ruby too. Java although there is a lot out there, can tend to be a bear as a web dev language, and provide some additional complication, weight and overhead.

Related: Is Hunter Walk right about operations & startups?

4. Search: Elastic Search

I like that they broke apart search technology as a separate category. It is a key component of most web applications, and I do see a lot of Elastic Search & Solr.

That said I think this may be a bit skewed. I think by far the number one solution would be NO SPECIFIC SEARCH technology. That’s right, many times devs choose a database centric approach, like FULLTEXT or others that perform painfully bad.

If this is you, consider these search solutions. They will bring you huge performance gains.

Check this: Are SQL Databases Dead?

5. Automation: Chef

As with search above, I’d argue there is a far more prevalent trend, that is #1 to use none of these automation technologies.

Although I do think chef, docker & puppet can bring you real benefits, it’s a matter of having them in the right hands. Do you have an operations team that is comfortable with using them? When they leave in a years time, will your new devops also know the technology you’re using? Can you find a good balance between automation & manual configuration, and document accordingly?

Read: Why are database & operations experts so hard to find?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Is Hunter Walk right about operations & startups?

The.Rohit - Flickr

The.Rohit – Flickr

Join 26,000 others and follow Sean Hull on twitter @hullsean.

Hunter Walk blogged recently about the importance of building great operations teams. And while he was speaking primarily about business operations, the startup technical operations teams are equally difficult to get right.

1. performance & scalability

As your grows like Birchbox, your customer growth curve may begin to look like a hockey stick. That’s a good problem to have. Will your web application be able to keep up with the onslaught of traffic those customers bring?

Getting performance and scalability just right, will mean fewer site crashes during those key moments when all eyes are on your site.

Also: Is top operations talent hard to find?

2. Operations is key to architecture

Developers will always have strong opinions on architecture. However they may be heavily influenced by their own mandate, features, deliverability & deadlines. So it’s no surprise that they may sometimes choose to build on ORM’s, the middleware brought to you by Hibernate, Cake PHP, Active Record & the like.

And while these technologies seem a necessity in todays modern architectures, they play havoc with your long term scalability. Strong technical operations teams mean a better vision in this area. Heading off your reliance on these technologies will mean managing technical debt before it takes down your country.

Read: Are generalists better at scaling the web?

3. Operations informs strategy

Did you build in those operational switches to turn off the heaviest code, when your site gets overloaded? Operations strategy can help you see these problems on the horizon before they overwhelm you.

Have you considered building a browse only mode for your site? If you’ve ever visited Facebook or Yelp after hours you may have been greeted with the message “We can’t save your comments. Please try again later”. A small innocuous message to end users doesn’t disrupt their enjoyment of the site terribly. But from an technical operations perspective it’s huge. It means teams can perform backups, upgrades and maintenance without interrupting day-to-day activity on the site.

Related: Is scalability a big business?

4. Operations means resilience

We only learn real disaster recovery lessons from storms like Sandy. That’s because resilience highlighted best when it is a real & urgent need.

In technical operations, getting backups right & testing your recovery plan all form key steps in your path to excellence. Get them right before you need them, and ensure repeatability.

Read: Is high availability a real possibility?

5. Operations means technical strength

At the end of the day, getting technical operations right, means you can move from strength to strength. It means building on a solid foundation the likes of Google, Facebook, Foursquare & Etsy. It means you can evolve & grow with your customers, and meet their needs confidently.

Check out: Do startup CEO’s underestimate operational costs?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Are startup CEO’s hiding their scalability problems?

Russian_Dolls

Join 27,000 others and follow Sean Hull on twitter @hullsean.

Your site is running fine right? You have 1000 customers, and it usually runs smoothly. Just this one lingering question, why does it take five high performance EC2 instances to run the database, all on flash drives? Goood question!

The truth is one of the highest trafficed sites I managed, pulled in 100 million uniques a month, and only used three backend databases. That site was one of these wildly popular celebrity gossip sites, the ultimate guilty pleasure when you’re at the office and can’t watch reality tv!

Snickers aside, this is huge traffic. And all of the above was built on Drupal, with no ORM in the mix. It could even run, albeit noticeably slower, while memcache was disabled.

1. Servers with solid state drives

I’m very excited to see Amazon introduce servers with SSD drives. They can bring you 100x improvement of disk I/O, and that my friends is the end all and be all for databases. So why complain?

If you deploy on these boxes right out of the gates, it may be like using a crutch. You become dependent on it, and ignore real performance tuning. Solid state drives still won’t obviate that ORM middleware you’re using.

Also: Do managers & CEO’s underestimate operational costs?

2. Memcache saving your bad queries

Memcache is also a powerful tool. It sits between the database and your webservers, reducing load on the database by as much as 10x. That’s a great way to get better response time, and reduce drag on your db tier. But it’s still worthwhile performance tuning without it.

Why? If you can get your site to run without caching, it will run blazingly fast *with* it. Don’t use it as a crutch, use it as rocket fuel for your well tuned site.

Read this: Do startups need techops?

3. A legion of read slaves

I’ve seen smaller sites, using a ton of read slaves. All of it deployed to cover up slow & redundant queries pouring out of an ORM middleware layer, in this case Cake PHP.

Again, read slaves are great, but tune & test with less hardware, and get the performance up the hard way. With elbow grease!

Related: Howto automate MySQL query analysis with Amazon RDS

4. Really really big memory

64G, 128G, 256G of main memory? If I wax on about the days when you’d get excited by 64k, I’ll sound like an old timer. But with those extreme limitations, you had to write tight code. Otherwise it just wouldn’t do anything.

Really really big memory of today’s servers allows us to get lazy. I hear developers say “Hey, the database is 10G of data, and we have 64G main memory, so the whole thing will fit in memory. Problem solved!”

Duhhh… No. Why not? Because you still have to slice and dice that data. You still have to scan through for bits & pieces that aren’t indexed, then sort, and organize that into temporary memory space. In DBA speak, you’re still doing a ton of logical IOs.

Picture it another way, imagine the days when you’re on horseback, riding across the west. You travel light cause frankly your horse can carry only so much. Then along come cars, and you start loading up the trunk. You add the kitchen sign, and the rear tires are hanging on the ground. All seems fine until you hit a steep mountain, and you’re car is almost stalling at 20mph. If you had only carried the same load as you did on horseback, you’d be speeding across the country at lightning pace.

Read: Is Amazon RDS hard to manage?

5. Deploying poor code

Deadlines are looming, and new features must be deployed. So performance testing can wait until later. The code works after all.

Been there, done that. Code gets deployed and all of a sudden there are spikes on server load in the evening. Ops & DBA teams are screaming, “Who wrote this code?”.

Load testing should be a part of everyday QA & test. It’s the only way to avoid growing scalability problems.

Check this: Are SQL databases dead?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters