Hacking Job Search – Three Meaty Ideas

Also find the author on twitter @hullsean.

Demand for talented engineers has never been higher. It is in fact the dirty little secret of the startup industry, that there are simply not enough qualified folks to fill the positions.

What this means for you is that you have a lot of options. What it means for a hiring manager is that you will have to work even harder to find the right candidate. Just going to a recruiter isn’t enough. Use your network, go to meetups, follow Gary’s Guide daily.

Also check out our Mythical MySQL DBA piece where we talk about the shortage of DBAs and operations folks.

Further if you’ve dabbled in freelance or independent consulting, I wrote an interesting an in depth look at Why do people leave consulting. Understanding this can help avoid it in your own career, or avoid your resources leaving for better shores.

Find us on twitter @hullsean and linkedin where we share content and ideas everyday!

1. Build your reputation

As they say, your reputation precedes you. So start building it now. Fulltime or freelance, you want to be known.

Speaking, yes you can do it. Start with some small meetups, volunteer to speak on a topic. A ten person room is easier than 30, 50 or 100. Once you have a couple under your belt, fill out a CFP for Velocity, OSCon or some software developers conference. There are many.

Blog – if you’re not already doing so you should. Start with once a week. Comment on industry topics, controversial ideas, or engineering know-how. Prospects can look at this and learn a lot more than from a business card.

Write a book, yes you can. It may sound impossible, but the truth is that publishers are always looking for technical writers. Pick a topic near and dear to you. It’ll also give you endless material for your blog.

Go to meetups, you really need to be getting out there and networking. Get some Moo Business Cards and start working on your elevator pitch!

Social media – being active here helps your blog, and helps people find you. Twitter is a great place to do this. Interact with colleagues and startup founders, VCs and more. If you’re a hiring manager or CTO, you may find great programmers and devops this way.

We also wrote a more in-depth article Consulting and Freelance 101. It’s a three part guide with a lot of useful nugets.

Also take a look at our MySQL DBA Interview Guide which is as helpful to devops and DBAs as it is to managers hiring them!

Above all else, build your network & your reputation. It will put you in front of more people as a person, not a commodity or a resume in a pile of hundreds.

2. Qualify prospects

You definitely don’t want to take the first offer you get, and managers don’t want to hire the first candidate that comes along. You want two or three to choose from. Best way to do this is to have options.

If you’re a candidate, network or work through your colleagues. When you do get a lead, be sure you’re speaking to an economic buyer. If you’re not you’ll need to try to find that person who actually signs the checks. They are the ones who ultimately make the decision, so you want to sell yourself to them.

Get a Deposit – I know I know, if it’s your first freelance job, you don’t want to scare them off. Or maybe you do? The only prospects that would be scared off by this are ones who may not pay down the line. Dragging their feet with a deposit can also mean bureaucratic red tap, so be patient too.

Sara Horowitz has an excellent book Freelancers bible, we recommend you grab a copy right now!

Commodity You Are Not so don’t sell yourself as one. What do I mean? You are not an interchangeable part. You have special skills, you have personality, you have things that you’re particularly good at. These traits are what you need to focus on. The dime-a-dozen skills should sit more in the background.

You’ll also need to price and package your services. We talked about this in-depth in Consulting Essentials – Getting the Business.

We also think there is a reason Why Generalists are better at scaling the web.

3. Play the numbers game

For hiring managers this doesn’t mean working through recruiters that might be bringing subpar talent, it means networking through industry events, meetups, startup pitch and venture capital events. There are a few every single day in NYC and there’s no reason not to go to some of them.

For candidates, be eyeing a few different companies, and following up on more than one prospect. You should really think of this process as an integral and enjoyable part of your career, not a temporary in between stage. Networking doesn’t happen overnight, but from a regular process of meeting and engaging with colleagues over years and years in an industry.

At the end of the day hiring is a numbers game so you should play it as such. Keep searching, and always be watching the horizon.

Read this far? Grab our Scalable Startups for more tips and special content.

No iPhones Were Harmed in the Creation of this Outage

Apple’s recent iMessage outage had some users confused. What do you mean I can’t text my favorite cat photos?? How can Apple do this to me!?!?

What happened?

Apple provides services to everyone who uses it’s platform. iCloud for example stores your contacts, calendar, photos, apps and documents in the cloud. No more syncing to itunes to make sure all your stuff is backed up. It’s automatic in the cloud. Yes or course unless iCloud is down.

Same goes for iMessage. Apple has quietly introduced this, as a more feature rich version of text messaging. It’s great until the service isn’t available. What gives?

All these services are backed magically or not so magically by computer servers. These computers sit in datacenters, managed by operations teams, and to some degree with automation. All the things that brought down AWS & AirBNB & Reddit with it could also take out Apple. A serious storm like Sandy also presents real risks.

iMessage is a text and SMS replacement service for iPhones & iPads. It is more feature rich, offering device synchronization, group texting & return receipt. But in a very big way it is also an attempt for Apple to muscle into the market and further extend it’s platform reach.

100% uptime ain’t easy

Even for firms that promise insanely good uptime, five nines remains very very hard to achieve in practice.

For starters all the components behind your service, need to be redundant. Multiple load balancers, webservers, caching servers, and of course databases that hold all your business assets.

But as the repeated AWS outages attest, even redundancy here isn’t enough. You also need to use multiple cloud providers. Here you can mirror across clouds so even an outage in one won’t bring down your business.

What about in the world of messaging? Well you can bet your customers don’t likely know or care about high availability, uptime, or any of these other web operations buzzwords. But they sure understand when they can’t use their service. It may give companies like Apple pause as they try to stretch themselves into areas outside their core business of iphones, ipads, and the IOS platform itself.

iMessage – messaging standards power play

When I first upgraded to an iPhone 4S, the first thing I noticed was the light blue bubbles when texting certain people. Why was that, I wondered? I quickly found out about iMessage, which was conveniently configured, to replace my old and trusty text messaging.

Texts or SMS work across all phones, smartphone or not, and apple or not. But open standards don’t lend themselves well to market muscle and dominance. So it makes sense that Apple would be pushing into this space. I met more than one blackberry owner who loved using bbm to keep in touch with colleagues. It’s like your own private club. And that muscle further strengthens Apple’s platform overall. Just take a look at how the Android Ecosystem is broken if you need an example of what not to do.

The flip side is it means you have more to manage. More servers, more services, more dimensions to your business. More frequent outages that can tarnish your reputation.

A lot complaining and publicity like the iMessage outage received, may just be an indication that you’re big enough for people to care.

Alternatives abound…

There is huge competition in the messaging space. The outage and it’s publicity further underline this fact.

For example on the iPhone for messaging there is ChatOn, Whatsapp, LINE, SKYPE & wechat just to name a few.

Interestingly, while researching this article, I downloaded WhatsApp to give it a try. Only 99 cents, why not. Turns out that they had not one, but two outages, just a week ago. Seems Apple isn’t the only one experiencing growing pains.

A lot of complaining and publicity could be a sign that you’re big enough for people to care!

Read this far? Grab our Scalable Startups monthly.

Cloud DBA and Management Interview

What does a cloud computing expert need to know? This is the last of a three part guide to interviewing for a cloud operations position. You can find them here – part one Operations Interview and part two Deployment Interview.

Here’s my guide to do just that.

1. Database administration experience

Although in some shops the DBA role is a completely separate one, there are many others where the Linux and Operations teams manage these services as well. We do have a some other material Oracle DBA Interview questions and MySQL DBA Interview Guide. Here’s a taste of what to expect.

o What is RAID? Which type is best?

RAID is a way to share a whole bunch of disks on one server. Databases like Oracle or MySQL do a lot of writing and reading from disk. If there are more disks sharing this work, it’s like you have more waiters in your restaurant. Faster serivce.

Although some folks still hang onto RAID 5 as an option, it’s generally a very bad one. It has a serious write penalty because of parity checking it must perform. Most databases do a lot of writing, even when user transactions are not doing INSERT or UPDATE. What’s more if a disk fails, RAID 5 although technically online, will be so slow as to be effectively unusable while the long slow rebuild happens.

What’s the answer then? RAID 10! It mirrors each volume, and then stripes across those mirrored sets. Fast I/O, fast recovery. Done & done.

o What are the tradeoffs with more indexes versus fewer?

In all relational databases, you build indexes on data. Indexes are just like the ones you think of in the yellow pages, phonebooks of yore. An index on first name means you can look up Obama by Barack as well. Index on street addresses means you can lookup on the White House. So the more indexes you have, the more different ways you can search for & fetch what you want.

On the other hand the penalty here, is that whenever you add new data & records to this database, all those indexes must be updated. That’s overhead, which slows down writes.

So the tradeoff is more indexes – faster fetching, slower writing. Fewer indexes slower fetching, faster writing.

o What do NoSQL databases eliminate? How do they achieve great speed?

There are quite a few different types of NoSQL databases. So I’m generalizing quite a lot here. One thing NoSQL databases eliminate is the ability to JOIN data across different columns. By removing this great feature of relational databases, they dramatically simplify the underlying implementation. No free lunch!

What else? Many of these databases cut corners on what’s called durability. What is durability? Imagine you are in a lecture hall and bring your notebook or are waiting tables, and taking orders. It might be quicker to do so without writing things down. You keep it all in your head. Great, but what if you forget something? You have to go ask for the order again! It may be faster, but more prone to error. Losing data is not something to be taken lightly. NoSQL databases don’t always flush data to permanent storage.

Whether or not an web operations candidate uses command line may seem like a small issue. But it speaks to what their DNA is, and the strength of their foundation. Strength and comfort on the command line is key.

o What is Amazon RDS? When should I use it?

Amazon has a managed relational database solution called RDS. It’s basically MySQL, Oracle or SQL Server, but modified so you can’t shoot yourself in the foot. Administrative tasks are simplified, but so are your configuration options.

I wrote an in-depth Amazon RDS use cases article. It mostly covers MySQL, but the general rules apply to Oracle & SQL Server. At the end of the data RDS is a lot less configurable and flexible. But if you don’t have a regular DBA on staff, it will probably simplify your administration of these servers.

o What are read-replicas? What about Multi-az?

Read-replicas are read-only copies of your data. Using MySQL these are fairly stock master-slave configurations. Note since they’re the standard technology, they’re still asyncronous. So yes the read-replica can lag behind.

Multi-az is a proprietary technology, and Amazon doesn’t disclose what’s under the hood. However it’s likely running on top of something like DRBD which is a distributed filesystem. This allows the underlying disk I/O to be mirrored across the internet, and to another availability zone. You’ll enjoy syncronous copies of your data, and no data consistency problems. Keep in mind those that the alternate server is offline or cold and can take time to come online.

o What is the primary bottleneck of hosting databases in the cloud? How has Amazon recently addressed this?

As I explained above disk I/O remains the largest bottleneck for relational databases, even if the entire dataset fits in memory. Why? Because sorting, joining, and rearranging data can take orders of magnitude more memory to magically do in memory. And that’s not even talking about durability guarentees.

The cloud has traditionally lagged quite a lot behind physical servers in terms of disk I/O so some internet firms have shyed away from moving to the cloud. EBS volumes were typically limited to a few hundred IOPs.

Amazon’s recently announced Provisioned IOPs. It’s a mouthful of a name for a very big development. It means you can provision how fast you want those virtual disks to be. For individual volumes the limit seems to be 2000 IOPs but you can also software raid across many of those virtual disks. For Amazon RDS the limit is reportedly 10,000 IOPs. This new feature will make a huge difference for hosting large high I/O databases in Amazon’s cloud.

2. Architecture & Management Questions

o Why does the API battle between Amazon & Eucalyptus (FOSS) matter?

As large applications are architected to build hardware components, and resources in the cloud, the API they work through becomes key. Sticking to an open standard for this API means you can change cloud vendors and/or build on multiple ones. We talked about this multi-cloud solution as a key way to avoid outages like AirBNB and Reddit experienced when AWS had an outage.

Following on the heels of that article, we were quoted about multi-cloud by Brandon Butler in his Network World piece .

o Do you use command line tools? Why?

A good web operations candidate should be very comfortable with command line tools. Everything in Linux is command line. It’s like broadway acting to movie acting, or literature to books. It’s the original source, much more powerful, what’s more it indicates and requires much stronger theoretical knowledge of the underlying systems being managed.

o What can go wrong with backups? How do we test them?

Everything can go wrong with them. They can fail to complete. Be backups of the wrong service or resource. Even the backup software itself can have bugs. The only way to sleep well at night is if you run firedrills and restore your application and data top to bottom.

o Should we encrypt filesystems in the cloud? What are the risks?

This depends on your environment and how sensitive your data is. If you’re collecting credit card data for instance, it may be key. However some surprising blips may push other applications to encrypt as well. Bugs in the hypervisor could potentially make your data vulnerable. What’s more if the cloud provider gets subpeonaed, it may well capture your server and data into the net. Better safe than sorry. Remember you don’t know where your data actually resides, but you do control who has access if you’re encrypted.

We wrote a very in-depth piece on Deploying on Amazon EC2 where we discuss questions such as encryption in more depth.
o Should we use offsite backups?

It’s definitely worth doing this. One more layer of insurance.

o What is load balancing? Why is it difficult with databases?

Load balancing puts a digital traffic circle into your infrastructure, giving you two roads or paths to resources. However those resources have to be exactly the same. With databases you are constantly writing to tables, and updating records. When you scale those horizontally, it becomes impossible to keep track of changes.

Relational databases are inherently difficult to scale. Most environments scale a single authoritative master vertically, and add multiple read-only slaves horizontally to allow the appplication to serve more customers.

o Why use a package manager? Can we install from source?

Package managers simplify the installation of software components. A team such as Redhat, Ubuntu or Debian builds a distribution, and compiles all components storing them in a repository. Installing packages this way allows your setup to be standard across servers. This allows more automation, and is simpler for another admin to figure out what you have, down the line when it passes to someone elses shoulders.

Installing from source is generally a bad idea. Although it allows you to tweak and configure each piece of software the way you want, tightly and efficiently, it also means everything is custom. No commoditization advantages.

o What is horizontal scalability?

This involves adding more hardware, more individual servers to service the same application and users.

o What is vertical scalability?

This means scaling up or growing your existing single server, so it is larger, has more memory, cpu or faster disk.

o What can go wrong with automatic failover?

Just about everything. Applications and services can stall, disks can fail, servers can hang. What’s more networks can exhibit latency. Automatic failover is ultimately a piece of software or algorithm trying to diagnose and handle situations. And it does so based on a very small list of rules or heuristics. The real world is messy, so this can often lead to false failure detection, and potentially loss of data.

o How do cloud vendors implement vertical scalability?

This may vary dramatically between cloud providers. Ultimately, however since virtualization allows you to boot a disk image onto any hardware, you can snapshot your current root volume or disk and then boot it on another server, one that is larger, smaller and so forth. About the only thing you need to watch out for is 32 versus 64 bit questions.

If you haven’t already, don’t forget to checkout the rest of this series – part one Operations Interview and part two Deployment Interview.

Read this far? Grab our newsletter – startup scalability.

Crisis Management in the Crosshairs – Sandy

Crisis Management During Sandy

The news this past week has brought endless images of devastation. All metropolitan region, the damage is apparent.

More than once in conversation I’ve commented “That’s similar to what I do.” The response is often one of confusion. So I go on to clarify. Web operations is every bit about disaster recovery and crisis management in the datacenter. If you saw Con Edison down in the trenches you might not know how that power gets to your building, or what all those pipes down there do, but you know when it’s out! You know when something is out of order.

That’s why datacenter operations can learn so much about crisis management from the handling of Hurricane Sandy.

This is a followup to our popular article last week Real Disaster Recovery Lessons from Sandy.

1. Run Fire Drills

Nothing can substitute for real world testing. Run your application through it’s paces, pull the plugs, pull the power. You need to know what’s going to go wrong before it happens. Put your application on life support, and see how it handles. Failover to backup servers, restore the entire application stack and components from backups.

2. Let the Pros Handle Cleanup

This week Fred Wilson blogged about a small data room his family managed, for their personal photos, videos, music and so forth. He ruminated on what would have happened to that home datacenter, were he living there today when Sandy struck.

It’s a story many of us can related to, and points to obvious advantages of moving to the cloud. Handing things over to the pros means basic best practices will be followed. EBS storage, for example is redundant, so a single harddrive failure won’t take you out. What’s more S3 offers geographically distributed redundant copies of your data.

After last week’s AWS outage I wrote that AirBNB & Reddit didn’t have to fail. What’s more in the cloud, disaster recovery is also left to the professionals.

Web Operations teams do what Con Edison does, but for the interwebs. We drill down into the bowels of our digital city, find the wires that are crossed, and repair them. Crisis management rules the day. I can admire how quickly they’ve brought NYC back up and running after the wrath of storm Sandy.

3. Have a few different backup plans

Watching New Yorkers find alternate means of transportation into the city has been nothing short of inspirational. Trains not running? A bus services takes it’s place. L trains not crossing the river? A huge stream of bikes takes to the williamsburg bridge to get workers to where they need to go.

Deploying on Amazon can be a great cloud option, but consider using multiple cloud providers to give you even more redundancy. Don’t put all your eggs in one basket.

Some very important things to remember about MySQL backups.

4. Keep Open Lines of Communication

While recovery continued apace, city dwellers below 34th street looked to text messages, and old school radios to get news and updates. When would power be restored? Does my building use gas or steam to heat? Why are certain streets coming back online, while others remain dark?

During an emergency like this one, it becomes obvious how important lines of communication are. So to in datacenter crisis management, key people from business units, operations teams, and dev all must coordinate. Orchestrating that is and art all by itself. A great CTO knows how to do this.

Read this far? Grab our monthly scalable startups.

Cloud Deployment Interview

What does a cloud computing expert need to know? In part one of the cloud interview guide we covered some basic unix & Linux systems administration skills, and cloud computing and infrastructure concepts. Those are key starting points. You might also want to jump to part 3 cloud dba, architecture and management interview questions.

In this second part, let’s dig into deploying applications in the cloud, and day to day operations skills. There’s a lot of material here. We recommend picking a few questions out of the bunch and focusing on those questions, rather than trying to cover all of them.

Also while on the topic of hiring, keep in mind that Hiring is a Numbers Game.

1. Deploying in the Cloud

Deploying applications into virtual or cloud datacenters involves understanding and evaluating providers. Many just deploy on Amazon EC2 as it is far and away the largest cloud hosting solution, with the most robust offering.

You might also like our MySQL DBA Interview Guide as well.

o What sets amazon apart from the other cloud providers?

There are probably two things that set Amazon apart from other cloud infrastructure solutions. EBS or elastic block storage being one. Although the others have storage solutions, and Rackspace is working on their own virtualized storage, Amazon seems to be the furthest ahead with their offering. It is fully virtual, allows arbitrary chunks of storage to be attached to instances, and allows instances to boot of ebs volumes.

The other major point is that since Amazon has grown so large, so quickly, it has more datacenters, in more geographically dispersed areas than other providers. Since these are organized into logical resources, and can be accessed through API, it makes your application infrastructure truly virtual.

o What are some other large cloud providers?

Joyent, Rackspace cloud, Storm on Demand, GoGrid and VoxCloud. There are certainly many others. Take a look at this Quora post: Most Reliable Cloud Providers.

o Tell one vendor management story.

Everyone who has managed operations, has worked with vendors at one point or another. For example if you’ve worked with Rackspace you know that it’s pretty easy to get a human on the line. Amazon on the other hand allows you to do-it-yourself for everything, and only later added on a support service option. So their service pattern and history are different.

Also check out 3 Things CEOs should know about the cloud.

o How do you troubleshoot a problems?

There isn’t really a right or wrong answer to this question, but it’s a nice starting point to discussion. It can also help illustrate a candidates communication skills, and how specifically they walk through solving a problem. What problem they choose as an illustration, and how they work through to a resolution is an important indicator of operations experience.

Pros and cons of Amazon versus Rackspace, configuration management & automation and cloud management solutions like Scalr and Rightscale… these and other skills are a important for a cloud deployment expert.

o What is puppet and chef?

Puppet is a configuration management system which allows ops teams to build templates for servers, and deploy many servers based on those templates. It further allows centralized control of configuration, to automate the management of a large number of servers.

Chef grew out of frustrations of Puppet, and is a sort of next generation configuration management system.

The term infrastructure as code may be thrown around. Since all cloud resources can be provisioned through API calls, everything in server deployment can be *theoretically* done via code, from spinup of servers, to installing packages, to configuring, code checkout, seeding databases and more.

Also our article What is Infrastructure provisioning and why is it important.

o What are some of the pros and cons of configuration management for operations?

Pros include allowing a smaller team to automate the deployment of a large fleet of servers, standardization, and consistency. Cons include complexity when needing to do surgical, urgent changes, and complexity when coming into an existing environment that you’ve inherited.

o How is rightscale different? What does it provide?

Rightscale is a layer on top of your cloud provider. They provide a common interface and dashboard from which to deploy servers. Templating, automation, and multi-cloud support make it a great solution for teams that have less technical expertise on staff or less hands to manage things.

o How about scalr?

They’re another management solution, that supports multiple cloud providers. They offer templating, and auto-scaling too.

While you’re here, take a look at our Myth of Five Nines – Why HA is Overrated.

2. Day to day skills

o What type of programming experience do you have?

The answer is that every ops guy or girl should be able to code, just as every developer should have some basic operational experience. Should and does are often two different things, so ask for some examples.

o shell scripts

Bash, csh, Perl and Python are all part of the Linux administrators toolbox. Writing backup scripts, log rotation, automating routine tasks and so forth are all common needs of an operations expert.

Regular expressions are a part of Unix and used in scripting to search files, cronjobs, and ETL jobs. Ask for some basic examples.

o What is continuous integration?

The old model of code deployment was called waterfall, and allowed long careful planning, coding of new features, testing, and finally deployment. The cycle could take weeks or months and iterative change took a lot of time. Continuous integration also known as agile deployments, allows a much more frequent in some cases many times per day deployment of changes.

o What are metrics good for?

Just like in website visitor tracking, and business analytics, server level analytics and tracking is possible. Collecting server metrics such as load averages, memory, disk and cpu usage over time can be invaluable. When an application slows or server stalls, checking historical metrics can often quickly reveal problems or causes.

What are some examples? nagios, ganglia, cacti, munin, opennms

o What is unit testing?

This allows for software to be build in small testable compontents. When the compontents are coded, tests are also written that test whether they are operating properly, and whether dependencies are also installed and working.

Metrics, monitoring, load testing, firewalls, security & patching, Saas, Paas and IaaS there is a wide swath of skills needed to be competent as a web operations engineer. You’ve got your work cut out for you!
o What is load testing?

By performing some benchmarks, load testing can make estimates about how the application and code will perform when more users are hitting it.

o Security & networking

Sometimes a systems administrator is a generalized admin and sometimes there is a networking specialist on staff who doesn’t allow anyone else to touch that domain.

o What are firewall rules?

Unix services use port numbers to expose those services to the world. Since all servers on the internet are identified by IP addresses, firewall rules are defined around IP addresses or groups of them, and the ports they’re allowed to access.

o What is DNS?

DNS stands for domain name services. This is the sort of yellow pages of the internet. DNS allows a server name to be converted to it’s underlying IP address. It’s a very important service for any network, and generally includes many backup servers for when the primaries experience problems.

o What is a virtual private network?

A VPC provides a network link between a physical datacenter or your offices network, and your cloud provider. It allows you to elastically grow your existing datacenter using virtual resources, while treating those new boxes more like servers in your existing datacenter. IP addresses and subnets are controlled by your existing network rules and admins.

o Why is security important in web operations?

Since your business assets are primarily stored in digital form, the security of those assets depends on the security of your computer systems. Passwords, firewalls and encryption are all relevant.

o Why is patching software important?

Since security is a moving target, and vulnerabilities are constantly being discovered in software, patching and updates are important. Staying fairly current in applying patches means you network and systems will be more secure.

o What is intrusion detection?

Bugs in software open up vulnerabilities and ways into systems. Intrusion detection attempts to detect that such intrusions and avoid further damage.

o What is Saas – Software as a Service?

An example is dropbox, and other so-called hold-my-data type solutions fall into this category.

o What is Iaas – Infrastructure as a Service?

This is raw iron, the virtualized datacenters, hosting providers such as Amazon, GoGrid, Joyent, and Rackspace.

o What is Paas – platform as a service?

Solutions such as heroku, squarespace, wpengine and engineyard fall into this category. Some provide a platform such as the WordPress CMS, with arbitrary scaling options. Others like Heroku and EngineYard allow Ruby applications to be deployed without the need for a lot of fuss at the operational level.

We’re not done yet. In part three of this series, we’ll hit on dba skills, and a series of general questions that cut across the spectrum of web operations. Or jump back to part one of the cloud interview guide.

Read this far? Grab our newsletter – startup scalability.

Real Disaster Recovery Lessons from Sandy

Also find Sean Hull’s ramblings on twitter @hullsean.

Having just spent the last 24 hours in lower manhattan, while Hurricane Sandy rolled through, it’s offered some first hand lessons on disaster recovery. Watching the city and state officials, Con Edison, first responders and hospitals deal with the disaster brings some salient insights.

1. What are your essentials?

Planning for disaster isn’t easy. Thinking about essentials is a good first question. For a real-life disaster scenario it might mean food, water, heat and power. What about backup power? Are your foods non-parishable? Do you have hands free flashlight or lamp? Have you thought about communication & coordination with your loved ones? Do you have an alternate cellular provider if your main one goes out?

With business continuity, coordinating between business units, operations teams, and datacenter admins is crucial. Running through essential services, understanding out they interoperate, who needs to be involved in what decisions and so far is key.

Here’s a real-world story where we lost a database, what caused it and how we recovered.

2. What can you turn off?

While power is being restored, or some redundant services are offline, how can you still provide limited or degraded service? In the case of Sandy, can we move people to unaffected areas? Can we reroute power to population centers? Can we provide cellular service even while regular power is out?

[quote]Hurricane Sandy has brought devastation to the East Coast. But strong coordinated efforts between NYC, State & Federal agencies has reduced the impact dramatically. We can learn a lot about disaster recovery in web operations from their model.

For web applications and datacenters, this can mean applications built with feature flags, we’ve mentioned before on this blog.

Also very important, architect your application to have a browse only mode. This allows you to service customers off of multiple webservers in various zones or regions, using lots of read-replicas or read-only MySQL slave databases. It’s easy to build lots of read-only copies of your data while there are no changes or transactions taking place.

More redundancy equals more uptime.

Like this topic? Grab our newsletter

3. Did we test the plan?

A disaster is never predictable, but watching the emergency services for the city was illustrative of some very good response. They outlined mandatory evacuation zones, where flooding was expected to be worst.

In a datacenter, fire drills can make a big difference. Running through them gives you a sense of the time it takes to restore service, what type of hurdles you’ll face, and a checklist to summarize things. In real life, expect things to take longer than you planned.

Probably the hardest part of testing is to devise scenarios. What happens if this server dies? What happens if this service fails? Be conservative with your estimates, to provide more time as things tend to unravel in an actual disaster.

Here are 5 ways to avoid EC2 outages.

4. Redundancy

In a disaster, redundancy is everything. Since you don’t know what the future will hold, better to be prepared. Have more water than you think you’ll need. Have additional power sources, bathrooms, or a plan B for shelter if you become flooded.

With Amazon’s recent outage, quite a number of internet firms failed. In our view AirBNB, FourSquare and Reddit Didn’t Have to Fail. Spreading your virtual components and services across zones and regions would help, but further across multiple cloud providers not just Amazon Web Services, but Joyent, Rackspace or other third party providers would give you further insurance against a failure in one single provider.

Redundancy also means providing multiple paths through system. From load balancers, to webservers and database servers, object caches and search servers, do you have any single points of failure? Single network path? Single place where some piece of data resides?

5. Remember the big picture

While chaos is swirling, and everyone is screaming, it’s important that everyone keep sight of the big picture. Having a central authority projecting a sense of calm and togetherness doesn’t hurt. It’s also important that multiple departments, agencies, or parts of the organization continue to coordinate towards a common goal. This coordinated effort could be seen clearly during Sandy, while Federal, State and City authorities worked together.

In the datacenter, it’s easy obsess over details and lose site of the big picture. Technical solutions and decisions need to be aligned with ultimate business needs. This also goes for business units. If a decision is unilaterally made that publish cannot be offline for even five minutes, such a tight constraint might cause errors and lead to larger outages.

Coordinate together, and everyone keep sight of the big picture – keeping the business running.

Speaking of the big picture, here’s Why generalists are better at scaling the web.

Read this far? Grab our newsletter Scalable Startups.

Cloud Operations Interview

What does a cloud computing expert need to know? How do you hire a cloud computing expert? Competition for operations & DBAs is fierce, so you’ll want to know how to find the best.

If you’re a systems administrator or ops guy, you may want to prepare for an interview for such a position. Meanwhile, if you’re a director of it or operations, a recruiter or manager in HR, you’ll want to have some idea how to find the right candidate.

Here’s my guide to do just that. You may also jump to part two Cloud Deployment Interview or the last part three Cloud DBA, Architecture and Management Interview.

1. Solid unix systems administrator

At the top of the list, a cloud operations expert needs to understand Unix and more importantly Linux. Here are some sample questions to get the conversation moving:

o What is web operations and what have you done day-to-day?

Prepare some stories.

o What’s your favorite feature of the linux kernel?

This is an open ended question, but a systems administrator should have some knowledge here. The kernel is the most basic piece of software that runs when a computer boots up, whether it is a desktop or a server. This piece of software coordinates everything, manages resources, and directs traffic.

o Name some distributions of linux. What is a distro?

Linux is built by a collaborative team of thousands on the internet. That’s what makes it open source. The distributions, include the operating system, along with a collection of software to go along with it. All the supporting utilities, libraries and servers must be compiled and held in a repository. That’s what makes up a distribution. Debian, Redhat and Ubuntu are a few popular ones.

A cloud operations expert needs to have a wide ranging skillset, from unix administration, architecture, scalability, database & webserver administration, troubleshooting & performance, load & stress testing. You’ll also want someone who has learned hard lessons from some failures, has some war stories to tell and has a hard nose for stability.

o What’s the difference between apache and nginx?

These two pieces of software are both webservers, that is they respond to the HTTP protocol, and can serve HTML pages. They also have a myriad of plugins to support different languages and features. The difference? Nginx (pronounced engine-X) is a newer incarnation. It’s been rearchitected from the ground up, building on all the things learned from Apache over the years. Its tighter, more efficient code, and easier to configure.

You might also enjoy our Intro to EC2 Cloud Deployments Guide.

o What is a key value store? examples?

There are lots of examples of these types of databases. They are a very simple memory cache that can interface with most applications. Memcache is a popular example of a key value store. Redis, CouchDB and Voldemort can also do this.

o What is a page cache? Reverse proxy cache? examples?

These are all the same thing. They are basically a very minimal webserver without all the plugins or bells and whistles. You put one of these in front of your webserver to handle all the easy stuff, and speed up overall throughput. Varnish is a popular example.

o What filesystem do you prefer?

This is a bit arcane, but one should have some opinions here. xfs is a popular filesystem, though ext3 and ext4 are also common. Emphasize the journaling aspect here. Journaling means that if you pull the cord or your server crashes, the filesystem can recover upon reboot. It does this by journaling changes, much how a database keeps a redolog cache of recent changes to database tables.

o Command line tools

There are lots of commands in the day-to-day toolbox of a web ops expert. Here are some examples:
rsync (pronounced our-sync) – sync files between servers & do checksums to allow easy restarts
scp (pronounced s-c-p) – secure copy, similar to rsync but no checksums, so less reliable
curl (pronounced kurl) – diagnose & test urls and HTTP from the command line
cron (pronounced cron) – run commands at scheduled times
ssh (pronounced s-s-h) – secure shell, the most basic tool to reach a cloud server
ifconfig (pronounced if-config) – check the network interfaces on the server
vi/emacs (pronounced v-i and e-macks) – terminal editors, to modify config files
uptime (pronounced up-time) – display the current load average of the server
top (pronounced top) – interactive display of system metrics like memory, load, swap & processes
ps (pronounced p-s) – shows running processes on the server
/var/log/messages – essential system logfile

o What are application servers? How are they different from webservers?

Tomcat & Glassfish are two examples of application servers. These handle heavier weight languages & applications like Java. Application server on some level is just a more heavyduty webserver and these days Apache can be thought of as an application server also.

2. Cloud concepts

o What is virtualization? What is a hypervisor?

Virtualization allows you to run one or more computers within a computer. You can do virtualization on a desktop, sharing network, memory, cpu and disk resources among a number of virtual servers. But more importantly in cloud computing or IaaS offerings you can do virtualization at the datacenter level. The hypervisor layer is a datacenter virtualization technology that provisions server resources, and balances shared network and disk resources.

o What is an image?

In Amazon the world, the AMI or amazon machine image is a snapshot of a server state at one moment in time. This image is take at the block level, and includes the master block record, the first block on disk that a server boots from. All that is the state of a server, when it is shutdown, is what is stored on disk or in this image. All config files, logfiles, and anything else writing to disk.

o What is multi-tenant?

This means that there are multiple servers sharing resources. The tenants are the customers who each want to get the server, cpu, memory, network and disk that they paid for.

o What is the downside to shared resources?

Contention for resources is always the challenge. If your fellow tenants are not very thirsty, this can work to your advantage. But if they’re also heavy users, the hypervisor layer has manage the balancing act. You may get a spike of disk I/O at one point, but later get a dearth. This can cause a relational database like MySQL or Oracle to suddenly look stalled.

o What is instance-store? What is ebs?

Instance store servers were Amazon’s original offering, where servers had their own local (and slow) storage. This storage was ephemeral, so all machine state was lost on reboot. These servers also boot slowly. EBS also known as elastic block storage is a virtualized storage option, similar to NAS or NFS. You can create arbitrary chunks of storage, and attach them to servers, all from command line APIs. Cool!

o What is virtual private cloud?

With the VPC offering, Amazon drops a router into your existing datacenter. You can then provision virtual servers to your hearts content, and they all appear to be servers in your existing datacenter. Elastically scale, within the network and security model you’re already using.

o What is a hybrid approach to cloud adoption?

Keeping your investments in hardware and datacenter is obviously an appealing option for firms that have large existing environment. A hybrid approach with a VPC allows you to get your feet wet, but still keep essential applications on physical servers.

o What is Amazon EC2?

Elastic Compute Cloud refers to the virtual servers you spinup in Amazon Web Services.

o What is Amazon RDS, Oracle RDS, Mysql RDS?

Amazon has various relational and non-relational database offerings. RDS stands for relational database service.

RDS or roll your own – which is better? Here are some use cases to help you decide.

o What is multi-az?

Amazon’s infrastructure offering isn’t just a single datacenter with servers. The beauty of what they’ve built is that they offer a number of datacenters (called availability zones) in each of many regions such as Northern Virginia, Oregon and Singapore.

Incidentally multi-az is a key feature to how businesses can protect themselves from failure. Amazon recently had an outage, but AirBNB, Reddit & Foursquare didn’t have to fail.

o What does a CDN do? How does it work? examples?

A CDN is a content delivery network. Remember all those files that make up a webpage? Images, video, css files? Turns out serving these components from servers *closer* to your customer, make their webpages load much faster. CDNs are networks of servers that hold the content of your pages, and serve them faster.

It works by replacing content paths with a special one from your provider. A simple change in your code will allow content to dynamically load from across the web. Cool!

CloudFront is Amazon’s offering coupled with S3 for file storage. Akamai is another big provider.

We’re not done yet. In part two on deployments and http://www.iheavy.com/2012/11/01/cloud-deployment-interview/”>part three of this series, we’ll hit on other important skills a cloud ops expert should have including scripting, database administration (Our MySQL Interview Guide), scalability, performance, configuration management, metrics, monitoring, and some all important war stories!

Here are some questions to pique your interest:

o Why does the API battle between Amazon & Eucalyptus (FOSS) matter?
o Do you use command line tools? why?
o What can go wrong with backups? how do we test them?
o Should we encrypt filesystems in the cloud? what are the risks?
o Should we use offsite backups?
o What is DRBD?
o Why is auditing important? access control?
o What is load balancing? why is it difficult with databases?
o How do you perform a benchmark? perform load testing?
o Why use a package manager? can we install from source?

Our Deploying MySQL on Amazon EC2 Guide is also related to this interview process.

You may also jump to part two Cloud Deployment Interview or the last part three Cloud DBA, Architecture and Management Interview.

Read this far? Grab our newsletter – startup scalability.

AirBNB didn't have to fail

Today part of Amazon Web Services failed, taking down with it a slew of startups that all run on Amazon’s Cloud infrastructure. AirBNB was one of the biggest, but also Heroku, Reddit, Minecraft, Flipboard & Coursera down with it. Its not the first time. What the heck happened, and why should we care?

1. Root Cause

The AWS service allows companies like AirBNB to build web applications, and host them on servers owned and managed by Amazon. The so-called raw iron of this army of compute power sits in datacenters. Each datacenter is a zone, and there are many in each of their service regions including US East (Northern Virginia), US West (Oregon), US West (Northern California), EU (Ireland), Asia Pacific (Singapore), Asia Pacific (Tokyo), South America (Sao Paulo), and AWS GovCloud.

Today one of those datacenters in the Northern Virginia region had a failure. What does this mean? Essentially firms like AirBNB that hosted their applications ONLY in Northern Virginia experienced outages.

As it turns out, Amazon has a service level agreement of 99.95% availability. We’ve long since said goodbye to the five nines. HA is overrated.

2. Use Redundancy

Although there are lots of pieces and components to a web infrastructure, two big ones are webservers and database servers. Turns out AirBNB could make both of these tiers redundant. How do we do it?

On the database side, you can use Amazon’s multi-az or alternately read-replicas. Each have different service characteristics so you’ll have to evaluate your application to figure out what will work for you.

Then there is the option to host mysql or Percona directly on Amazon servers yourself and use replication.

[quote]Using redundant components like placing webservers and databases in multiple regions, AirBNB could avoid an Amazon outage like Monday’s that affected only Northern Virginia.[/quote]
When do I want RDS versus mysql? Here are some use cases for RDS versus roll your own MySQL.

Now that you’re using multiple zones and regions for your database the hard work is completed. Webservers can be hosted in different regions easily, and don’t require complicated replication to do it.

3. Have a browsing only mode

Another step AirBNB can take to be resilient is to build a browsing only mode into their application. Often we hear about this option for performing maintenance without downtime. But it’s even more valuable during a situation like this. In a real outage you don’t have control over how long it lasts or WHEN it happens. So a browsing only mode can provide real insurance.

For a site like AirBNB this would mean the entire website was up and operating. Customers could browse and view listings, only when they went to book a room would the encounter an error. This would be a very small segment of their customers, and a much less painful PR problem.

Facebook has experience intermittent outages of it’s service. People hardly notice because they’ll often only see a message when they are trying to comment on someone’s wall post, send a message or upload a photo. The site is still operating, but not allowing changes. That’s what a browsing only mode affords you.

[quote]A browsing only mode can make a big difference, keeping most of the site up even when transactions or publish are blocked.

Drupal, an open source CMS system that powers sites like Adweek.com, TheHollywoodReporter.com, and Economist.com uses this technology. It supports a browsing only mode out of the box. An amazon outage like this one would only stop editors from publishing new stories temporarily. A huge win to sites that get 50 to 100 million with-an-m pageviews per month.

4. Web Applications need Feature Flags

Feature flags give you an on/off switch. Build them into heavy duty parts of your site, and you can disable those in an emergency. Host components multiple availability zones for extra peace of mind.

One of our all time most popular posts 5 Things Toxic to Scalability included some indepth discussion of feature flags.

5. Consider Netflix’s Simian Army

Netflix takes a very progressive approach to availability. They bake redundancy and automation right into all of their infrastructure. Then they run an app called the Chaos Monkey which essentially causes outages, randomly. If resilience from constantly falling and getting back up can’t make you stronger, I don’t know what can!

Take a look at the Netflix blog for details on intentional load & stress testing.

6. Use multiple cloud providers

If all of the above isn’t enough for you, taking it further you’d do as George Reese of enstratus recommends and use multiple cloud providers. Not being beholden to one company could help in more situations than just these type of service disruptions too.

Basic EC2 Best Practices mean building redundancy into your infrastructure. Multiple cloud providers simply take that one step further.

Read this far? Grab our newsletter on scalability and startups!

Going Solo for Fun or Profit?

Sara Horowitz has serious chops. Independent herself, she started Freelancer’s Union way back in 1995. Back then it was tougher as a freelancer. Through her great efforts, we’ve all benefited.

So when I saw she’d published a guide called “Freelancer’s Bible”, I was quick to grab a copy. And the book doesn’t disappoint. I wish this book had existed when I got my start way back when I first moved to NYC in 1996.

A budding freelancer

As a budding freelancer, you’ve got a ton of new skills to pickup, where to start? Flip to Part 1 and you’ll get a quick hitchhikers guide, with advice on setting up your office and organizing your time, to pitching to prospects, networking, and building a portfolio of different types of clients to keep your workflow steady. You also learn how to package your services, and the myriad ways to set fees from hourly, to project based and day rates to packages.

[quote]Network & market yourself, package & price services, communicate well and manage timelines, deliver, bill and finally get paid. Each step is outlined here in easy to read bullets, and helpful “Ask Sara” sections. Easy layout, and a pleasure to read.[/quote]

As I was reading these early chapters, I thought it would be nice to have a chapter on social media. Turns out I spoke too soon, as I flip through the pages, chapter 9 is all about marketing and social media, online tools to build your reputation and influence. Also I like that she appeals to the practical approach. For example Sara emphases that you let go of strategy, and experiment with different options, and methods. This is exactly what I’ve done over the years, and it’s the best way to find out what works for your personal style, as well as your industry. Trial and error!

I’ve written a guide on this topic myself. Take a look at my three part Guide to independent consulting 101.

Advice for Seasoned & Growing Operations

I’ve been working as a freelancer for 17 years now, and I’ve certainly learned a lot. So when I flip through the book, it confirms many of those lessons. But I also found material that I could use. For example chapter 7 Troubleshooting she has examples of situations where you and your client are out of sync, and offers “triple-a communication” solutions to those problems. This is the type of advice you’ll definitely need, as these scenarios are inevitable in freelance work. Also I found Chapter 10 Ways to Grow very helpful. Her list of “How do you know it’s time to grow” outlines some surprising and helpful thoughts on what to do if you have too much work. I’ve started dabbling with subcontracting and hiring additional help, so these chapters I’m finding very helpful.


There are a few things that I’d differ with Sara slightly on. Here are my thoughts.

Avoiding Contracts and Lawyers

I don’t get to heavy with lawyers and contracts. I know I know people say this is crazy, but over the years my method has served me well. It starts with a simple premise – I never intend to go to court. What do I mean? It costs too much, both in real dollars, time spent, but most of all stress. If you’ve ever been on jury duty you know what I mean.

With that, you pull the perceived safety net completely out from under yourself. So I am careful and cautious as a result. My *contracts* are simple emails, in which I outline what I’ll do, what the client will do, and who will do what when. I do all this in plain language, without any lawyer-ese. What I do get though is a confirmed *yes* in an email. This email thread is above and beyond verbal conversations and phone calls. It allows clarification down the line if you and the client have differences.

I also insist on a deposit of some kind. It doesn’t have to be a lot, but it is a hoop that you ask your client to jump through. This is very important. So-called dead beat clients will fail this test. If they are very very hesitant to provide a deposit, they are either uncomfortable with you, or are short on budget. Either case should be a red flag. Managing this relationship is very very important, when you plan never to rely on legal recourse for differences.

Who’s Played? Get paid!

On page 195 she talks about the Freelancer’s Union Client Scorecard, and “outing” clients who don’t pay. I personally think this is a bad road to go down. Why? Well there are a few reasons.

1. There are two sides to every coin

When a company hires an outside resource, they don’t have control over day-to-day operations, and overseeing what the person is doing. And yes, sadly there are many levels of work quality. So there can be differences. In my experience all those differences can and should be worked out. Communication is key and I think if you follow all of Sara’s advice on triple-A communication, you’ll avoid these situations. I do feel though that these ***

2. You can ask for an insurance deposit

Asking for a deposit from a new prospect is an important step. Without a past history of paying, and paying timely, this is a hoop you’re asking them to jump through. It proves that the budget exists, it proves that the team or director that hired you has communicated that to AP, and simply that you’re in the system. In my experience after the first check, things tend to go smoothly. If you’re experiencing trouble with this step, ask yourself – Are we on the same page? Where is the disconnect? Is the client confident you’ll deliver, and complete? Where is the hesitation?

3. It could hurt you in the end

Lastly these type of “outing” boards might hurt you in the end. If you gain a reputation for creating bad publicity or press for one firm, others may not want to work with you. I also think they are a distraction from communicating and resolving issues, and/or finding other work.

[quote]Apply all of Sara’s advice, especially those around Triple-A Communication, and you’ll likely do very well as a solopreneur. Let’s avoid becoming part of the 44% of freelancer’s who’ve reportedly had trouble getting paid![/quote]

Don’t undercharge for Services

Another point I’ll underline is charging for services. There is some talk in the book of wage wars, and 44% of freelancers not getting paid. In my experience being a freelancer is more like being another corporation. Corps fight with each other all the time. They have differences, and duke it out. It’s a bit dog eat dog out there. If you’re not prepared for that, you may be in for an uphill battle. Over the years I’ve certainly had differences with clients, but I’ve never not gotten paid. I *have* however turned away work, if I got a bad feeling about the client.

I wrote a critique of John Greathouse’s Beware the Consultant that might interest readers here. Take a look at my article Beware the Client.

That said you should be charging more than your fulltime brothers and sisters. Let’s give an example. Say your fulltime job would pay 75k/year. This theoretically is about $37.50/hr (40 hours x 50 weeks). However as a freelancer you must also pay for benefits like health insurance, retirement funds, downtime when you’re not billing, overhead of networking and meetings. You also have some additional taxes to pay. I my experience at minimum you should be charging roughly double this amount just to break even. If you’re not, it simply won’t make financial sense to stay freelancing. More likely you should be charging roughly 3x this base hourly amount. If you’re not, you may over time drift back towards fulltime employment.

I wrote another article on this topic Why do people leave consulting.

All of this should be part of educating the client. It’s often forgotten when firms look at outsourcing to get projects completed. So you should explain all of these costs clearly, and compare yourself to larger firms and agencies. These folks tend to be a *LOT* more expensive than a solopreneur.

All together now…

Sara’s bible is one every freelancer should have a copy of. It is the most complete book for a solo operator I’ve seen. Besides a few criticisms I have, it is a superb book and sure to be a reference you’ll turn to again and again.

Read this far? Grab our newsletter Scalable Startups.

Why do people leave consulting?

Join 12,100 others and follow Sean Hull on twitter @hullsean.

As a long time freelancer, it’s a question that’s intrigued me for some time. I do have some theories…

First, definitions… I’m not talking about working for a large consulting firm. Although this role may be called “consultant”, my meaning is consultant as sole proprietor, entrepreneur, gun for hire or lone wolf.

1. Make more money in a fulltime role

I’ve met a lot of people who fall into this trap. They take a fulltime role simply because it pays better. That raises a lot of questions…

o Are you pricing right?

You could be pricing to high to get *enough* work. You may also be pricing too low to cover benefits, health insurance and so forth. Or perhaps you can’t sell to your rate. You can be smart skills-wise, but do you feel your clients pain? Are you good at being a businessman? Consistent?

o Can you sell, and put together an appealing proposal?

o Can you execute to the clients satisfaction?

o Can you followup consistently while accounts payable gets tied up in knots?

o Can you followup if your client executes past their spend?

Running a business is complicated, and a lot of expenses can be hard to juggle. You will find times when a client may have spent a little faster than their revenue, and have trouble finding money when the invoice arrives. Followup, patience and persistence is key.

Read: Why high availability is so very hard to deliver

Want more? We wrote an in depth 3 part guide to consulting.

2. Make a consistent paycheck in a fulltime position

o Are you networking enough?

If you take a longterm gig and get comfortable, your pipeline can dry up. And your pipeline is the key to your longterm strength, and regular business. You must get out there, and let people know about you, your services, and your availability.

If you don’t network regularly, post across the web, engage on social media channels, blog regularly and so forth, you’ll likely just land a series of 6-12 month fulltimeish gigs through recruiters or headshops.

Related: 5 ways to evaluate independent consultants

[quote]Being a freelancer or entrepreneur involves wearing many hats. Finding business involves networking & marketing. Delivering to their needs involves emotional intelligence. And actually getting paid on time is a whole artform in itself. Leave a good taste in their mouth and your reputation will spread quickly by word of mouth.[/quote]

o Do you really *LIKE* being an entrepreneur?

Are you consistent? Consulting is like running a marathon, if you burn out you may give up!

Have a large web property or application which is experiencing some growing pains? Take a look at how we do performance reviews. It may be just what you’re looking for.

Related: MySQL interview guide for managers and candidates alike

3. Do you like the lifestyle of larger corporate environments?

o Fulltime roles allow for much more jedi sword play. Maneuvering up the ranks involves relationship building as much as consulting, but with a more well defined ladder to climb.

o Sometimes you’ll find pass the buck and pointing fingers quite common.

o There are roles involving managing people and processes. These less often lend themselves to short term or situational consulting arrangements. If you lean towards those roles

Trying to hire top tech talent? Here’s our MySQL DBA hiring guide & interview questions

[quote]Working as a sole proprietor for a couple of decades has taught me to be very entrepreneurial. It is every bit about building a real-world startup[/quote]

4. Want to do more cutting edge & at the keyboard work

Consulting can and often does allow you to bump into the latest technologies, and get your feet wet with what cutting edge firms are doing. However in a fulltime role you can more completely immerse yourself in the technology, and those long term solutions.

Also: Why devops talent is in short supply

o You can take part in R&D – Google’s 20% projects, for example

o You can build hypothetical projects

o You can work in more idealistic environments, operations and even lectures & training

Though you can certainly do all of this as a freelancer, you have to build enough capital, and so forth to make it work.

Juggling job roles as a consultant isn’t easy. What a CTO must never do.

5. Don’t like running a small business

Consulting as a sole proprietor and staying in business for almost twenty years, I’ve learned that it is every bit about running a small business or startup.

A. Acquiring customers, networking, marketing
B. Understanding their needs and delivering to improve their position
C. Pricing in a your customers understand
D. Offering value to your customers, at a competitive price
E. Managing relationships so your brand or reputation precedes you
F. Making sure payments and invoicing isn’t a hurdle, followup
G. Pacing yourself like a marathon runner – keep doing what you’re doing right

Read this far? Get our scalable startups monthly newsletter. We cover these topics in detail, year in and year out.