Is there a serious skills shortage around devops space?


As devops adoption picks up pace, the signs are everywhere. Infrastructure as code once a backwater concept, and a hoped for ideal, has become an essential to many startups.

Why might that be?

Join 37,000 others and follow Sean Hull on twitter @hullsean.

My theory is that devops enables the business in a lot of profound ways. Sure it means one sysadmin can do much more, manage a fleet of servers, and support a large user base. But it goes much deeper than that.

Being able to standup your entire dev, qa, or production environment at the click of the button transforms software delivery dramatically. It means it can happen more often, more easily, and with less risk to the business. It means you can do things like blue/green deployments, rolling out featues without any risk to the production environment running in parallel.

What kind of chops does it take?

Strong generalist skills

For starters you’ll need a pragmatist mindset. Not fanatical about one technology, but open to the many choices available. And as a generalist, you start with a familiarity with a broad spectrum of skills, from coding, troubleshooting & debugging, to performance tuning & integration testing.

Stir into the mix good operating system fundamentals, top to bottom knowledge of Unix & Linux, networking, configuration and more. Maybe you’ve built kernels, compiled packages by hand, or better yet contributed to a few open source projects yourself.

You’ll be comfortable with databases, frontend frameworks, backend technologies & APIs. But that’s not all. You’ll need a broad understanding of cloud technologies, from GCP to AWS. S3, EC2, VPCs, EBS, webservers, caching servers, load balancing, Route53 DNS, serverless lambda. Add to all of that programmable infrastructure through CloudFormation or Terraform.

Related: 30 questions to ask a serverless fanboy

Competent programmer

Although as a devop you probably won’t be doing frontend dev, you’ll need some cursory understanding of those. You should be competent at Python and perhaps Nodejs. Maybe Ruby & bash scripts. You’ll need to understand JSON & Yaml, CloudFormation & Terraform if you want to deliver IAC.

Related: Does a 4-letter-word divide dev & ops?

Strong sysadmin with ops mindset

These are fundamental. But what does that mean? Ops mindset is born out of necessity. Having seen failures & outages, you prioritize around uptime. A simpler stack means fewer moving parts & less to manage. Do as Martin Weiner would suggest & use boring tech.

But you’ll also need to reason about all these components. That’ll come from dozens of debug & troubleshooting sessions you’ll do through years of practice.

Related: How to hire a developer that doesn’t suck

Understand build systems & deployment models

Build systems like CircleCI, Jenkins or Gitlab offer a way to automate code delivery. And as their use becomes more widespread knowing them becomes de rigueur. But it doesn’t end there.

With deployments you’ll have a lot to choose from. At the very simplest a single target deploy, to all-at-once, minimum in service and rolling upgrades. But if you have completely automated your dev, qa & prod infra buildout, you can dive into blue/green deployments, where you make a completely knew infra for each deploy, test, then tear down the old.

Related: Is AWS too complex for small dev teams?

Personality to communicate across organization

I think if you’ve made it this far you will agree that the technical know-how is a broad spectrum of modern computing expertise. But you’ll also need excellent people skills to put all this into practice.

That’s because devops is also about organizational transformation. Yes devs & ops have to get up to speed on the tech, but the organization has to get on board too. Many entrenched orgs pay lip service to devops, but still do a lot of things manually. This is out of fear as much as it stands as technical debt.

But getting past that requires evangelizing, and advocating. For that a leader in the devops department will need superb people skills. They’ll communicate concepts broadly across the organization to win hearts and minds.

Related: Will Microservices just die already?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Will microservices just die already?


I was just reading Dave Kerr’s piece
The Death of Microservice Madness in 2018.

Not just because it has an awesome title, but because it was trending on for a while, and that is always a good quality signal.

And I’m all about quality. ๐Ÿ™‚

Join 37,000 others and follow Sean Hull on twitter @hullsean.

I quickly found that I agreed with him on a lot of points. There were also a bunch of serious criticisms in there that I hadn’t heard before.

Here are some of my comments on the piece:

Dave, this piece is genius. You hit on a lot of stuff here, and offered critical thought with such finesse. It’s not easy to stand up and be contrary to the trends!

o increased complexity for devs
– so true, setting up the entire suite of services on dev is tough
– and lets not forget about integration testing, which also becomes tough

Check out: The Myth of five nines

o systems have poorly defined boundaries
– very true. We can break them up into easy teams at the start, but over time things get messy, and they overlap.

Read: Lambda & serverless interview questions

o complexities of state

– Do you use a monolithic db? If so the architecture isn’t really microservices.
– If each service has it’s own, transactions that touch multiple services become very tough.
– And what about backups for all these individual databases?
– How about at restore time? How do you manage them all to restore at a SINGLE POINT IN TIME?

Check out: my get started with serverless & lambda in 5 minutes guide

o Databases without schemas push logic into the application

– They sure do. And ones without complex joins do too. It’s a dirty little secret of NoSQL

Check out: How to hire a developer that doesn’t suck

o Versioning can be hard
– Absolutely. Sure each service has it’s own version, but as Dave says you have to manage cross version compatibility. if they are truly independent, this will drift over time, in and unpredictably complex way
– And what about backup versions?

Related: Lambda & serverless interview questions

o Distributed transactions
– with a monolithic db broken up into little pieces, sometimes… maybe often, you will need to do things across data in multiple services. then what?

I like the graphic Dave put together. It’s great. I do like serverless too. I’m also critical of it. I wrote a piece 30 questions to ask a serverless fanboy ๐Ÿ™‚

Also: Is AWS too complex for small dev teams?

Get more. I write one piece every month & share it through email.. Tech, startups & innovation. My latest Can daily notes help projects succeed?

How can I get started with lambda and nodejs in 5 minutes?


I know these learn-to-do-x in 5 minutes type articles are a dime a dozen. But it’s true, we’re short on time, and we just wanna jump in. So let’s go!

Join 38,000 others and follow Sean Hull on twitter @hullsean.

Rather than go the old route of doing everything manually, and struggling, we’re going to give ourselves a skeleton to start with.

Enter, serverless framework. What’s it do? It’s a command line tool written in nodejs, which allows you to create a lambda project from a template.

From there you edit a yml file to tell serverless what to build & how. Then you put your code inside of the handler.js file. Sounds simple right?

1. Create

If you haven’t already done it, install nodejs. There are lots of docs on the interwebs. For mac users, “brew install node” does the trick!

Next install the serverless package.

$ npm install serverless

Great! If you got dependency errors, get digging. Those moments of troubleshooting & patience teach you a lot. ๐Ÿ™‚

Ok, now let’s kick the tires. We’ll create our new project.

$ serverless create --template aws-nodejs --path myEndpoint
$ cd myEndpoint

Related: 30 questions to ask a serverless fanboy

2. Edit serverless.yml

service: myEndpoint

frameworkVersion: ">=1.1.0 <2.0.0"

  name: aws
  runtime: nodejs4.3

    handler: handler.endpoint
      - http:
          path: ping
          method: get

Ok, what are we looking at here? Framework is the version of the serverless framework. Provider is aws, because serverless is attempting to build cross-platform support. You may also use azure, openwhisk, google cloud functions etc. Runtime is your language.

Under functions, our main one is currentTime. handler tells serverless framework what code to matchup with your function name. And finally events tell serverless about the API endpoint to configure.

There's a lot of magic going on under the hood. The serverless framework us using CloudFormation to build things in the background for you. CloudFormation is like Latin, it is a foundational construct to the entire AWS world. You can formalize any object, from servers to sqs queues, dynamodb tables, security groups, IAM users, S3 buckets, ebs volumes etc etc. You get the idea.

Want to see what serverless did? Head over to your aws dashboard, navigate to CloudFormation. You should see a new stack there called myEndpoint-dev. Scroll down and click the "Template" tab. You'll see the exact JSON code in all it's gory detail!

Related: 5 surprising features of Amazon Lambda serverless computing

3. Edit handler.js

Next up let's add a bit of code.

'use strict';

// return the current time in JSON format
module.exports.endpoint = (event, context, callback) => {
  const response = {
    statusCode: 200,
    body: JSON.stringify({
      message: `Hello, the current time is ${new Date().toTimeString()}.`,

  callback(null, response);

Whenever this function gets called, we'll just return the current time. Pretty self explanatory.

Related: Are you getting errors building lambda functions? I got you covered!

4. Deploy!

Now the fun party. Let's deploy the code.

$ serverless deploy

Simple command, but it's doing a lot of work. Serverless framework is packaging up your nodejs code into a zip file and uploading it to aws for you. You should see some output telling you what happened.

$ serverless deploy
Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Uploading CloudFormation file to S3...
Serverless: Uploading artifacts...
Serverless: Uploading service .zip file to S3 (1.2 KB)...
Serverless: Validating template...
Serverless: Updating Stack...
Serverless: Checking Stack update progress...
Serverless: Stack update finished...
Service Information
service: myEndpoint
stage: dev
region: us-east-1
stack: myEndpoint-dev
api keys:
  GET -
  currentTime: myEndpoint-dev-currentTime

Related: Is Amazon too big to fail?

5. Test

Awesome, now it's time to make sure it's working.

You can invoke the function directly using serverless' "invoke" command like this:

$ serverless invoke --function currentTime --log
    "statusCode": 200,
    "body": "{\"message\":\"Hello, the current time is 20:46:02 GMT+0000 (UTC).\"}"
START RequestId: ed5e427c-fe22-11e7-90cc-a1fe66d674ce Version: $LATEST
END RequestId: ed5e427c-fe22-11e7-90cc-a1fe66d674ce
REPORT RequestId: ed5e427c-fe22-11e7-90cc-a1fe66d674ce	Duration: 0.67 ms	Billed Duration: 100 ms 	Memory Size: 1024 MB	Max Memory Used: 21 MB	


But we created an API endpoint didn't we? Yep. You can hit that. If you have a browser open, go ahead and copy/past the url listed in the endpoints section of your deploy process.

You can also use curl like this:

$ curl
{"message":"Hello, the current time is 20:46:18 GMT+0000 (UTC)."}

Related: Is Amazon Web Services too complex for small dev teams?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don't work with recruiters

Are you getting errors building Amazon lambda functions? Don’t fret I got you!

aws lambda python

This post has gotten popular. So I wrote a better easier guide to diving into serverless lambda here. It’s quicker. I promise!

Why does Amazon make lambda functions so hard to create? Well my guess is that when you live at the bleeding edge you should expect to get scrapes!!

Check out: How daily notes help me work better with clients

Everybody is trying to build lambda functions these days. And it’s no wonder. Once you get them running, Amazon takes care of all the infrastructure drudge work! So cool.

Join 32,000 others and follow Sean Hull on twitter @hullsean.

I’ve been spending some time trying to get answers out of AWS support, and let me tell you it’s no fun. Yes all this stuff is new technology, and nobody has expertise in it in the way say you might in Linux or Oracle or another technology that’s been around for a decade.

Still you’d hope the techs would have some clue. In the end it was a slog dealing with support, and I think I was the one teaching them!

I did find Matt Perry’s howto which is pretty good.

Hopefully my own notes can help someone, so read on!

1. No lambda_function?

The very first issue you’re gonna run into is if you name the file incorrectly, you get this error:

Unable to import module 'lambda_function': No module named lambda_function

If you name the function incorrectly you get this error:

Handler 'handler' missing on module 'lambda_function_file': 'module' object has no attribute 'handler'

On the dashboard, make sure the handler field is entered as function_filename.actual_function_name and make sure they match up in your deployment package.

If only the messages were a bit more instructive that would have been a simpler step, but oh well!

Also: Is Amazon too big to fail?

2. No module named MySQLdb

This is a very tricky one. I mean after all you just spent all this time building your deployment package specifically for lambda, so what gives??

"Unable to import module 'lambda_function': No module named MySQLdb"

Turns out when you use a virtualenv, files will be installed into proj/lib/python2.7/site-packages/ or lib64. However Lambda wants them in the root proj/ directory! So move them there. I know I know. Weird, but that’s what they want.

Related: When hosting on Amazon turns bloodsport

3. Can’t find libmysqlclient

If you’re using the MySQLdb library like I was, you’ll eventually bump into this error:

Unable to import module 'lambda_function': cannot open shared object file: No such file or directory

Turns out that /usr/lib/ needs to be COPIED from /usr/lib. Don’t do “mv” or your system won’t have the mysql lib anymore!

Related: Are SQL databases dead?

4. Use the Amazon Lambda environment

One thing the support pointed out is that AWS as *supported images* for lambda development.

After all the errors above were resolved, it’s not clear to me that the supported AMI’s are truly required. However if you’re hitting intractable problems building a properly lambda deploy, you might wanna look at building one of these boxes.

Read: Why dropbox didn’t have to fail

5. Build your lambda deploy package

Now let’s roll it all together. Here’s are all the steps to build your deploy package.

- SSH to the instance
- mkdir test
- virtualenv test
- source proj/bin/activate
- sudo yum groupinstall 'Development Tools'
- sudo yum install mysql
- sudo yum install mysql-devel
- pip install MySQL-python
- cd test
- emacs -nw
- add your code to that file
- save the
- mv proj/lib/python2.7/site-packages/* proj/
- mv proj/lib64/python2.7/site-packages/* proj/
- rm -rf proj/lib (don't need dist-packages in the deploy pkg)
- rm -rf proj/lib64 (don't need dist-packages
- zip -r *

Also: How to hire a developer that doesn’t suck?

6. Upload your code

Uploading your code via the AWS dashboard is fine when you’re first testing things. But after a while it’ll get tiring going in the front door.

Create a new lambda function by specifying the basics as follows:

aws lambda create-function \
--function-name testfunc1 \
--runtime python2.7 \
--role arn:aws:iam::996225510001:role/lambda_basic_execution \
--handler lambda_function_file.handler_name \
--zip-file file://

And when you want to update your function, do the following:

aws lambda update-function-code \
--function-name testfunc1 \
--zip-file file://

Also: How to deploy on EC2 with Vagrant

Good luck with lambda. Once you get past Amazon’s weak documentation it’s pretty cool to be in a serverless computing environment. Happy deploying!

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters