What makes a highly valued docker expert?

via GIPHY

What exactly do we need to know about to manage docker effectively? What are the main pain points?

Join 38,000 others and follow Sean Hull on twitter @hullsean.

The basics aren’t tough. You need to know the anatomy of a Dockerfile, and how to setup a docker-compose.yml to ease the headache of docker run. You also should know how to manage docker images, and us docker ps to find out what’s currently running. And get an interactive shell (docker exec -it imageid). You’ll also make friends with inspect. But what else?

1. Manage image bloat

Docker images can get quite large. Even as you try to pair them down they can grow. Why is this?

Turns out the architecture of docker means as you add more stuff, it creates more “layers”. So even as you delete files, the lower or earlier layers still contain your files.

One option, during a package install you can do this:

RUN apt-get update && apt-get install -y mypkg && rm -rf /var/lib/apt/lists/*

This will immediately cleanup the crap that apt-get built from, without it ever becoming permanent in that layer. Cool! As long as you use “&&” it is part of that same RUN command, and thus part of that same layer.

Another option is you can flatten a big image. Something like this should work:

$ docker export 0453814a47b3 | docker import – newimage

Related: 30 questions to ask a serverless fanboy

2. Orchestrate

Running docker containers on dev is great, and it can be a fast and easy way to get things running. Plus it can work across dev environments well, so it solves a lot of problems.

But what about when you want to get those containers up into the cloud? That’s where orchestration comes in. At the moment you can use docker’s own swarm or choose fleet or mesos.

But the biggest players seem to be kubernetes & ECS. The former of course is what all the cool kids in town are using, and couple it with Helm package manager, it becomes very manageable system. Get your pods, services, volumes, replicasets & deployments ready to go!

On the other hand Amazon is pushing ahead with it’s Elastic Container Service, which is native to AWS, and not open source. It works well, allowing you to apply a json manifest to create a task. Then just as with kubernetes you create a “service” to run one or more copies of that. Think of the task as a docker-compose file. It’s in json, but it basically specifies the same types of things. Entrypoint, ports, base image, environment etc.

For those wanting to go multi-cloud, kubernetes certainly has an appeal. But amazon is on the attack. They have announced a service to further ease container deployments. Dubbed Amazon Fargate. Remember how Lambda allowed you to just deploy your *code* into the cloud, and let amazon worry about the rest? Imaging you can do that with containers, and that’s what Fargate is.

Check out what Krish has to say – Why Kubernetes should be scared of AWS

Related: What’s the luckiest thing that’s happened in your career?

3. Registries & Deployment

There are a few different options for where to store those docker images.

One choice is dockerhub. It’s not feature rich, but it does the job. There is also Quay.io. Alternatively you can run your own registry. It’s as easy as:

$ docker run -d -p 5000:5000 registry:2

Of course if you’re running your own registry, now you need to manage that, and think about it’s uptime, and dependability to your deployment pipeline.

If you’re using ECS, you’ll be able to use ECR which is a private docker registry that comes with your AWS account. I think you can use this, even if you’re not on ECS. The login process is a little weird.

Once you have those pieces in place, you can do some fun things. Your jenkins deploy pipeline can use docker containers for testing, to spinup a copy of your app just to run some unittests, or it can build your images, and push them to your registry, for later use in ECS tasks or Kubernetes manifests. Awesome sauce!

Related: Is Amazon Web Services too complex for small dev teams?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Curve ball technology questions and solutions

via GIPHY

I’ve been on the phone with a lot of companies lately. You might be surprised that some of the challenges firms struggle with in the cloud, are repeated over and over.

Join 38,000 others and follow Sean Hull on twitter @hullsean.

I put together some of the most common ones I’ve heard, and my thoughts on the right solution.

1. How would you load test?

Here’s an interesting question. Do you talk about tools? How do you approach the problem?

The first thing I talk about is simulating real users. If your site normally has 1000 active users, how will it behave when it has 5000, 10,000, 100,000 or 1million? We can simulate this by using a load testing tool, and monitoring the infrastructure and database during that test.

But how accurate are those tests? What do active users do? Login to the site? Edit and change some data? Where do active users spend most of their time? Are there some areas of the site that are busier than others? What about some dark corner of the site or product that gets less use, but is also less tuned? Suddenly a few users decide that want that feature, and performance slides!

Real world usage patterns are unpredictable. There is as much art as science to this type of load testing.

Related: 30 questions to ask a serverless fanboy

2. Why is Amazon S3’s 99.999999999% promise *not* enough??

I’ve heard people say before that S3 is extremely reliable. But is it?

According to their SLA, the durability guarantee is 11 nines. What does this mean? Durability is confidence that a file is saved. That you will not lose it. It’s on storage, and that storage has redundant copies. Great. You can be confident you will never lose a file.

What about uptime? That SLA is 99.99% or an hour a month. Surprise! That amounts to an hour of DOWNTIME per month. And if your product fails when S3 files are missing, guess what, your business is down for an hour a month.

That’s actually quite a *lot* of downtime.

Solution: You better be doing cross-region replication. You have the tools and the cloud, now get to work!

Related: What’s the luckiest thing that’s happened in your career?

3. Why is continuous integration not about tools?

I hear a lot of talk about continuous integration. I’ve even seen it as a line item on a todo list I was handed. Hmmm…

I asked the manager, “so it says here setup CI/CD. Are there already unit tests written? What about integration tests?” Turns out the team is not yet on board with writing tests. I gently explain that automated builds are not going to get very far without tests to run. 🙂

CI/CD requires the team to be on-board. It’s first a cultural change in how you develop code.

It means more regular code checkins. It means every engineer promises not to break the build.

It means write enough tests for good code coverage.

Related: How I use progress reports to achieve consulting success

4. What can VPC peering do for you?

Amazon has made VPC peering a lot easier. But what the heck is it?

In the world of cloud networking, everything is virtual. You have VPCs and inside those you have public and private subnets. What if you have multiple AWS accounts? VPC peering can allow you to connect backend private subnets, without going across the public internet at all.

As security becomes front & center for more businesses, this can be a huge win.

What’s more it’s easier now because it is semi managed by AWS.

Related: Is upgrading Amazon RDS like a sh*t storm that will not end?

5. Why go with a New York based resource?

As more and more small startups put together teams to build their MVP, the offshore market has never been hotter. And there are very talented engineers in faraway places, from Eastern Europe, to India and China. And South America too.

At ⅓ to ¼ the price, why hire US based people? Well one reason might be compliance. If you have sensitive data, that must be handled by US nationals, that might be one reason.

Why New York based? Well there is the value of being face-to-face and working side by side with teams. It may also ease the language barrier & communication. And timezone challenges sometimes make communication difficult.

And lastly ownership. With resources that are focused solely on you, and for which you are a big customer, you’re likely to get more personalized focused attention.

Related: Is Amazon Web Services too complex for small dev teams?

6. What are some common antipatterns in the cloud

Antipatterns are interesting. Because you see them regularly, and yet they are the *wrong* way to solve a problem, either they’re slower, or there is a better more reliable way to solve it.

o Using EFS Amazon’s NFS solution, instead of putting assets in S3.

It might help you avoid rewriting code, but in the end S3 is definitely the way to go.

o Hardcoded IPs in security group rules instead of naming a group.

Yes I’ve seen this. If you specify each webserver individually, what happens when you autoscale? Answer, the new nodes break! The solution is to put all the webservers in a group, and then add a security group rule allowing access from that group. Voila!

o Passing credentials around instead of using AWS instance level roles

Credentials are the bane of applications. You hardcode them and things break later. Or they create a vulnerability that you forget about. That’s why AWS invented roles. But did you know a server *itself* can have a role? That means that server and any software running on it, has permissions to certain APIs within the amazon universe. You can change a servers roles or it’s underlying policies, while the server is still running. No restart required!

Implement CI/CD as a task item

Don’t forget culture & process are the big hurdles. Installing a tool is easy. Getting everyone using it everyday is the challenge!

Reducing and managing docker image bloat

As you change your docker images, you add layers. Even as you delete things, the total image size only grows! Seems counterintuitive. What’s more when you do all that work with yum or apt-get those packages stay lying around. One way is to install packages and then cleanup all in one command. Another way is to export and import an finished image.

ssh-ing into servers or giving devs kubectl

Old habits die hard! I was watching Kelsey Hightower’s keynote at KubCon. He made some great points about kubernetes. If you give all the devs kubectl, then it’s kind of like allowing everybody to SSH into the boxes. It’s not the way to do it!

Related: Which tech do startups use most?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

I’m speaking at Techhub on Wednesday – stop by!

This wednesday I’ll be giving a talk at the newly launched New York outpost of TechHub. The talk is entitled Intro to building a web/mobile app on AWS

Join 38,000 others and follow Sean Hull on twitter @hullsean.

Although I focus on Amazon Web Services as the default cloud, the concepts could apply equally to GCP or Azure.

Want to get a head start? Download the slide deck here.

1. A short history of application hosting

Just to give some context, I’ll start by a quick walk through compute history. From the server cabinet in the back office, to the early managed hosting providers and then on to today’s modern cloud offerings, I’ll explain how we got here.

Related: 30 questions to ask a serverless fanboy

2. What the heck is serverless?

With that new context in mind, I’ll talk about that evolution one step further, to managed functions. What’s that you ask? Just hand over your code to the cloud, and let them handle running the servers, provisioning load balancers, and reacting to your customers when they hit the endpoint.

Related: What’s the luckiest thing that’s happened in your career?

3. Introducing a reference architecture

No presentation is complete without a proper diagram. My reference architecture makes use of Amazon’s many cloud services, including API endpoint, cognito for user authentication, lambda for serverless functions, dynamodb to store state information, S3 for storing objects, CloudFront for the edge caching network, and Route53 for the domain name.

Related: Ben Horowitz’s choice wisdom for startup entrepreneurs

4. Architecture walkthrough

Each of the components I mention above, requires some explanation. I’ll talk about how to setup a serverless project, how to define and manage your API endpoint. This is where users first touch your application. I’ll introduce user authentication with Amazon’s own service or a third party like OneLogin or Auth0. From there you’ll see how Amazon’s nosql database Dynamodb works, and how you can store your original & edited images in S3. And no site would be complete without an edge cache, and we’ll have that setup too. Then store your domain name in Route53 and point it to your API.

Voila site complete!

Related: How I use progress reports to achieve consulting success

5. About Sean Hull

Of course I’ll also talk a bit about myself. Mostly what I’m doing these days, and the types of boutique consulting services I offer.

I’ll also encourage everyone to Signup for my monthly newsletter. I discuss cloud, startup & innovation topics once a month.

It’s a great way to keep in touch!

Related: Which tech do startups use most?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

What’s the luckiest thing that’s happened in your career?

via GIPHY

On more than a few occasions I’ve been asked what it’s like working remote. The inevitable followup is wow, you’re lucky. You can call it luck, but I just finished talking to 50 companies, put together proposals for all of them, and 49 said no to remote.

Lucky indeed!

Join 38,000 others and follow Sean Hull on twitter @hullsean.

But to be sure, there have certainly been events that I look back on, that have had a seminal influence on my career. I thought of three. Here they are.

1. I got a newspaper route

Yes it’s true. Way back in the dawn of time, news was delivered on paper, and those would often be delivered to your doorstep, every morning. That and for only pennies.

At the time I think I was ten years old, and I was super excited because I wanted to be part of the world. And this was really the only job you could get at that age. It required a lot of organizing. You had a book with lists of all your customers, you had to keep track of who had paid, and so forth. Some wanted to pay monthly, while others insisted on weekly.

Read: 8 questions to ask an aws expert

I was lucky not only to get the paper route, but to have parents that encouraged me in this way, and could teach me how to be responsible and reliable. Yeah those were super important early lessons.

When I saw that I had a captive audience, it wasn’t long before I was selling incidental services. Do you need your driveway shoveled? I got it. How about mowing your lawn or raking those leaves? Or walking your dog? I spun this into a whole bunch of side businesses.

It was exciting because I was making my own money at such a young age, and felt the lure of self-determination. I loved that feeling.

Also: The art of resistence in advice & consulting

2. Linux happened

In the early 90’s Linux came on the scene. It might seem like a meaningless blip on the radar to you, but to me it was everything. At University I worked in the computer lab, where we were the operations staff. Those systems all ran Unix. So to go home and use a windows box, it was demoralizing.

Then out of nowhere this guy from Finland started building off of Tennenbaum’s book, Minix! I had worked on that at University, so I immediately saw the implications. I mean heck why can’t all that great software run on PCs, it’s just a matter of getting the drivers working. Big vendors didn’t want to do it, but millions of hackers around the world were happy to pickup the mantle.

Related: The 4 letter word dividing devops

From there I build a tower, cobbled together hardware, memory, disk bus. Do you want to go IDE Or SCSI? Better choose a graphics card that is supported if you want to get X11 running on that. And of course an optical mouse so that it really feels like you’re sitting at a sun workstation!

“No one has as much luck around the greens as one who practices a lot.”  
–Chi Chi Rodriguez

To me that was pure magic. I mean from boot up to graphical interface, the entire stack was built by people just like me. And I could look at all of it. So cool! Even better that we were fighting the good fight against Bill Gates & the borg! 🙂

Check out: When clients don’t pay

3. Meeting the $65/hr consultant

This is a funny story too. One of the first jobs I had in NYC was not a consulting gig. It was a small design shop in the late nineties. Their biggest customer was Miramax Films. So they were doing really cutting edge stuff. And a lot of cool tech too. After their lead engineer quite, I became the defacto go-to person for all tech projects. I guess you could say I was CTO of a team of 5.

For one of our projects we needed help. The CEO had won business to do some Oracle development, which I didn’t have a lot of experience in. So he hired a consultant to help out. Very nice guy, a bit older than me. In fact I think he was about as old then as I am today.

Read: Is Amazon too big to fail?

As he was a smoker, we stepped outside together at one point, and I chatted him up a bit. I was so eager to learn. I don’t recall if he shared his rate or I learned it some other way. But I was shocked and blown away. To me it seemed like an insane amount of money. I remember him saying something to me. “Don’t worry Sean, someday you’ll be consulting too, and making just as much.”

“No one is going to know or care about your failures, and neither should you. All you have to do is learn from them and those around you. [A]ll that matters in business is that you get it right once. Then everyone can tell you how lucky you are.” 
–Mark Cuban

Well I am a very competitive person. I also knew that I was smarter than this guy, but he had a bit more experience than me. So from there I started sniffing around. I talked to recruiters and anyone I knew. Within two weeks, I had gotten an offer for $80/hr. Shortly after that I gave my notice.

I have to thank that guy for challenging the way he did. And I’ve never looked back since!

Related: When you have to take the fall

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Surprising wisdom – thoughts on Ben Horowitz’s new startup tale

I took a recent flight to San Francisco to have meetings with a few startups. Naturally I needed some good reading to immerse myself in for the flight over and back.

Join 38,000 others and follow Sean Hull on twitter @hullsean.

I have to admit, though I’m not a management consultant, I do pickup the big ones from time to time. Good to Great, How to Make Friends & Influence People, The Lean Startup, Innovators Dillemma & Who Moved My Cheese among my favorites.

I’d seen Ben Horowitz’s book “The Hard Thing about Hard Things” in the news. But I also really love the a16z podcast, and although I don’t know a ton about the VC business, I thought it would be a good read.

Boy is that an understatement! The book is so readable & so accessible, there are nuggets of value in there for anyone in the startup world, or building their career, CEO or not!

On the efficient market hypothesis

I had always assumed Adam Smith’s invisible hand was a good theory, almost as scripture. So to see a different perspective on this, and one backed up by real experience. That’s cool.


“No, markets weren’t “efficient” at finding the truth; they were just very efficient at converging on a conclusion — often the wrong conclusion”. p52

What’s more for investors out there, it means good old fashioned investigative work can still turn up gems, that are worth investing in. Word to the wise.

Related: What I learned from David Maister’s book on trust & advising clients.

Questions for interviews

Add this one to your list of great interview questions. And if you’re being interviewed, why not volunteer this as an action plan. Great advice!


“What will you do in your first month on the job?” p122

Related: A review of Eli Pariser’s insightful book The Filter Bubble

On the Freaky Friday management technique

Freaky Friday was a movie way back in the 80’s. In it a mother & daughter are at each others throats, frustrated with the each other. They end up switching places, and quickly learn to sympathize with the other’s plight in life.

Horowitz decided to put this method to use between two of his managers. Pretty ingenious.

“After just one week walking in the other’s moccasins, both executives quickly diagnosed the core issues causing the conflict. They then swiftly acted to implement a simple set of processes that cleared up the combat and got the teams working harmoniously. p253

Related: A review of the Power of Habit by Charles Duhigg

On ancient wisdom & hard choices

“In life, everybody faces choices between doing what’s popular, easy, and wrong versus doing what’s lonely, difficult, and right.” p212

Every time you make the hard, correct decision you become a bit more courageous and every time you make the easy, wrong decision you become a bit more cowardly. p213

Boy, can I relate to these bits of wisdom. Running my own business all these years, it hasn’t been easy. There have been ups and downs, and times when people told me to take a different road. But that courage. When you find it, it can be real fuel for you moving forward.

Related: Can a growth mindset help you recover from setbacks – Carol Dweck

On instincts


I realized that embracing the unusual parts of my background would be the key to making it through. It would be those things that would give me unique perspectives and approaches to the business. p276

When I work with entrepreneurs today, this is the main thing that I try to convey. Embrace your weirdness, your background, your instinct. p276

I hadn’t really thought of this, and it’s an interesting point. It may be one of the reasons why customers hire me, that I hadn’t realized. Certainly I can give an original viewpoint. But I think I will try to put this to work in the future.

Related: Startup of You – Reid Hoffman’s great book on career growth

On publicity in ventura capital

This is a curious & fascinating point about the history of venture capital. Ripe for disruption indeed!

Marc discovered that the original venture capital firms in the late 1940s and early ’50s were modeled after the original investment banks such as J.P. Morgan and Rothschild. Those banks also did not do PR for a very specific reason: The banks funded wars—and sometimes both sides of the same war—so publicity was not a good idea. p271

Related: A review of Nicholas Carr’s book The Big Switch – Rewiring the world from Edison to Google

On the Andreesen Horowitz business model

This is pretty cool. Apparently the A16Z business model built a VC firm by helping startup founders in disruptive ways.

We decided to systematize and professionalize the network. p269

They modeled the firm after Michael Ovitz’s Creative Artists Agency. They had managed to “shift the economics of the industry from the corporations to the talent” p270

Related: Deborah Tannen offers us insights on conversation & interruption across the sexes

On forecasting

I guess I instinctively understood this one, but it’s interesting to see it in black and white like that.


You should expect experienced people to be able to forecast their results more accurately than junior people. p250

Related: 5 things I learned from Frans Johansson about innovation from his book Medici Effect

On checking yourself

Keep yourself honest, ask this question.

“It’s a good idea to ask, “What am I not doing?” p52

Related: 5 things I learned from Gif Constable in his book Talking to Humans

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

25 lessons from Adrian Mouat’s Using Docker book

I spent some time digging through Adrian Mouat’s great book on Docker. Although it’s almost two years old now, it is still chock full of useful information on container goodness.

Join 38,000 others and follow Sean Hull on twitter @hullsean.

I flipped through page after page, and chapter after chapter, and found the bits that I thought were particularly useful. And I have summarized those here.

1. Basics

o docker-compose organizes docker runs with a yaml config
o multiple services in one container is an antipattern
o deleting files don’t reduce container size, because they still exist in previous layer
o export followed by import can be a quick way to reduce image size
o docker-machine allows you to provision containers on virtual hosts locally or in the cloud

Related: 5 surprising features of Amazon Lambda serverless computing

2. Testing

o build a private registry node, then push & pull images through it with deploy pipeline
o unit tests are key and provide tests for individual functions in your code
o component tests are also important to test api endpoints for example
o integration tests can be useful, verifying an auth service or external API is working with app
o end-to-end tests verify that the entire application is working

Related: 30 questions to ask a serverless fanboy

3. Networking

o by default containers can talk, consider –icc=false & –iptables=true
o passing secrets with env variables or better yet use a file, vault or kms
o SkyDNS on top of etcd can provide a powerful service discovery solution
o use registrator project to automatically register containers when they start
o orchestration with swarm (native), fleet, mesos or Kubernetes

Related: Is upgrading Amazon RDS like a sh*t storm that will not end?

4. Security

o don’t run as root – because a breakout would have root on host
o use limits on memory, cpu, restarts & filesystem to avoid DoS
o defang setuid root binaries with a find +6000 & chmod a-s
o use gpg keys & verify checksums when downloading software
o selinux & AppArmor may help, but buyer beware

Related: Is Amazon Web Services too complex for small dev teams?

5. Miscellaneous

o you can use logsprout to send docker image logs to logstash
o add elasticsearch on top with kibana as frontend to give a great searchable logging UI
o Jason Wilder’s docker-gen can streamline config file creation from templates
o we can modularize compose files with the extends keyword (like library import)
o audit containers & use docker diff to find issues

Related: Are you getting errors building lambda functions? I got you covered!

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

6 Devops interview questions

via GIPHY

Devops is in serious demand these days. At every meetup or tech event I attend, I hear a recruiter or startup founder talking about it. It seems everyone wants to see benefits of talented operations brought to their business.

Join 37,000 others and follow Sean Hull on twitter @hullsean.

That said the skill set is very broad, which explains why there aren’t more devs picking up the batton.





I thought it would be helpful to put together a list of interview questions. There are certainly others, but here’s what I came up with.

1. Explain the gitflow release process

As a devops engineer you should have a good foundation about software delivery. With that you should understand git very well, especially the standard workflow.

Although there are other methods to manage code, one solid & proven method is gitflow. In a nutshell you have two main branches, development & master. Developers checkout a new branch to add a feature, and push it back to development branch. Your stage server can be built automatically off of this branch.

Periodically you will want to release a new version of the software. For this you merge development to master. UAT is then built automatically off of the master branch. When acceptance testing is done, you deploy off of master to production. Hence the saying always ship trunk.

Bonus points if you know that hotfixes are done directly off the master branch & pushed straight out that way.

Related: 8 questions to ask an AWS expert

2. How do you provision resources?

There are a lot of tools in the devops toolbox these days. One that is great at provisioning resources is Terraform. With it you can specify in declarative code everything your application will need to run in the cloud. From IAM users, roles & groups, dynamodb tables, rds instances, VPCs & subnets, security groups, ec2 instances, ebs volumes, S3 buckets and more.

You may also choose to use CloudFormation of course, but in my experience terraform is more polished. What’s more it supports multi-cloud. Want to deploy in GCP or Azure, just port your templates & you’re up and running in no time.

It takes some time to get used to the new workflow of building things in terraform rather than at the AWS cli or dashboard, but once you do you’ll see benefits right away. You gain all the advantages of versioning code we see with other software development. Want to rollback, no problem. Want to do unit tests against your infrastructure? You can do that too!

Related: Does a 4-letter-word divide dev & ops?

3. How do you configure servers?

The four big choices for configuration management these days are Ansible, Salt, Chef & Puppet. For my money Ansible has some nice advantages.

First it doesn’t require an agent. As long as you have SSH access to your box, you can manage it with Ansible. Plus your existing shell scripts are pretty easy to port to playbooks. Ansible also does not require a server to house your playbooks. Simply keep them in your git repository, and checkout to your desktop. Then run ansible-playbook on the yaml file. Voila, server configuration!

Related: How to hire a developer that doesn’t suck

4. What does testing enable?

Unit testing & integration testing are super import parts of continuous integration. As you automate your tests, you formalize how your site & code should behave. That way when you automate the deployment, you can also automate the test process. Let the software do the drudgework of making sure a new feature hasn’t broken anything on the site.

As you automate more tests, you accelerate the software development process, because you’re doing less and less manually. That means being more agile, and makes the business more nimble.

Related: Is AWS too complex for small dev teams?

5. Explain a use case for Docker

Docker a low overhead way to run virtual machines on your local box or in the cloud. Although they’re not strictly distinct machines, nor do they need to boot an OS, they give you many of those benefits.

Docker can encapsulate legacy applications, allowing you to deploy them to servers that might not otherwise be easy to setup with older packages & software versions.

Docker can be used to build test boxes, during your deploy process to facilitate continuous integration testing.

Docker can be used to provision boxes in the cloud, and with swarm you can orchestrate clusters too. Pretty cool!

Related: Will Microservices just die already?

6. How is communicating relevant to Devops

Since devops brings a new process of continuous delivery to the organization, it involves some risk. Actually doing things the old way involves more risk in the long term, because things can and will break. With automation, you can recovery quicker from failure.

But this new world, requires a leap of faith. It’s not right for every organization or in every case, and you’ll likely strike a balance from what the devops holy book says, and what your org can tolerate. However inevitably communication becomes very important as you advocate for new ways of doing things.

Related: How do I migrate my skills to the cloud?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Is there a serious skills shortage around devops space?

via GIPHY

As devops adoption picks up pace, the signs are everywhere. Infrastructure as code once a backwater concept, and a hoped for ideal, has become an essential to many startups.

Why might that be?

Join 37,000 others and follow Sean Hull on twitter @hullsean.

My theory is that devops enables the business in a lot of profound ways. Sure it means one sysadmin can do much more, manage a fleet of servers, and support a large user base. But it goes much deeper than that.





Being able to standup your entire dev, qa, or production environment at the click of the button transforms software delivery dramatically. It means it can happen more often, more easily, and with less risk to the business. It means you can do things like blue/green deployments, rolling out featues without any risk to the production environment running in parallel.

What kind of chops does it take?

Strong generalist skills

For starters you’ll need a pragmatist mindset. Not fanatical about one technology, but open to the many choices available. And as a generalist, you start with a familiarity with a broad spectrum of skills, from coding, troubleshooting & debugging, to performance tuning & integration testing.

Stir into the mix good operating system fundamentals, top to bottom knowledge of Unix & Linux, networking, configuration and more. Maybe you’ve built kernels, compiled packages by hand, or better yet contributed to a few open source projects yourself.

You’ll be comfortable with databases, frontend frameworks, backend technologies & APIs. But that’s not all. You’ll need a broad understanding of cloud technologies, from GCP to AWS. S3, EC2, VPCs, EBS, webservers, caching servers, load balancing, Route53 DNS, serverless lambda. Add to all of that programmable infrastructure through CloudFormation or Terraform.

Related: 30 questions to ask a serverless fanboy

Competent programmer

Although as a devop you probably won’t be doing frontend dev, you’ll need some cursory understanding of those. You should be competent at Python and perhaps Nodejs. Maybe Ruby & bash scripts. You’ll need to understand JSON & Yaml, CloudFormation & Terraform if you want to deliver IAC.

Related: Does a 4-letter-word divide dev & ops?

Strong sysadmin with ops mindset

These are fundamental. But what does that mean? Ops mindset is born out of necessity. Having seen failures & outages, you prioritize around uptime. A simpler stack means fewer moving parts & less to manage. Do as Martin Weiner would suggest & use boring tech.

But you’ll also need to reason about all these components. That’ll come from dozens of debug & troubleshooting sessions you’ll do through years of practice.

Related: How to hire a developer that doesn’t suck

Understand build systems & deployment models

Build systems like CircleCI, Jenkins or Gitlab offer a way to automate code delivery. And as their use becomes more widespread knowing them becomes de rigueur. But it doesn’t end there.

With deployments you’ll have a lot to choose from. At the very simplest a single target deploy, to all-at-once, minimum in service and rolling upgrades. But if you have completely automated your dev, qa & prod infra buildout, you can dive into blue/green deployments, where you make a completely knew infra for each deploy, test, then tear down the old.

Related: Is AWS too complex for small dev teams?

Personality to communicate across organization

I think if you’ve made it this far you will agree that the technical know-how is a broad spectrum of modern computing expertise. But you’ll also need excellent people skills to put all this into practice.

That’s because devops is also about organizational transformation. Yes devs & ops have to get up to speed on the tech, but the organization has to get on board too. Many entrenched orgs pay lip service to devops, but still do a lot of things manually. This is out of fear as much as it stands as technical debt.

But getting past that requires evangelizing, and advocating. For that a leader in the devops department will need superb people skills. They’ll communicate concepts broadly across the organization to win hearts and minds.

Related: Will Microservices just die already?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Very easy cloudformation template comparison with simple terraform for beginners

via GIPHY

If you search a bit on google, you’ll find lots of sample templates for both of these systems. However I found they had a lot of complexity.

When you’re just starting, you want a very simple example. So I thought I’d put one together.

Join 38,000 others and follow Sean Hull on twitter @hullsean.

I’m going to compare both terraform & cloudformation. They get you to the same endpoint, but do it slightly differently.

Very basic terraform template

Ok, you’ve got terraform installed right? If not there are howtos here.

Now let’s create a server.

Create a directory “terraform” then cd into it. Edit this file as main.tf

provider "aws" {
    region = "us-east-1"
}
resource "aws_instance" "example" {
    ami = "ami-40d28157"
    subnet_id = "subnet-111ddaaa"
    instance_type = "t2.micro"
    key_name = "seanKey"
}

Please change the subnet to a valid one for you. In the real world you would definitely *not* hardcode a subnet like this. But I wanted to keep this example very simple. Don’t know what subnet to use? Navigate your aws dashboard over to “VPC” and dig around.

Also of course edit for your key.

Ok, you’re ready to test. Let’s first ask terraform what it will do with the “plan” command:

levanter:terraform sean$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but
will not be persisted to local or remote state storage.


The Terraform execution plan has been generated and is shown below.
Resources are shown in alphabetical order for quick scanning. Green resources
will be created (or destroyed and then created if an existing resource
exists), yellow resources are being changed in-place, and red resources
will be destroyed. Cyan entries are data sources to be read.

Note: You didn't specify an "-out" parameter to save this plan, so when
"apply" is called, Terraform can't guarantee this is what will execute.

+ aws_instance.example
    ami:                      "ami-40d28157"
    availability_zone:        ""
    ebs_block_device.#:       ""
    ephemeral_block_device.#: ""
    instance_state:           ""
    instance_type:            "t2.micro"
    key_name:                 "seanKey"
    network_interface_id:     ""
    placement_group:          ""
    private_dns:              ""
    private_ip:               ""
    public_dns:               ""
    public_ip:                ""
    root_block_device.#:      ""
    security_groups.#:        ""
    source_dest_check:        "true"
    subnet_id:                "subnet-111ddaaa"
    tenancy:                  ""
    vpc_security_group_ids.#: ""


Plan: 1 to add, 0 to change, 0 to destroy.
levanter:terraform sean$

Related: What is devops and why is it important?

Build & change with Terraform

Next you want to ask terraform to go ahead and do the work. Because above we only did a dry-run.

levanter:terraform sean$ terraform apply
aws_instance.example: Creating...
  ami:                      "" => "ami-40d28157"
  availability_zone:        "" => ""
  ebs_block_device.#:       "" => ""
  ephemeral_block_device.#: "" => ""
  instance_state:           "" => ""
  instance_type:            "" => "t2.micro"
  key_name:                 "" => "seanKey"
  network_interface_id:     "" => ""
  placement_group:          "" => ""
  private_dns:              "" => ""
  private_ip:               "" => ""
  public_dns:               "" => ""
  public_ip:                "" => ""
  root_block_device.#:      "" => ""
  security_groups.#:        "" => ""
  source_dest_check:        "" => "true"
  subnet_id:                "" => "subnet-111ddaaa"
  tenancy:                  "" => ""
  vpc_security_group_ids.#: "" => ""
aws_instance.example: Still creating... (10s elapsed)
aws_instance.example: Still creating... (20s elapsed)
aws_instance.example: Creation complete

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

The state of your infrastructure has been saved to the path
below. This state is required to modify and destroy your
infrastructure, so keep it safe. To inspect the complete state
use the `terraform show` command.

State path: terraform.tfstate
levanter:terraform sean$ 

One thing I like is terraform shows us the progress at command line. Cloudformation isn’t so nicely finished. 🙂

Ok, let’s add a tag name to our server. We’re going to add just three lines to our main.tf file:

provider "aws" {
    region = "us-east-1"
}

resource "aws_instance" "example" {
    ami = "ami-40d28157"
    subnet_id = "subnet-111ddaaa"
    instance_type = "t2.micro"
    tags {
        Name = "terraform-box"
    }
}

Now we do terraform apply again. Look how easy that change is to make!

levanter:terraform sean$ terraform apply
aws_instance.example: Refreshing state... (ID: i-0ddd063bbbbce56e2)
aws_instance.example: Modifying...
  tags.%:    "0" => "1"
  tags.Name: "" => "terraform-box"
aws_instance.example: Modifications complete

Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

The state of your infrastructure has been saved to the path
below. This state is required to modify and destroy your
infrastructure, so keep it safe. To inspect the complete state
use the `terraform show` command.

State path: terraform.tfstate
levanter:terraform sean$ 

Navigate to the EC2 dashboard and you should see the first column showing your new name.

That was cool!

Chances are you don’t wanna leave these components sitting around. Let’s cleanup. That’s easy too!

levanter:terraform sean$ terraform destroy
Do you really want to destroy?
  Terraform will delete all your managed infrastructure.
  There is no undo. Only 'yes' will be accepted to confirm.

  Enter a value: yes

aws_instance.example: Refreshing state... (ID: i-0ddd063bbbbce56e2)
aws_instance.example: Destroying...
aws_instance.example: Still destroying... (10s elapsed)
aws_instance.example: Still destroying... (20s elapsed)
aws_instance.example: Still destroying... (30s elapsed)
aws_instance.example: Still destroying... (40s elapsed)
aws_instance.example: Still destroying... (50s elapsed)
aws_instance.example: Still destroying... (1m0s elapsed)
aws_instance.example: Destruction complete

Destroy complete! Resources: 1 destroyed.
levanter:terraform sean$ 

Related: Top questions to ask on a devops interview

Very basic CloudFormation template example

Hopefully you wrote down your subnet name & keyname. So this will be easy.

Let’s create a “cfn” directory and cd into it.

Next edit main.yml

AWSTemplateFormatVersion: '2010-09-09'

Resources:
  EC2Instance:
    Type: AWS::EC2::Instance
    Properties:
      InstanceType: t2.micro
      SubnetId: subnet-333dfe6a
      KeyName: "iheavy"
      ImageId: "ami-40d28157"

Now let’s build that with cloudformation. You need to have the awscli installed. Here’s amazon’s howto.

Now let’s create. Cloudformation organizes things as “stacks.

aws cloudformation create-stack --template-body file://sean-instance.yml --stack-name cfn-test

Since I didn’t define “outputs” to keep the yaml simple, the command above should just return without error.

You can go into the aws dashboard, and navigate to “CloudFormation” and see the stack being created. You can also see under “EC2” a new instance has been created.

Related: How do I migrate my skills to the cloud?

Add an instance name with tags in Cloud Formation

As we did with terraform, let’s add a name to the server. This is just a tag, not a hostname, so it’s only useful throughout the AWS API.

AWSTemplateFormatVersion: '2010-09-09'

Resources:
  EC2Instance:
    Type: AWS::EC2::Instance
    Properties:
      InstanceType: t2.micro
      SubnetId: subnet-333dfe6a
      KeyName: "iheavy"
      ImageId: "ami-40d28157"
      Tags:
        - Key: "Name"
          Value: "cfn-box"

Note the three new lines at the bottom. Ok, let’s apply those changes:

levanter:cfn sean$ aws cloudformation update-stack --template-body file://sean-instance.yml --stack-name cfn-test

Navigate to the EC2 dashboard and you should see the first column showing your new name.

Time to cleanup. Let’s delete that stack:

levanter:cfn sean$ aws cloudformation delete-stack --stack-name cfn-test12
levanter:cfn sean$ 

Related: Is upgrading Amazon RDS like a sh*t storm that will not end?

Conclusions

Terraform just supports JSON or it’s HCL (hashicorp configuration language). Actually the latter way of formatting is better supported.

On the CloudFormation side you can use yaml or json.

However CloudFormation can be clunky and frustrating to work with. For example to dry-run in terraform is easy. Just use “plan”. And isn’t something we’re going to do over and over?

In CloudFormation there is a “validate-template” option, but this just checks your JSON or YAML. It doesn’t hit amazon’s API or test things in any real way. They have added something called Change Sets, but I haven’t tried them too much yet.

Also CloudFormations error messages are really lacking. They often give you a syntax error or tell you a resource is incomplete without real details on where or how. It makes debugging slow and tedious. Sometimes I see errors at create-stack calls. Other times that succeeds only to find errors within the CloudFormation dashboard.

Terraform is wayyyyy better.

Related: Is Amazon Web Services too complex for small dev teams?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Will microservices just die already?

via GIPHY

I was just reading Dave Kerr’s piece
The Death of Microservice Madness in 2018.

Not just because it has an awesome title, but because it was trending on news.ycombinator.com for a while, and that is always a good quality signal.

And I’m all about quality. 🙂

Join 37,000 others and follow Sean Hull on twitter @hullsean.

I quickly found that I agreed with him on a lot of points. There were also a bunch of serious criticisms in there that I hadn’t heard before.





Here are some of my comments on the piece:


Dave, this piece is genius. You hit on a lot of stuff here, and offered critical thought with such finesse. It’s not easy to stand up and be contrary to the trends!

o increased complexity for devs
– so true, setting up the entire suite of services on dev is tough
– and lets not forget about integration testing, which also becomes tough

Check out: The Myth of five nines


o systems have poorly defined boundaries
– very true. We can break them up into easy teams at the start, but over time things get messy, and they overlap.

Read: Lambda & serverless interview questions


o complexities of state

– Do you use a monolithic db? If so the architecture isn’t really microservices.
– If each service has it’s own, transactions that touch multiple services become very tough.
– And what about backups for all these individual databases?
– How about at restore time? How do you manage them all to restore at a SINGLE POINT IN TIME?

Check out: my get started with serverless & lambda in 5 minutes guide


o Databases without schemas push logic into the application

– They sure do. And ones without complex joins do too. It’s a dirty little secret of NoSQL

Check out: How to hire a developer that doesn’t suck

o Versioning can be hard
– Absolutely. Sure each service has it’s own version, but as Dave says you have to manage cross version compatibility. if they are truly independent, this will drift over time, in and unpredictably complex way
– And what about backup versions?

Related: Lambda & serverless interview questions

o Distributed transactions
– with a monolithic db broken up into little pieces, sometimes… maybe often, you will need to do things across data in multiple services. then what?

I like the graphic Dave put together. It’s great. I do like serverless too. I’m also critical of it. I wrote a piece 30 questions to ask a serverless fanboy 🙂

Also: Is AWS too complex for small dev teams?

Get more. I write one piece every month & share it through email.. Tech, startups & innovation. My latest Can daily notes help projects succeed?