Consulting – Scalable Startups

Bidding on projects is an art as much as a science. Exciting a customer, around skills and past successes is as important as being able to see details that haven’t yet materialized.

Join 38,000 others and follow Sean Hull on twitter @hullsean.

So how does one approach this challenge. One way is to steer towards time and materials, and let things evolve in their own way. But that may not always work.

Here are my thoughts on how to navigate a fixed-fee project.

Overhead costs

When thinking about costing of projects, there are a lot of hidden costs. For fulltime folks, there is the cost of overhead around office space, supplies, training, liability & health insurance, retirement, time off and even severance in some cases.

There is also the cost of time, to hire the right team, manage them, and bring all the pieces together to success product out the door.

Lots of intangibles.

Evolving scope

When looking at a project, to come up with a realistic fixed bid, the scope must be carefully considered. If the bridge has two spans at either end and you decide to add one in the middle, does that mean a project of twice the size?

Both the vendor and manager must together attempt to break down the full scope into smaller pieces. Inevitably there will be some amount of emergent tasks and the scope will change and evolve.

Both consultant and customer must be realistic about this. You can call them product features or in the agile universe stories, but at the end of the day when you have many pieces surprises will happen.

The devil is surely in the details!

Horse Trading Skills

Given that we know things will change, the customer and vendor should plan for change.

If both parties have a realistic perspective, there is the possibility of exchanging original scoped items for emergent or evolving scoped surprises.

That is both need to be comfortable doing some sort of horse trading, to keep the levels balanced. The client then gets some leeway, as does the consultant in deliverables.

It’s not easily, but truly necessary in a fixed priced project. Because a scope never really sits still.

Underbidding

Another approach that may work is underbidding to win the project. Here your scope is expected to change, and it becomes a painful process each time. If you are strong on sales, this may work, but you’re sure to get an endless stream of change orders, and many many scrapes and bruises.

Related: Why I ask for a deposit

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

June 11, 2018

How to use terraform to setup vpc & bastion box

via GIPHY

If you’re building infrastructure on AWS or GCP you need a sandbox in which to place your toys. That sandbox is called a VPC. It’s one of those lovely acronyms that we in the tech world take for granted.

Join 38,000 others and follow Sean Hull on twitter @hullsean.

Those letters stand for Virtual Private Cloud, one of many networks within your cloud, that serve as a firewall, controlling access to servers, applications and other resources.

1. What is it for?

VPC partitions off your cloud, allowing you to control who gets into what. A VPC typically has a private Zone and a public Zone.

Within your private Zone you’ll have 2 or more private subnets and within your public, you’ll have two or more public subnets. These each sit in different availability zones, or data centers within a region. Having at least two means you can be redundant right from the start.

2. How to setup the VPC

Terraform has some excellent community modules that help you get on the ground running. One of those facilitates creating a VPC for you. When you create your VPC, the main things you want to think about are:

o what region am I building in?
o what az’s do I want to use?
o what network cidr’s to use?

You’ll have important outputs when you build your vpc. In particular the private subnets, public subnets and default security groups, which you will reference over and over in all of your terraform code. That’s because RDS databases, ec2 instances, redis clusters and many other resources sit inside of a subnet.

module "my-vpc" {
  source = "terraform-aws-modules/vpc/aws"

  name = "my-vpc"
  cidr = "10.0.0.0/16"

  azs             = ["us-east-1a","us-east-1b"]
  private_subnets = ["10.0.1.0/24", "10.0.2.0/24"]
  public_subnets  = ["10.0.101.0/24", "10.0.102.0/24"]

  enable_nat_gateway   = true
  single_nat_gateway   = true
  reuse_nat_ips        = false
  enable_vpn_gateway   = false
  enable_dns_hostnames = true

  tags = {
    Terraform   = "true"
    Environment = "dev"
  }
}

Note, this module can do a *lot* more. For example you can attached an unchanging or fixed IP (elastic IP in aws terminology) to the NAT device. This is useful so that your application appears to be coming from a single box all the time. It allows upstream providers, APIs and other integrations to whitelist you, allows your application and servers to tie into those services predictably and cleanly.

Also note that we created some nice tags. These tags become more and more important as you automate more of your infrastructure, because you will dig through the dashboard from time to time and can easily figure out what is what. You can also use a tag such as “monitoring = yes” to filter for resources that your monitoring system should tie into.

3. How to add the bastion

You want to deploy all servers in private subnets. That’s because the internet is a dangerous place these days. Everything and I mean everything. From there you provide only two ways to reach those resources. A loac balancer fronts your applications, opening ports 80, 443 or other relavant ports. And a jump box fronts your ssh access.

Place the bastion box in your PUBLIC subnet, so that you can reach it from the outside internet.

Again we’re using an amazing community terraform module, which also implements another cool feature for us. Note we deploy mykey onto the box. Think of this as your master key. But you may want to provide other users access to these machaines. In that case, simply place their public keys into my-public-keys-bucket.

Terraform will automatically deploy a key copying job onto this box via user-data script. The job will run via cron every 15 minutes, and copy (sync rather) public keys into the authorized keys file. This will allow you to add/remove users easily.

There are of course many more sophisticated networks which would require more nuanced user control, but this method is great for starters. 🙂

module "my-bastion" {
  source                      = "github.com/terraform-community-modules/tf_aws_bastion_s3_keys"
  instance_type               = "t2.micro"
  ami                         = "ami-976152f2"
  region                      = "us-east-1"
  key_name                    = "mykey"
  iam_instance_profile        = "s3_readonly"
  s3_bucket_name              = "my-public-keys-bucket"
  vpc_id                      = "${module.my-vpc.vpc_id}"
  subnet_ids                  = "${module.my-vpc.public_subnets}"
  keys_update_frequency       = "*/15 * * * *"
  additional_user_data_script = "date"
  name  = "my-bastion"
  associate_public_ip_address = true
  ssh_user = "ec2-user"
}

# allow ssh coming from bastion to boxes in vpc
#
resource "aws_security_group_rule" "allow_ssh" {
  type            = "ingress"
  from_port       = 22
  to_port         = 22
  protocol        = "tcp"
  security_group_id = "${module.my-vpc.default_security_group_id}"
  source_security_group_id = "${module.my-bastion.security_group_id}" 
}

4. Add an EC2 instance

Now that we have a bastion box in the public subnet, we can use it as a jump box to resources sitting in the private subnets.

Let’s add an ec2 instance in one of our private subnets first. Then in the test section, you can actually reach those boxes by configuring your ssh config.

Here’s the code to create an ec2 instance. Create a file testbox.tf and add these lines. Then do the usual “$ terraform plan && terraform apply”

resource "aws_instance" "example" {
  ami           = "ami-976152f2"
  instance_type = "t2.micro"
  subnet_id = "${module.my-vpc.public_subnets}"
  key_name = "mykey"
}

5. Testing

In order to test, you’ll need to edit your local ssh config file. This sits in ~/.ssh/config and defines names you can use on your local machine, to hit resources out there on the internet via ssh. Each definition includes a host, an ssh key and a user.

Below we define our bastion box. With that saved to our ssh config file, we can do “$ ssh bastion” and login to it without any password. Excellent!

The second section is even cooler. Remember that our testbox sits in a private subnet, so there is no route to it from the internet at all. Even if we changed it’s security group to allow all ports from all source IPs, it would still not be reachable. 10.0.1.19 is not an internet IP, it is one only defined within the world of our private subnet.

The second section defines how to use bastion as a proxy to reach the testbox. Once that is added to our ssh config file, we can do “$ ssh testbox” and magically reach it in one hop, by using the bastion as a proxy.

Host bastion
   Hostname ec2-22-205-135-133.compute-1.amazonaws.com
   IdentityFile ~/.ssh/mykey.pem
   User ec2-user
   ForwardAgent yes


Host testbox
   Hostname 10.0.1.19
   IdentityFile ~/.ssh/mykey.pem
   User ec2-user
   ProxyCommand ssh bastion -W %h:%p

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

June 6, 2018June 6, 2018

How best to do discovery in cloud & devops engagements?

via GIPHY

Customers reach out to me asking to do implementations, that is architecting applications, deploying code to the cloud, optimizing, tuning, and automating all the things.

Join 38,000 others and follow Sean Hull on twitter @hullsean.

But there are also a portion of engagements the require an amount of discovery. Some of that is technical in nature, and some is more around people and process.

Here are my thoughts.

1. Technical discovery

This is the most obvious type of discovery I might do. It would involve code reviews to begin, and then architecture reviews. Diagrams, microservice communication, apis and so forth.

Here’s a sample executive summary I did for one engagement, with names changed.

Next there is infrastructure, which of course should be defined in code. Terraform and CloudFormation provide good solutions here.

There also is hopefully documentation to review. This includes README’s and code comments, but also confluence docs as well.

2. Process discovery

Understanding the process of how the engineering team builds software, and gets new features to customers cannot be overstated.

What is the methodology? How are deployments managed? Do they break often? How quickly can a developer get changes to production?

I’d recommend this a16z podcast on devops to get a better understanding of this process.

Related: When clients don’t pay

3. Team discovery

This is another area that is key to success. Is there an offshore team? Are SRE’s working remote? Are devs all here in New York or elsewhere? How well is communication happening? Are there trouble spots? Bottlenecks?

In particular it’s worth looking at strengths, weaknesses, opportunities and threats to team and cohesion.

Related: A CTO must never do this

4. Tools discovery

I’m often surprised how many firms don’t know what they have. As enterprises grow, and as team turnover changes, the institutional knowledge can sometimes move with them.

In these cases review of systems and tools in place can be very helpful. Tracking a product, its deployment, and the components in place to facilitate that.

This process can uncover surprises and much room for improvement.

Related: When you have to take the fall

5. In Summary

I’ve uncovered opportunities for improvement in all of the four areas. Although technical discovery high on the list, the other areas can also be ripe areas for investigation.

Production quality, efficiency, and speed of execution and overall team morale and communication all contribute to the velocity of the firm in the marketplace.

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

May 24, 2018May 24, 2018

How to find consulting clients?

via GIPHY

I get asked this question by people a lot. Whenever I attend conferences, meetups, or social events. How do you *do* consulting? I’d love to be doing that, how do I get there?

Join 38,000 others and follow Sean Hull on twitter @hullsean.

From what I can tell, the most important factor is being hungry. How do you teach that? If you’re fiercely independent to begin with, you may be willing to have a roomate, or do without luxury for a while, in order to build up your nest egg. To be sure you need some cushion to get started.

But you also need customers. Where the heck do you find them? Here are some tips…

1. What not to do?

The first thing you’ll find is that recruiters seem to have a lot of “customers”. Maybe I can just go that route. Sure, but just remember, those are not *your* customers. Your customer is the recruiting company. They have the relationship.

Why does this matter? Because you can’t build a business this way. You are effectively working as an employee of the recruiting company. Nothing wrong with it, but it’s not an entrepreneurial path. You don’t really have responsibility nor control over the full lifecycle of your business.

So again I would summarize, don’t use “hire a freelancer” sites.

And don’t use consulting headshops or recruiters.

2. Do socialize

So how then? Well you need to first *peer* with economic buyers. What do I mean by that? Well if you go to technical conferences, that’s fine, but your peers are other engineers. These are not buyers. At best you may get a weak referral.

Hiring managers, CTOs, directors of operations, all will attend events where their peers will be found. If you want to be a professional services consultant, these folks are whom you need to socialize with.

First, experiment. Go to a lot of different types of events, meetups, small single track conferences. And ask where others go? What’s on their radar? Also introduce people to eachother. This may sound counterintuitive, but it’s important. Don’t have an agenda of “I’m looking for a job” but rather, I have a lot to offer.

Network by interviewing. This may appear to be an odd one, but it sometimes works. Take any interviews you can. Discuss how you solve problems. Learn by failing a few. If you talk to ten firms that have a seat to fill, one of them may go the consulting route, even though that wasn’t their original thought.

Talk to recruiters. It may sounds at odds with what I said above, but it can be very valuable. Recruiters have their finger on the pulse. Even if their not physicians, they can measure the pulse. So too they may not know redshift from Oracle, but they’re hearing what the industry is looking for. They have a great perspective to share.

Go to non-peer events. Expand your horizons. Surprise yourself with who might attend other events. Tell your story. In the process, ask others where they spend time.

Ping all the people. Yes keep in touch with folks. You may create a newsletter to help with this. See below.

Keep the pipeline warm. Once you get a gig, you may quickly give up the socializing because you just want to do the work. But this will fail in the long term. You have to like the socializing and keep doing it. Even while you have an engagement or two going.

Always *GET* cards. Giving them is fine, but be sure to get the contact of the other person. 99% will not followup. That’s your job!

Related: When clients don’t pay

3. Build your reputation

When people search your name, they should find you. On social networks like Linkedin, github, twitter, google plus, Slideshare, StackOverflow etc. Create profiles on all of these. Link them back to your professional site.

You *do* have a professional site right? If not go get a domain right now. devopsguru.io, backenddeveloper.guru, whatever! The domain doesn’t matter that much, most traffic will come from google, and it won’t be going to your homepage anyway.

Speak at non-peer events & conferences. Lunch & learns. Co-working spaces, incubators. These all have events, they’re all looking for experts. You may also apply to CFP’s regularly. Hey you might even get some conference passes out of it!

As I also mentioned above, a newsletter is also a good idea. Add every single person you meet in your professional context. It gives you something else to talk about as you are socializing. 🙂

Related: Why i ask for a deposit

4. Positioning

This one is counterintuitive. Why can’t I just do the thing I love.

Well sure maybe you can. But finding an underserved niche is a fast track to success. To my mind it’s crucial. So find out what is in demand. I know you’ve been talking to recruiters, right?

So yes pay attention to the wind. And pivot as necessary. Keep reading and stay up to date on new tech.

5. What you might find

Don’t expect to get in at large companies like google & facebook, or with defense contractors. There’s a terrible amount of bearocracy, and you would need a larger team to become an approved vendor. Also many of these larger well organized firms already have tons of talent.

You’re better suited to less organized, or newer companies. Because you want to be able to raise the bar for them. Better you start where the bar is a bit lower.

Examples…

o small early stage startups

These folks have some money, but they are still small so they may not need a fullsize engineering organization. They also need things done yesterday. So they are ripe with opportunities.

o medium size startups

Same as above. But they may be having trouble finding your skills. Because you’ve found that niche, right?

o greenfield

Startups building an mvp, where the skies the limit. You have the opportunity to build out the first gen. Go for it!

o second generation & legacy

Once a startup has seen it’s first round of developers leave, they may be in a spot, where the business is great, but the product needs lifting. You’re looking at a quote-unquote legacy application, and you need to use your skills to tune, troubleshoot, and identify technical debt.

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

April 9, 2018

What makes a highly valued docker expert?

via GIPHY

What exactly do we need to know about to manage docker effectively? What are the main pain points?

Join 38,000 others and follow Sean Hull on twitter @hullsean.

The basics aren’t tough. You need to know the anatomy of a Dockerfile, and how to setup a docker-compose.yml to ease the headache of docker run. You also should know how to manage docker images, and us docker ps to find out what’s currently running. And get an interactive shell (docker exec -it imageid). You’ll also make friends with inspect. But what else?

1. Manage image bloat

Docker images can get quite large. Even as you try to pair them down they can grow. Why is this?

Turns out the architecture of docker means as you add more stuff, it creates more “layers”. So even as you delete files, the lower or earlier layers still contain your files.

One option, during a package install you can do this:

RUN apt-get update && apt-get install -y mypkg && rm -rf /var/lib/apt/lists/*

This will immediately cleanup the crap that apt-get built from, without it ever becoming permanent in that layer. Cool! As long as you use “&&” it is part of that same RUN command, and thus part of that same layer.

Another option is you can flatten a big image. Something like this should work:

$ docker export 0453814a47b3 | docker import – newimage

2. Orchestrate

Running docker containers on dev is great, and it can be a fast and easy way to get things running. Plus it can work across dev environments well, so it solves a lot of problems.

But what about when you want to get those containers up into the cloud? That’s where orchestration comes in. At the moment you can use docker’s own swarm or choose fleet or mesos.

But the biggest players seem to be kubernetes & ECS. The former of course is what all the cool kids in town are using, and couple it with Helm package manager, it becomes very manageable system. Get your pods, services, volumes, replicasets & deployments ready to go!

On the other hand Amazon is pushing ahead with it’s Elastic Container Service, which is native to AWS, and not open source. It works well, allowing you to apply a json manifest to create a task. Then just as with kubernetes you create a “service” to run one or more copies of that. Think of the task as a docker-compose file. It’s in json, but it basically specifies the same types of things. Entrypoint, ports, base image, environment etc.

For those wanting to go multi-cloud, kubernetes certainly has an appeal. But amazon is on the attack. They have announced a service to further ease container deployments. Dubbed Amazon Fargate. Remember how Lambda allowed you to just deploy your *code* into the cloud, and let amazon worry about the rest? Imaging you can do that with containers, and that’s what Fargate is.

Check out what Krish has to say – Why Kubernetes should be scared of AWS

3. Registries & Deployment

There are a few different options for where to store those docker images.

One choice is dockerhub. It’s not feature rich, but it does the job. There is also Quay.io. Alternatively you can run your own registry. It’s as easy as:

$ docker run -d -p 5000:5000 registry:2

Of course if you’re running your own registry, now you need to manage that, and think about it’s uptime, and dependability to your deployment pipeline.

If you’re using ECS, you’ll be able to use ECR which is a private docker registry that comes with your AWS account. I think you can use this, even if you’re not on ECS. The login process is a little weird.

Once you have those pieces in place, you can do some fun things. Your jenkins deploy pipeline can use docker containers for testing, to spinup a copy of your app just to run some unittests, or it can build your images, and push them to your registry, for later use in ECS tasks or Kubernetes manifests. Awesome sauce!

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

April 6, 2018

Curve ball technology questions and solutions

via GIPHY

I’ve been on the phone with a lot of companies lately. You might be surprised that some of the challenges firms struggle with in the cloud, are repeated over and over.

Join 38,000 others and follow Sean Hull on twitter @hullsean.

I put together some of the most common ones I’ve heard, and my thoughts on the right solution.

1. How would you load test?

Here’s an interesting question. Do you talk about tools? How do you approach the problem?

The first thing I talk about is simulating real users. If your site normally has 1000 active users, how will it behave when it has 5000, 10,000, 100,000 or 1million? We can simulate this by using a load testing tool, and monitoring the infrastructure and database during that test.

But how accurate are those tests? What do active users do? Login to the site? Edit and change some data? Where do active users spend most of their time? Are there some areas of the site that are busier than others? What about some dark corner of the site or product that gets less use, but is also less tuned? Suddenly a few users decide that want that feature, and performance slides!

Real world usage patterns are unpredictable. There is as much art as science to this type of load testing.

2. Why is Amazon S3’s 99.999999999% promise not enough??

I’ve heard people say before that S3 is extremely reliable. But is it?

According to their SLA, the durability guarantee is 11 nines. What does this mean? Durability is confidence that a file is saved. That you will not lose it. It’s on storage, and that storage has redundant copies. Great. You can be confident you will never lose a file.

What about uptime? That SLA is 99.99% or an hour a month. Surprise! That amounts to an hour of DOWNTIME per month. And if your product fails when S3 files are missing, guess what, your business is down for an hour a month.

That’s actually quite a *lot* of downtime.

Solution: You better be doing cross-region replication. You have the tools and the cloud, now get to work!

3. Why is continuous integration not about tools?

I hear a lot of talk about continuous integration. I’ve even seen it as a line item on a todo list I was handed. Hmmm…

I asked the manager, “so it says here setup CI/CD. Are there already unit tests written? What about integration tests?” Turns out the team is not yet on board with writing tests. I gently explain that automated builds are not going to get very far without tests to run. 🙂

CI/CD requires the team to be on-board. It’s first a cultural change in how you develop code.

It means more regular code checkins. It means every engineer promises not to break the build.

It means write enough tests for good code coverage.

4. What can VPC peering do for you?

Amazon has made VPC peering a lot easier. But what the heck is it?

In the world of cloud networking, everything is virtual. You have VPCs and inside those you have public and private subnets. What if you have multiple AWS accounts? VPC peering can allow you to connect backend private subnets, without going across the public internet at all.

As security becomes front & center for more businesses, this can be a huge win.

What’s more it’s easier now because it is semi managed by AWS.

5. Why go with a New York based resource?

As more and more small startups put together teams to build their MVP, the offshore market has never been hotter. And there are very talented engineers in faraway places, from Eastern Europe, to India and China. And South America too.

At ⅓ to ¼ the price, why hire US based people? Well one reason might be compliance. If you have sensitive data, that must be handled by US nationals, that might be one reason.

Why New York based? Well there is the value of being face-to-face and working side by side with teams. It may also ease the language barrier & communication. And timezone challenges sometimes make communication difficult.

And lastly ownership. With resources that are focused solely on you, and for which you are a big customer, you’re likely to get more personalized focused attention.

6. What are some common antipatterns in the cloud

Antipatterns are interesting. Because you see them regularly, and yet they are the *wrong* way to solve a problem, either they’re slower, or there is a better more reliable way to solve it.

o Using EFS Amazon’s NFS solution, instead of putting assets in S3.

It might help you avoid rewriting code, but in the end S3 is definitely the way to go.

o Hardcoded IPs in security group rules instead of naming a group.

Yes I’ve seen this. If you specify each webserver individually, what happens when you autoscale? Answer, the new nodes break! The solution is to put all the webservers in a group, and then add a security group rule allowing access from that group. Voila!

o Passing credentials around instead of using AWS instance level roles

Credentials are the bane of applications. You hardcode them and things break later. Or they create a vulnerability that you forget about. That’s why AWS invented roles. But did you know a server *itself* can have a role? That means that server and any software running on it, has permissions to certain APIs within the amazon universe. You can change a servers roles or it’s underlying policies, while the server is still running. No restart required!

Implement CI/CD as a task item

Don’t forget culture & process are the big hurdles. Installing a tool is easy. Getting everyone using it everyday is the challenge!

Reducing and managing docker image bloat

As you change your docker images, you add layers. Even as you delete things, the total image size only grows! Seems counterintuitive. What’s more when you do all that work with yum or apt-get those packages stay lying around. One way is to install packages and then cleanup all in one command. Another way is to export and import an finished image.

ssh-ing into servers or giving devs kubectl

Old habits die hard! I was watching Kelsey Hightower’s keynote at KubCon. He made some great points about kubernetes. If you give all the devs kubectl, then it’s kind of like allowing everybody to SSH into the boxes. It’s not the way to do it!

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

March 19, 2018

What’s the luckiest thing that’s happened in your career?

via GIPHY

On more than a few occasions I’ve been asked what it’s like working remote. The inevitable followup is wow, you’re lucky. You can call it luck, but I just finished talking to 50 companies, put together proposals for all of them, and 49 said no to remote.

Lucky indeed!

Join 38,000 others and follow Sean Hull on twitter @hullsean.

But to be sure, there have certainly been events that I look back on, that have had a seminal influence on my career. I thought of three. Here they are.

1. I got a newspaper route

Yes it’s true. Way back in the dawn of time, news was delivered on paper, and those would often be delivered to your doorstep, every morning. That and for only pennies.

At the time I think I was ten years old, and I was super excited because I wanted to be part of the world. And this was really the only job you could get at that age. It required a lot of organizing. You had a book with lists of all your customers, you had to keep track of who had paid, and so forth. Some wanted to pay monthly, while others insisted on weekly.

Read: 8 questions to ask an aws expert

I was lucky not only to get the paper route, but to have parents that encouraged me in this way, and could teach me how to be responsible and reliable. Yeah those were super important early lessons.

Luck is what happens when preparation meets opportunity!

— Michael Jordan (@itsMichaelJ) October 7, 2012

When I saw that I had a captive audience, it wasn’t long before I was selling incidental services. Do you need your driveway shoveled? I got it. How about mowing your lawn or raking those leaves? Or walking your dog? I spun this into a whole bunch of side businesses.

It was exciting because I was making my own money at such a young age, and felt the lure of self-determination. I loved that feeling.

Also: The art of resistence in advice & consulting

2. Linux happened

In the early 90’s Linux came on the scene. It might seem like a meaningless blip on the radar to you, but to me it was everything. At University I worked in the computer lab, where we were the operations staff. Those systems all ran Unix. So to go home and use a windows box, it was demoralizing.

Then out of nowhere this guy from Finland started building off of Tennenbaum’s book, Minix! I had worked on that at University, so I immediately saw the implications. I mean heck why can’t all that great software run on PCs, it’s just a matter of getting the drivers working. Big vendors didn’t want to do it, but millions of hackers around the world were happy to pickup the mantle.

From there I build a tower, cobbled together hardware, memory, disk bus. Do you want to go IDE Or SCSI? Better choose a graphics card that is supported if you want to get X11 running on that. And of course an optical mouse so that it really feels like you’re sitting at a sun workstation!

“No one has as much luck around the greens as one who practices a lot.”
–Chi Chi Rodriguez

To me that was pure magic. I mean from boot up to graphical interface, the entire stack was built by people just like me. And I could look at all of it. So cool! Even better that we were fighting the good fight against Bill Gates & the borg! 🙂

Check out: When clients don’t pay

3. Meeting the $65/hr consultant

This is a funny story too. One of the first jobs I had in NYC was not a consulting gig. It was a small design shop in the late nineties. Their biggest customer was Miramax Films. So they were doing really cutting edge stuff. And a lot of cool tech too. After their lead engineer quite, I became the defacto go-to person for all tech projects. I guess you could say I was CTO of a team of 5.

For one of our projects we needed help. The CEO had won business to do some Oracle development, which I didn’t have a lot of experience in. So he hired a consultant to help out. Very nice guy, a bit older than me. In fact I think he was about as old then as I am today.

Read: Is Amazon too big to fail?

As he was a smoker, we stepped outside together at one point, and I chatted him up a bit. I was so eager to learn. I don’t recall if he shared his rate or I learned it some other way. But I was shocked and blown away. To me it seemed like an insane amount of money. I remember him saying something to me. “Don’t worry Sean, someday you’ll be consulting too, and making just as much.”

“No one is going to know or care about your failures, and neither should you. All you have to do is learn from them and those around you. [A]ll that matters in business is that you get it right once. Then everyone can tell you how lucky you are.”
–Mark Cuban

Well I am a very competitive person. I also knew that I was smarter than this guy, but he had a bit more experience than me. So from there I started sniffing around. I talked to recruiters and anyone I knew. Within two weeks, I had gotten an offer for $80/hr. Shortly after that I gave my notice.

I have to thank that guy for challenging the way he did. And I’ve never looked back since!

Related: When you have to take the fall

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

February 26, 2018

6 Devops interview questions

via GIPHY

Devops is in serious demand these days. At every meetup or tech event I attend, I hear a recruiter or startup founder talking about it. It seems everyone wants to see benefits of talented operations brought to their business.

Join 37,000 others and follow Sean Hull on twitter @hullsean.

That said the skill set is very broad, which explains why there aren’t more devs picking up the batton.

I thought it would be helpful to put together a list of interview questions. There are certainly others, but here’s what I came up with.

1. Explain the gitflow release process

As a devops engineer you should have a good foundation about software delivery. With that you should understand git very well, especially the standard workflow.

Although there are other methods to manage code, one solid & proven method is gitflow. In a nutshell you have two main branches, development & master. Developers checkout a new branch to add a feature, and push it back to development branch. Your stage server can be built automatically off of this branch.

Periodically you will want to release a new version of the software. For this you merge development to master. UAT is then built automatically off of the master branch. When acceptance testing is done, you deploy off of master to production. Hence the saying always ship trunk.

Bonus points if you know that hotfixes are done directly off the master branch & pushed straight out that way.

2. How do you provision resources?

There are a lot of tools in the devops toolbox these days. One that is great at provisioning resources is Terraform. With it you can specify in declarative code everything your application will need to run in the cloud. From IAM users, roles & groups, dynamodb tables, rds instances, VPCs & subnets, security groups, ec2 instances, ebs volumes, S3 buckets and more.

You may also choose to use CloudFormation of course, but in my experience terraform is more polished. What’s more it supports multi-cloud. Want to deploy in GCP or Azure, just port your templates & you’re up and running in no time.

It takes some time to get used to the new workflow of building things in terraform rather than at the AWS cli or dashboard, but once you do you’ll see benefits right away. You gain all the advantages of versioning code we see with other software development. Want to rollback, no problem. Want to do unit tests against your infrastructure? You can do that too!

3. How do you configure servers?

The four big choices for configuration management these days are Ansible, Salt, Chef & Puppet. For my money Ansible has some nice advantages.

First it doesn’t require an agent. As long as you have SSH access to your box, you can manage it with Ansible. Plus your existing shell scripts are pretty easy to port to playbooks. Ansible also does not require a server to house your playbooks. Simply keep them in your git repository, and checkout to your desktop. Then run ansible-playbook on the yaml file. Voila, server configuration!

4. What does testing enable?

Unit testing & integration testing are super import parts of continuous integration. As you automate your tests, you formalize how your site & code should behave. That way when you automate the deployment, you can also automate the test process. Let the software do the drudgework of making sure a new feature hasn’t broken anything on the site.

As you automate more tests, you accelerate the software development process, because you’re doing less and less manually. That means being more agile, and makes the business more nimble.

5. Explain a use case for Docker

Docker a low overhead way to run virtual machines on your local box or in the cloud. Although they’re not strictly distinct machines, nor do they need to boot an OS, they give you many of those benefits.

Docker can encapsulate legacy applications, allowing you to deploy them to servers that might not otherwise be easy to setup with older packages & software versions.

Docker can be used to build test boxes, during your deploy process to facilitate continuous integration testing.

Docker can be used to provision boxes in the cloud, and with swarm you can orchestrate clusters too. Pretty cool!

6. How is communicating relevant to Devops

Since devops brings a new process of continuous delivery to the organization, it involves some risk. Actually doing things the old way involves more risk in the long term, because things can and will break. With automation, you can recovery quicker from failure.

But this new world, requires a leap of faith. It’s not right for every organization or in every case, and you’ll likely strike a balance from what the devops holy book says, and what your org can tolerate. However inevitably communication becomes very important as you advocate for new ways of doing things.

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

February 19, 2018

Is there a serious skills shortage around devops space?

via GIPHY

As devops adoption picks up pace, the signs are everywhere. Infrastructure as code once a backwater concept, and a hoped for ideal, has become an essential to many startups.

Why might that be?

Join 37,000 others and follow Sean Hull on twitter @hullsean.

My theory is that devops enables the business in a lot of profound ways. Sure it means one sysadmin can do much more, manage a fleet of servers, and support a large user base. But it goes much deeper than that.

Being able to standup your entire dev, qa, or production environment at the click of the button transforms software delivery dramatically. It means it can happen more often, more easily, and with less risk to the business. It means you can do things like blue/green deployments, rolling out featues without any risk to the production environment running in parallel.

What kind of chops does it take?

Strong generalist skills

For starters you’ll need a pragmatist mindset. Not fanatical about one technology, but open to the many choices available. And as a generalist, you start with a familiarity with a broad spectrum of skills, from coding, troubleshooting & debugging, to performance tuning & integration testing.

Stir into the mix good operating system fundamentals, top to bottom knowledge of Unix & Linux, networking, configuration and more. Maybe you’ve built kernels, compiled packages by hand, or better yet contributed to a few open source projects yourself.

You’ll be comfortable with databases, frontend frameworks, backend technologies & APIs. But that’s not all. You’ll need a broad understanding of cloud technologies, from GCP to AWS. S3, EC2, VPCs, EBS, webservers, caching servers, load balancing, Route53 DNS, serverless lambda. Add to all of that programmable infrastructure through CloudFormation or Terraform.

Competent programmer

Although as a devop you probably won’t be doing frontend dev, you’ll need some cursory understanding of those. You should be competent at Python and perhaps Nodejs. Maybe Ruby & bash scripts. You’ll need to understand JSON & Yaml, CloudFormation & Terraform if you want to deliver IAC.

Strong sysadmin with ops mindset

These are fundamental. But what does that mean? Ops mindset is born out of necessity. Having seen failures & outages, you prioritize around uptime. A simpler stack means fewer moving parts & less to manage. Do as Martin Weiner would suggest & use boring tech.

But you’ll also need to reason about all these components. That’ll come from dozens of debug & troubleshooting sessions you’ll do through years of practice.

Understand build systems & deployment models

Build systems like CircleCI, Jenkins or Gitlab offer a way to automate code delivery. And as their use becomes more widespread knowing them becomes de rigueur. But it doesn’t end there.

With deployments you’ll have a lot to choose from. At the very simplest a single target deploy, to all-at-once, minimum in service and rolling upgrades. But if you have completely automated your dev, qa & prod infra buildout, you can dive into blue/green deployments, where you make a completely knew infra for each deploy, test, then tear down the old.

Personality to communicate across organization

I think if you’ve made it this far you will agree that the technical know-how is a broad spectrum of modern computing expertise. But you’ll also need excellent people skills to put all this into practice.

That’s because devops is also about organizational transformation. Yes devs & ops have to get up to speed on the tech, but the organization has to get on board too. Many entrenched orgs pay lip service to devops, but still do a lot of things manually. This is out of fear as much as it stands as technical debt.

But getting past that requires evangelizing, and advocating. For that a leader in the devops department will need superb people skills. They’ll communicate concepts broadly across the organization to win hearts and minds.

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

January 11, 2018

Can daily notes help you work better with clients?

via GIPHY

Years ago I was working at a customer for a few weeks. There was some confusion as to what was going on, in terms of progress. Things weren’t moving as quickly as they expected.

After a lot of back and forth, I suggested I could provide detailed notes of what I had done.

Join 38,000 others and follow Sean Hull on twitter @hullsean.

After I put together my in-depth notes the customer was really happy. It seems these notes had highlighted a few problems that they didn’t know about. What’s more they even highlighted some people issues, where communicate was blocked. Whats more the notes underlined what I was doing, and this really improved the customers confidence in the work product.

1. Visibility

Keeping daily notes is a habit I found useful over and over again. If your client or customer comes to you and says, why are we paying $X, you can provide the notes as a detailed explanation of what they have gotten for their money.

2. Transparency

Transparency is a door that swings both ways. As I mentioned above it can be great when the customer is not sure how much work was done, or what the bill is for. But it can also highlight things they may not want done. For instance perhaps you were investigating a problem authenticating to a server. You determined that it was an important piece.

When the customer sees this in your notes they may say “Oh we don’t need to deal with that system. Please leave it alone” or they may say “We actually have Rakesh available to help us with that piece, so please communicate with him and he can resolve that”.

Related: When clients don’t pay

3. Trust

Most important of all, keeping detailed notes helps build trust. Many customers, hiring managers & CTOs are not command-line technical. And that’s perfectly normal. However looking over a long list of notes like these provides great insight to them as to what you do from day-to-day.

Do they need to know what every line means? No. But the visibility goes a long long way toward building trust in the consultant client relationship.

Week 1 April 1 – April 10

Here’s a sample of the kind notes I keep. Actually they cover a ten day period, but that’s because the initial day was towards the end of a week.

Friday April 1st
o coord with Jake on getting started
o dropbox for password, creds & server docs
o reviewing system network diagram
o reviewing techlist excel doc
– techlist
– server list & access
– database access
– projects -old
o reviewing systems access.docx
o testing AWS login credentials
– issue with permissions
– coordinating with Jake on Admin access
o testing AWS creds again
– access to all AWS services
– IAM for seanhull user
– enabling MFA for user
o questions for outgoing op Roger

Sat April 2nd (no hours)

Sun April 3rd (no hours)

Mon April 4th
o coord with Jake to get onboarded
o sending W9 form to Acme Inc.
o setup slack
o plan for today
– review aws servers
– review dg servers
– questions for Roger
– review docs
o coord with Roger on VPN access
– reach out to Larry
– emailed Larry CC Jake
– Larry requests Acme access CC to mgmt
– turns out VPN access isn’t required
– can just whitelist IP inside the relevant security groups
– coord with J, going ahead to add whitelist 1.2.3.4/32
o updating Acmemedia-sandbox security group
– trying to reach host, coord with Roger
– asked to drop ssh key onto servers
– asked about .ssh/config file – Did you get from Jake?
– found the AWS PEM folder that I overlooked 🙂
o configuring .ssh/config file
– copying up to iheavy.com
– setting permissions 600 on pem files
– ssh to sandbox successful!!
o adding whitelist to Acmemedia-prod security group
o updating Jake – access is working

Tue April 5th
o coord with Jake on todo list for today
o verifying mysql access
– review security groups
– no whitelisted IPs
– can reach from webserver?
– test db1 MySQL access via webserver, OK
– test db2 MySQL access via webserver OK
o reviewing monitoring system
– testing nagios access
– locating configurations
– reviewing dashboard
– understanding tests
– down system db1 – 108 days – why?
– down system p1 – LB1 sailthru check down for 85 days why?
– down staging – 174 days why?
– emailed nagios questions to Roger
– request to add me to nagios notifications group
o coord with Roger on questions
– nagios setup & stopped checks
– add to admin group
o github access for sandbox details doc
o login to Acmemedia wp
– check list of 25 plugins
– review recent backups on abc (8)
o login to DDD wordpress
– check list of 33 plugins
– review recent backups in abc (8)
o login to EEE wordpress
– check list of 31 plugins
– review recent backups in abc (37)
o login to FFF wordpress
– check list of 35 plugins
– review recent backups in abc (8)
o login to DDD
o login to EEE
o login to FFF
o emailed Roger – request details about Glasgow server
o review various Acme github pages

Also: The art of resistance or when you have to be the bad guy

Wed April 6th
o coord with Jake on todos for today
o reviewing github pages docs on various system processes
– git deployment server page
– git deployment process
– new deploy process Nov 2015
– wiki pages are a bit sparse overall
o tested jenkins login
– found API cache clear
– found varnish cache clear
o understand separation of dev & production
o digging into Jenkins docs
o understanding build process
o tried login to EEEv2 wp login, don’t have pass
– coordinating with Jake on that login
o checking on nfs disk full nagios alert
– can’t reach box
– notified Jake & Roger via slack
– slack with Lester
– yes nfs01 space 90% is normal
– new launch of EEE tomorrow & old stuff will be deleted then
o updating nfs security group
– ssh login working now.
o getting diskspace error on prod04
– messaged Lester, related to EEE launch tonight
o email from Jake – local dev & test environment setups are slow
– very overengineered for simple wordpress site
– not using multisites, so have FOUR SEPARATE setups
– different plugins on each install
– four sets of logins
– four places to update
– four places to test/qa
– migration may be complex based on custom Acme plugins
– shortcodes compatability across four sites
– not using ithemes security plugin
o discuss with Lester on slack
– API is hosted on datagram
– single point of failure for the site currently
– outage there would take the site down
– migrate to AWS using internal loadbalancer & webservers in 2 AZs

Thu April 7th
o call with Jake on EEEv2 launch today
– general observations of Acme sites & architecture
o reviewing access.Acmemedia.com
o discuss with Jake
– hosting media files on S3 vs nfs
– using multisite
– using wordpress through API only
– javascript based static site builder
– moving API to amazon EC2
– create slave MySQL db of master MySQL currently in datadotnet
o discuss with Roger
– launch plan
– two vhosts new.EEE.com
– old.EEE.com
– simply restart apache to enable switch
– refresh maxCDN after launch
o review EEEv2 deploy steps
– pre-deploy steps
– DNS for old.EEE.com
– add vhosts EEEv2.conf
– restart apache
– restart varnish
– clear maxcdn
o verified login to access.Acmemedia.com
– API log is in /var/log/httpd/production-access.log
– login as sandy & root
o not able to login to dashboard.Acmemedia.com
– tried admin & pass in datagram docs

o meeting onsite with Jake & Roger
– discuss deployment process
– discuss legacy systems
– discuss NFS vs S3 for media files
– discuss plugins & management
– discuss wordpress version upgrade process
– discuss plugin version upgrade process
– discuss Jenkins access, configs, success & error logs
– discuss managing secrets file
– script that takes webserver out of load balancer while apache restarting
o met Rachel, Louis, Lester, Rick, Stuart, Jack

Fri April 8th
o testing Acme stage build
o emailed Roger further questions
– where is secrets file configuration & process
– composer is PHP dependency management
– what are the steps to upgrade plugin only
o summarizing & notes on Acme
o put together steps for complete firedrill
– questions for Roger, requesting help with process
– build webserver with varnish & apache
– should setup separate NFS server
– should use Acmemedia.com bc it uses API heavily
– setup copy of API server & db
– setup mysql instance for wordpress
– setup amazon cloudfront for content
o outline additional questions for Roger
– how to upgrade plugin only
– composer for php dependency management
– how are secrets files managed & deployed outside developer access
o secrets management
– asked Roger for clarificaiton
o plugin-only installs
– reviewed jenkins configs
– various questions to Roger
– composer:install seems to be the key change (not just deploy which does all?)
– why is STAGING PLUGIN DEPLOY for ORIM different?
o what happens when github account is disabled!!
– jenkins changes for new github deploy account
– THIS WOULD BREAK ALL DEPLOYS & CI/CD pipeline
– capistrano changes?
– any other changes on sandbox
– any other dependencies for Roger github?
o email step-by-step outline to add a plugin
– reviewing steps with Roger
– making sure no missing pieces

Sat April 9th (1 off-hour)
o receiving nagios alert for p1
o emailed Roger, Jake about issue
o slack messaged Jake
o raises question about off-hours coverage

Sun April 10th (2 off-hours)

o p1 still throwing errors
o coordinating with Lester & Ralph on Slack
– reiterated this is *not* an issue with NFS
– because of large number of nagios alerts, p1 lost in the shuffle
– p1 is new error, 97% so more dire than the NFS issue
– Lester attempting to login, fails because of AWS security group
– adding his *own* home IP as whitelist (devs have access to AWS console)
– first time logging in from home?
– Lester deleted old DDD logfiles to clear up 1.2G
– plan to touch base again tomorrow about issue
o emailing Jake about status
o questions for Jake
– how to manage on-call & alerts
– how to manage developer access
– Roger mentioned secrets files are not shared with devs
o Lester questions, comments on servers & diskspace

Related: When you have to take the fall

Overhead costs

Evolving scope

Horse Trading Skills

Underbidding

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

1. What is it for?

2. How to setup the VPC

3. How to add the bastion

4. Add an EC2 instance

5. Testing

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

1. Technical discovery

2. Process discovery

3. Team discovery

4. Tools discovery

5. In Summary

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

1. What not to do?

2. Do socialize

3. Build your reputation

4. Positioning

5. What you might find

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

1. Manage image bloat

2. Orchestrate

3. Registries & Deployment

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

1. How would you load test?

2. Why is Amazon S3’s 99.999999999% promise *not* enough??

3. Why is continuous integration not about tools?

4. What can VPC peering do for you?

5. Why go with a New York based resource?

6. What are some common antipatterns in the cloud

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

1. I got a newspaper route

2. Linux happened

3. Meeting the $65/hr consultant

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

1. Explain the gitflow release process

2. How do you provision resources?

3. How do you configure servers?

4. What does testing enable?

5. Explain a use case for Docker

6. How is communicating relevant to Devops

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Strong generalist skills

Competent programmer

Strong sysadmin with ops mindset

Understand build systems & deployment models

Personality to communicate across organization

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

1. Visibility

2. Transparency

3. Trust

Week 1 April 1 – April 10

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

2. Why is Amazon S3’s 99.999999999% promise not enough??