Is there a serious skills shortage around devops space?

via GIPHY

As devops adoption picks up pace, the signs are everywhere. Infrastructure as code once a backwater concept, and a hoped for ideal, has become an essential to many startups.

Why might that be?

Join 37,000 others and follow Sean Hull on twitter @hullsean.

My theory is that devops enables the business in a lot of profound ways. Sure it means one sysadmin can do much more, manage a fleet of servers, and support a large user base. But it goes much deeper than that.





Being able to standup your entire dev, qa, or production environment at the click of the button transforms software delivery dramatically. It means it can happen more often, more easily, and with less risk to the business. It means you can do things like blue/green deployments, rolling out featues without any risk to the production environment running in parallel.

What kind of chops does it take?

Strong generalist skills

For starters you’ll need a pragmatist mindset. Not fanatical about one technology, but open to the many choices available. And as a generalist, you start with a familiarity with a broad spectrum of skills, from coding, troubleshooting & debugging, to performance tuning & integration testing.

Stir into the mix good operating system fundamentals, top to bottom knowledge of Unix & Linux, networking, configuration and more. Maybe you’ve built kernels, compiled packages by hand, or better yet contributed to a few open source projects yourself.

You’ll be comfortable with databases, frontend frameworks, backend technologies & APIs. But that’s not all. You’ll need a broad understanding of cloud technologies, from GCP to AWS. S3, EC2, VPCs, EBS, webservers, caching servers, load balancing, Route53 DNS, serverless lambda. Add to all of that programmable infrastructure through CloudFormation or Terraform.

Related: 30 questions to ask a serverless fanboy

Competent programmer

Although as a devop you probably won’t be doing frontend dev, you’ll need some cursory understanding of those. You should be competent at Python and perhaps Nodejs. Maybe Ruby & bash scripts. You’ll need to understand JSON & Yaml, CloudFormation & Terraform if you want to deliver IAC.

Related: Does a 4-letter-word divide dev & ops?

Strong sysadmin with ops mindset

These are fundamental. But what does that mean? Ops mindset is born out of necessity. Having seen failures & outages, you prioritize around uptime. A simpler stack means fewer moving parts & less to manage. Do as Martin Weiner would suggest & use boring tech.

But you’ll also need to reason about all these components. That’ll come from dozens of debug & troubleshooting sessions you’ll do through years of practice.

Related: How to hire a developer that doesn’t suck

Understand build systems & deployment models

Build systems like CircleCI, Jenkins or Gitlab offer a way to automate code delivery. And as their use becomes more widespread knowing them becomes de rigueur. But it doesn’t end there.

With deployments you’ll have a lot to choose from. At the very simplest a single target deploy, to all-at-once, minimum in service and rolling upgrades. But if you have completely automated your dev, qa & prod infra buildout, you can dive into blue/green deployments, where you make a completely knew infra for each deploy, test, then tear down the old.

Related: Is AWS too complex for small dev teams?

Personality to communicate across organization

I think if you’ve made it this far you will agree that the technical know-how is a broad spectrum of modern computing expertise. But you’ll also need excellent people skills to put all this into practice.

That’s because devops is also about organizational transformation. Yes devs & ops have to get up to speed on the tech, but the organization has to get on board too. Many entrenched orgs pay lip service to devops, but still do a lot of things manually. This is out of fear as much as it stands as technical debt.

But getting past that requires evangelizing, and advocating. For that a leader in the devops department will need superb people skills. They’ll communicate concepts broadly across the organization to win hearts and minds.

Related: Will Microservices just die already?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

How is automation impacting the dba role?

via GIPHY

I was at a dinner party recently, and talking with some colleagues. I had worked with them years back on Oracle systems.

One colleague Maria said she really enjoyed my newsletter.

Join 38,000 others and follow Sean Hull on twitter @hullsean.

She went on to say how much has changed in the last decade. We talked about how the database administrator, as a career role, wasn’t really being hired for much these days. Things had changed. Evolved a lot.

How do you keep up with all the new technology, she asked?

I went on to talk about Amazon RDS, EC2, lambda & serverless as really exciting stuff. And lets not forget terraform (I wrote a howto on terraform), ansible, jenkins and all the other deployment automation technologies.





We talked about Redshift too. It seems to be everywhere these days and starting to supplant hadoop as the warehouse of choice for analytics.

It was a great conversation, and afterward I decided to summarize my thoughts. Here’s how I think automation and the cloud are impacting the dba role.

My career pivots

Over the years I’ve poured all those computer science algorithms, coding & hardware skills into a lot of areas. Tools & popular language change. Frameworks change. But solid deductive reasoning remains priceless.

o C++ Developer

Fresh out of college I was doing Object Oriented Programming on the Macintosh with Codewarrior & powerplant. C++ development is no joke, and daily coding builds strength in a lot of areas. Turns out he application was a database application, so I was already getting my feet wet with databases.

o Jack of all trades developer & Unix admin

One type of job role that I highly recommend early on is as a generalist. At a small startup with less than ten employees, you become the primary technology solutions architect. So any projects that come along you get your hands dirty with. I was able to land one of these roles. I got to work on Windows one day, Mac programming another & Unix administration & Oracle yet another day.

o Oracle DBA

The third pivot was to work primarily on Oracle. I attended Oracle conferences & my peers were Oracle admins. Interestingly, many of the Oracle “experts” came from more of a business background, not computer science. So to have a more technical foundation really made you stand out.

For the startups I worked with, I was a performance guru, scalability expert. Managers may know they have Oracle in the mix, but ultimately the end goal is to speed up the website & make the business run. The technical nuts & bolts of Oracle DBA were almost incidental.

o MySQL & Postgres

As Linux matured, so did a lot of other open source projects. In particular the two big Open Source databases, MySQL & Postgres became viable.

Suddenly startups were willing to put their businesses on these technologies. They could avoid huge fees in Oracle licenses. Still there were not a lot of career database experts around, so this proved a good niche to focus on.

o RDS & Redshift on Amazon Cloud

Fast forward a few more years and it’s my fifth career pivot. Amazon Web Services bursts on the scene. Every startup is deploying their applications in the cloud. And they’re using Amazon RDS their managed database service to do it. That meant the traditional DBA role was less crucial. Sure the business still needed data expertise, but usually not as a dedicated role.

Time to shift gears and pour all of that Linux & server building experience into cloud deployments & migrating to the cloud.

o Devops, data, scalability & performance

Now of course the big sysadmin type role is usually called an SRE or Devops role. SRE being site reliability engineer. New name but many of the same responsibilities.

Now though infrastructure as code becomes front & center. Tools like CloudFormation & Terraform, plus Ansible, Chef & Jenkins are all quite mature, and being used everywhere.

Checkout your infrastructure code from git, and run terraform apply. And minutes later you have rebuilt your entire stack from bare metal to fully functioning & autoscaling application. Cool!

Related: 30 questions to ask a serverless fanboy

How I’ve steered DBA skills

There’s no doubt that data expertise & management skills are still huge. But the career role of database administrator has evolved quite a bit.

Related: 5 surprising features of Amazon Lambda serverless computing

Pros of automation & managing databases

For DBAs who are looking at the cloud from the old way of doing things, there’s a lot to love about it.

Automation brings repeatability to work & jobs. This is great. It raises the bar & makes us more professional, reducing manual processes & mistakes.

Infrastructure as code is self documenting. It means we have a better idea of day-to-day processes, and can more easily handoff to new folks as we change roles or companies.

Related: Why generalists are better at scaling the web

Cons of automation & databases

However these days cloud, automation & microservices have brought a lot of madness too! Don’t believe me check out this piece on microservice madness.

With microservices you have more databases across the enterprise, on more platforms. How do you restore all at the same time? How do you do point-in-time recovery? What if your managed service goes down?

Migration scripts have become popular to make DDL changes in the database. Going forward (adding columns or tables) is great. But should we be letting our deployment automation roll *BACK* DDL changes? Remember that deletes data right? 🙂

What about database drop & rebuild? Or throwing databases in a docker container? No bueno. But we’re seeing this more and more. New performance problems are cropping up because of that.

What about when your database upgrades automatically? Remember when you use a managed service, it is build for 1000 users, not one. So if your use case is different you may struggle.

In my experience upgrading RDS was a nightmare. Database as a service upgrades lack visibility. You don’t have OS or SSH access so you can’t keep track of things. You just simply wait.

No longer do we have “zero downtime”. With amazon RDS you have guarenteed downtime upgrades. No seriously.

As the field of databases fragments, we are wearing many more hats. If you like this challenge & enjoy being a generalist, you may feel at home here. But it is a long way from one platform one skill set career path.

Also fragmented db platforms means more complex recovery. I can’t stress this enough. It would become practically impossible to restore all microservices, all their underlying databases & all systems to one single point in time, if you need to.

Related: Is upgrading Amazon RDS like a sh*t storm that will not end?

DBAs, it’s time to step up and pivot

As the DBA role evolves, it also brings great opportunity. For those with solid database & data skills are sorely in need at startups and many fortune 500 organizations.

What I’m seeing is that organizations have lost much of the discipline they had as separate dba or operations departments. Schemaless databases have proliferated, and performance has suffered.

All these are more complex now, but strong DBA, performance & troubleshooting skills are needed now more than ever.

Related: The art of resistance in tech consulting

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

How can I get started with lambda and nodejs in 5 minutes?

via GIPHY

I know these learn-to-do-x in 5 minutes type articles are a dime a dozen. But it’s true, we’re short on time, and we just wanna jump in. So let’s go!

Join 38,000 others and follow Sean Hull on twitter @hullsean.

Rather than go the old route of doing everything manually, and struggling, we’re going to give ourselves a skeleton to start with.





Enter, serverless framework. What’s it do? It’s a command line tool written in nodejs, which allows you to create a lambda project from a template.

From there you edit a yml file to tell serverless what to build & how. Then you put your code inside of the handler.js file. Sounds simple right?

1. Create

If you haven’t already done it, install nodejs. There are lots of docs on the interwebs. For mac users, “brew install node” does the trick!

Next install the serverless package.

$ npm install serverless

Great! If you got dependency errors, get digging. Those moments of troubleshooting & patience teach you a lot. 🙂

Ok, now let’s kick the tires. We’ll create our new project.

$ serverless create --template aws-nodejs --path myEndpoint
$ cd myEndpoint

Related: 30 questions to ask a serverless fanboy

2. Edit serverless.yml

service: myEndpoint

frameworkVersion: ">=1.1.0 <2.0.0"

provider:
  name: aws
  runtime: nodejs4.3

functions:
  currentTime:
    handler: handler.endpoint
    events:
      - http:
          path: ping
          method: get

Ok, what are we looking at here? Framework is the version of the serverless framework. Provider is aws, because serverless is attempting to build cross-platform support. You may also use azure, openwhisk, google cloud functions etc. Runtime is your language.

Under functions, our main one is currentTime. handler tells serverless framework what code to matchup with your function name. And finally events tell serverless about the API endpoint to configure.

There's a lot of magic going on under the hood. The serverless framework us using CloudFormation to build things in the background for you. CloudFormation is like Latin, it is a foundational construct to the entire AWS world. You can formalize any object, from servers to sqs queues, dynamodb tables, security groups, IAM users, S3 buckets, ebs volumes etc etc. You get the idea.

Want to see what serverless did? Head over to your aws dashboard, navigate to CloudFormation. You should see a new stack there called myEndpoint-dev. Scroll down and click the "Template" tab. You'll see the exact JSON code in all it's gory detail!

Related: 5 surprising features of Amazon Lambda serverless computing

3. Edit handler.js

Next up let's add a bit of code.

'use strict';

// return the current time in JSON format
module.exports.endpoint = (event, context, callback) => {
  const response = {
    statusCode: 200,
    body: JSON.stringify({
      message: `Hello, the current time is ${new Date().toTimeString()}.`,
    }),
  };

  callback(null, response);
};

Whenever this function gets called, we'll just return the current time. Pretty self explanatory.

Related: Are you getting errors building lambda functions? I got you covered!

4. Deploy!

Now the fun party. Let's deploy the code.

$ serverless deploy

Simple command, but it's doing a lot of work. Serverless framework is packaging up your nodejs code into a zip file and uploading it to aws for you. You should see some output telling you what happened.

$ serverless deploy
Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Uploading CloudFormation file to S3...
Serverless: Uploading artifacts...
Serverless: Uploading service .zip file to S3 (1.2 KB)...
Serverless: Validating template...
Serverless: Updating Stack...
Serverless: Checking Stack update progress...
........................
Serverless: Stack update finished...
Service Information
service: myEndpoint
stage: dev
region: us-east-1
stack: myEndpoint-dev
api keys:
  None
endpoints:
  GET - https://ABCDEFGHIJK.execute-api.us-east-1.amazonaws.com/dev/ping
functions:
  currentTime: myEndpoint-dev-currentTime
$

Related: Is Amazon too big to fail?

5. Test

Awesome, now it's time to make sure it's working.

You can invoke the function directly using serverless' "invoke" command like this:

$ serverless invoke --function currentTime --log
{
    "statusCode": 200,
    "body": "{\"message\":\"Hello, the current time is 20:46:02 GMT+0000 (UTC).\"}"
}
--------------------------------------------------------------------
START RequestId: ed5e427c-fe22-11e7-90cc-a1fe66d674ce Version: $LATEST
END RequestId: ed5e427c-fe22-11e7-90cc-a1fe66d674ce
REPORT RequestId: ed5e427c-fe22-11e7-90cc-a1fe66d674ce	Duration: 0.67 ms	Billed Duration: 100 ms 	Memory Size: 1024 MB	Max Memory Used: 21 MB	


$

But we created an API endpoint didn't we? Yep. You can hit that. If you have a browser open, go ahead and copy/past the url listed in the endpoints section of your deploy process.

You can also use curl like this:

$ curl https://ABCDEFGHIJK.execute-api.us-east-1.amazonaws.com/dev/ping
{"message":"Hello, the current time is 20:46:18 GMT+0000 (UTC)."}
$ 

Related: Is Amazon Web Services too complex for small dev teams?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don't work with recruiters

Can misfits teach us a thing or two about innovation?

I just finished reading Alexa Clay & Kyra Maya Phillips tour de force, The Misfit Economy.

(Yes that’s an affiliate link. The first one I’ve ever posted on this blog. If you like the book, please 🙂 buy through my link. )
I have to admit I was surprised & delighted by the book.

Join 38,000 others and follow Sean Hull on twitter @hullsean.

Alexa & Kyra offer us a tantalizing question. Could it be that we could learn a lot from oddball innovators at the edge of the economy? When I say edge, I really mean it. She interviews Sam Hostetler who is building a business around milking camels, and then there’s Abdi Hasan a pirate from Galkayo northern Somalia. Yeah really! Or what about the German copycats Wimdu who built a complete replica of Airbnb by reverse engineering it.

1. Hack the cold call

Take the example of Lance Weiler. Early on the industry was very against digital. They didn’t see it as really making films.

“Part of [Lance] Weiler’s success was due to his ability to work the system. He wrote letters to major production companies telling them he wanted to make the first digital motion picture. After he didn’t hear back, he took a page from the con man’s handbook and wrote the same letters but intentionally misaddressed them so they were sent to the wrong companies. Sony for example would get a letter intended for Barco.”

He was later able to bring digital projection to Cannes & Sundance!

“For Weiler his big epiphany was when he realized he could be creative across all of it [the business]. Not just in the art product, but in financing, distribution, and business aspects of artistic production.”

Related: The art of resistence or when you have to be the bad guy

2. Copy the product

The german brothers Oliver, Marc & Alexander Samwer make a superb example of how copying can bring building prowess to compete against innovators that were first to market.

“in 1998 Marc Samwer had an instinct that eBay would thrive in the German market… his brothers agreed… they contacted eBay via email numerous times, recommending that the company replicate it’s platformin Germany. Claiming that eBay failed to respond, the brother’s started their own German-language auction site, Alando, which was then purchased by eBay for 38 million euros (over $50 million) only 100 days after it’s debut. Had the Samwers not copied, eBay might have remained complacent, not realizing its potential within the german market.”

Although not mentioned in the book, Inditex the wildly successful firm behind fashion brand Zara did much the same thing to the fashion industry. By mastering the supply chain, they enabled their company to take designs from the runway & replicate them, turning designs into real clothing in stores, in just two weeks! And indeed they really do replicate, borrow & straight copy those designs from what they see at fashion week. Sad & brilliant at the same time.

Related: When you have to take the fall

3. Don’t forget to hustle


“In the lexicon of the Misfit Economy, we define “hustle” as making something out of nothing. To move fast, to trade one thing for another, and to proactively create your own opportunities rather than waiting for opportunity to come your way. To hustle means getting your hands dirty, being lean and facile, working hard, being resourceful and resilient, and showing or having gumption, chutzpah, or mojo.”

And after all, isn’t that everything the startup industry aspires to? Agile teams? Growth hackers? Scrappy startups & innovation?

Related: When clients don’t pay

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Can daily notes help you work better with clients?

via GIPHY

Years ago I was working at a customer for a few weeks. There was some confusion as to what was going on, in terms of progress. Things weren’t moving as quickly as they expected.

After a lot of back and forth, I suggested I could provide detailed notes of what I had done.

Join 38,000 others and follow Sean Hull on twitter @hullsean.

After I put together my in-depth notes the customer was really happy. It seems these notes had highlighted a few problems that they didn’t know about. What’s more they even highlighted some people issues, where communicate was blocked. Whats more the notes underlined what I was doing, and this really improved the customers confidence in the work product.

1. Visibility

Keeping daily notes is a habit I found useful over and over again. If your client or customer comes to you and says, why are we paying $X, you can provide the notes as a detailed explanation of what they have gotten for their money.

Related: Are generalists better at scaling the web?

2. Transparency

Transparency is a door that swings both ways. As I mentioned above it can be great when the customer is not sure how much work was done, or what the bill is for. But it can also highlight things they may not want done. For instance perhaps you were investigating a problem authenticating to a server. You determined that it was an important piece.

When the customer sees this in your notes they may say “Oh we don’t need to deal with that system. Please leave it alone” or they may say “We actually have Rakesh available to help us with that piece, so please communicate with him and he can resolve that”.

Related: When clients don’t pay

3. Trust

Most important of all, keeping detailed notes helps build trust. Many customers, hiring managers & CTOs are not command-line technical. And that’s perfectly normal. However looking over a long list of notes like these provides great insight to them as to what you do from day-to-day.

Do they need to know what every line means? No. But the visibility goes a long long way toward building trust in the consultant client relationship.

Related: 5 conversational ways to evaluate great consultants

Week 1 April 1 – April 10

Here’s a sample of the kind notes I keep. Actually they cover a ten day period, but that’s because the initial day was towards the end of a week.

Friday April 1st
o coord with Jake on getting started
o dropbox for password, creds & server docs
o reviewing system network diagram
o reviewing techlist excel doc
– techlist
– server list & access
– database access
– projects -old
o reviewing systems access.docx
o testing AWS login credentials
– issue with permissions
– coordinating with Jake on Admin access
o testing AWS creds again
– access to all AWS services
– IAM for seanhull user
– enabling MFA for user
o questions for outgoing op Roger

Sat April 2nd (no hours)

Sun April 3rd (no hours)

Mon April 4th
o coord with Jake to get onboarded
o sending W9 form to Acme Inc.
o setup slack
o plan for today
– review aws servers
– review dg servers
– questions for Roger
– review docs
o coord with Roger on VPN access
– reach out to Larry
– emailed Larry CC Jake
– Larry requests Acme access CC to mgmt
– turns out VPN access isn’t required
– can just whitelist IP inside the relevant security groups
– coord with J, going ahead to add whitelist 1.2.3.4/32
o updating Acmemedia-sandbox security group
– trying to reach host, coord with Roger
– asked to drop ssh key onto servers
– asked about .ssh/config file – Did you get from Jake?
– found the AWS PEM folder that I overlooked 🙂
o configuring .ssh/config file
– copying up to iheavy.com
– setting permissions 600 on pem files
– ssh to sandbox successful!!
o adding whitelist to Acmemedia-prod security group
o updating Jake – access is working

Tue April 5th
o coord with Jake on todo list for today
o verifying mysql access
– review security groups
– no whitelisted IPs
– can reach from webserver?
– test db1 MySQL access via webserver, OK
– test db2 MySQL access via webserver OK
o reviewing monitoring system
– testing nagios access
– locating configurations
– reviewing dashboard
– understanding tests
– down system db1 – 108 days – why?
– down system p1 – LB1 sailthru check down for 85 days why?
– down staging – 174 days why?
– emailed nagios questions to Roger
– request to add me to nagios notifications group
o coord with Roger on questions
– nagios setup & stopped checks
– add to admin group
o github access for sandbox details doc
o login to Acmemedia wp
– check list of 25 plugins
– review recent backups on abc (8)
o login to DDD wordpress
– check list of 33 plugins
– review recent backups in abc (8)
o login to EEE wordpress
– check list of 31 plugins
– review recent backups in abc (37)
o login to FFF wordpress
– check list of 35 plugins
– review recent backups in abc (8)
o login to DDD
o login to EEE
o login to FFF
o emailed Roger – request details about Glasgow server
o review various Acme github pages

Also: The art of resistance or when you have to be the bad guy

Wed April 6th
o coord with Jake on todos for today
o reviewing github pages docs on various system processes
– git deployment server page
– git deployment process
– new deploy process Nov 2015
– wiki pages are a bit sparse overall
o tested jenkins login
– found API cache clear
– found varnish cache clear
o understand separation of dev & production
o digging into Jenkins docs
o understanding build process
o tried login to EEEv2 wp login, don’t have pass
– coordinating with Jake on that login
o checking on nfs disk full nagios alert
– can’t reach box
– notified Jake & Roger via slack
– slack with Lester
– yes nfs01 space 90% is normal
– new launch of EEE tomorrow & old stuff will be deleted then
o updating nfs security group
– ssh login working now.
o getting diskspace error on prod04
– messaged Lester, related to EEE launch tonight
o email from Jake – local dev & test environment setups are slow
– very overengineered for simple wordpress site
– not using multisites, so have FOUR SEPARATE setups
– different plugins on each install
– four sets of logins
– four places to update
– four places to test/qa
– migration may be complex based on custom Acme plugins
– shortcodes compatability across four sites
– not using ithemes security plugin
o discuss with Lester on slack
– API is hosted on datagram
– single point of failure for the site currently
– outage there would take the site down
– migrate to AWS using internal loadbalancer & webservers in 2 AZs

Thu April 7th
o call with Jake on EEEv2 launch today
– general observations of Acme sites & architecture
o reviewing access.Acmemedia.com
o discuss with Jake
– hosting media files on S3 vs nfs
– using multisite
– using wordpress through API only
– javascript based static site builder
– moving API to amazon EC2
– create slave MySQL db of master MySQL currently in datadotnet
o discuss with Roger
– launch plan
– two vhosts new.EEE.com
– old.EEE.com
– simply restart apache to enable switch
– refresh maxCDN after launch
o review EEEv2 deploy steps
– pre-deploy steps
– DNS for old.EEE.com
– add vhosts EEEv2.conf
– restart apache
– restart varnish
– clear maxcdn
o verified login to access.Acmemedia.com
– API log is in /var/log/httpd/production-access.log
– login as sandy & root
o not able to login to dashboard.Acmemedia.com
– tried admin & pass in datagram docs

o meeting onsite with Jake & Roger
– discuss deployment process
– discuss legacy systems
– discuss NFS vs S3 for media files
– discuss plugins & management
– discuss wordpress version upgrade process
– discuss plugin version upgrade process
– discuss Jenkins access, configs, success & error logs
– discuss managing secrets file
– script that takes webserver out of load balancer while apache restarting
o met Rachel, Louis, Lester, Rick, Stuart, Jack

Fri April 8th
o testing Acme stage build
o emailed Roger further questions
– where is secrets file configuration & process
– composer is PHP dependency management
– what are the steps to upgrade plugin only
o summarizing & notes on Acme
o put together steps for complete firedrill
– questions for Roger, requesting help with process
– build webserver with varnish & apache
– should setup separate NFS server
– should use Acmemedia.com bc it uses API heavily
– setup copy of API server & db
– setup mysql instance for wordpress
– setup amazon cloudfront for content
o outline additional questions for Roger
– how to upgrade plugin only
– composer for php dependency management
– how are secrets files managed & deployed outside developer access
o secrets management
– asked Roger for clarificaiton
o plugin-only installs
– reviewed jenkins configs
– various questions to Roger
– composer:install seems to be the key change (not just deploy which does all?)
– why is STAGING PLUGIN DEPLOY for ORIM different?
o what happens when github account is disabled!!
– jenkins changes for new github deploy account
– THIS WOULD BREAK ALL DEPLOYS & CI/CD pipeline
– capistrano changes?
– any other changes on sandbox
– any other dependencies for Roger github?
o email step-by-step outline to add a plugin
– reviewing steps with Roger
– making sure no missing pieces

Sat April 9th (1 off-hour)
o receiving nagios alert for p1
o emailed Roger, Jake about issue
o slack messaged Jake
o raises question about off-hours coverage

Sun April 10th (2 off-hours)

o p1 still throwing errors
o coordinating with Lester & Ralph on Slack
– reiterated this is *not* an issue with NFS
– because of large number of nagios alerts, p1 lost in the shuffle
– p1 is new error, 97% so more dire than the NFS issue
– Lester attempting to login, fails because of AWS security group
– adding his *own* home IP as whitelist (devs have access to AWS console)
– first time logging in from home?
– Lester deleted old DDD logfiles to clear up 1.2G
– plan to touch base again tomorrow about issue
o emailing Jake about status
o questions for Jake
– how to manage on-call & alerts
– how to manage developer access
– Roger mentioned secrets files are not shared with devs
o Lester questions, comments on servers & diskspace

Related: When you have to take the fall

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

What have I learned in 10 years of blogging?

via GIPHY

I was just reading Andrew Chen’s latest posting, where he distills many of the things he’s learned from blogging over a decade.

Join 38,000 others and follow Sean Hull on twitter @hullsean.

This reminded me that I’ve been blogging that long as well. And to be sure it has brought great benefits. In the way that public speaking gives you visibility, but also forces you to communicate better, form your voice, and so on.

All the great things you gain by talking to other people, and getting into the conversation.

1. Understand your audience

I struggled with this when I first started blogging. As any engineer might approach things, I thought I should publish technical material. What better way to show what I know. And further how I can help a customer.

What I didn’t realize is that all of your readers aren’t technical. So it goes a long way if you can appeal to a broader audience.

I found that my readers fell into a few big categories.

1. Fellow engineers & peers
2. Hiring managers & startup CTOs
3. Recruiters & other publishers

This really helped me divide up the types of content I would write, some directed towards each of the different audiences.

Related: Why does Reddit CTO Martin Weiner advocate boring tech?

2. Tell your story

I’ve written often about why I wrote the book on Oracle. In it I outlined a long arc of datacenter evolution which started with the maturity of Linux, and today provides the bedrock of the cloud that is Amazon Web Services among others.

What this also allowed me to do is tell my own history.

Related: 5 reasons devops should blog

3. Form your voice

Forming your voice is different than speaking to specific audiences. It’s about having opinions & getting into the line of fire. Being passionate about a subject, you’re sure to care & sit on one side or the other of a particular argument.

For example I argued the Android ecosystem was broken. Although Google has fixed some of these problems, many remain as a symptom of the platform itself.

I also argued with Fred Wilson’s estimation of Apple being overvalued. At the time in May 2014 the price was at $85. Now it sits comfortably at $177.

Related: How to hire a developer that doesn’t suck

4. Put yourself out there

Putting yourself out there isn’t easy. You’ll be open to criticism. And sometimes you’ll be wrong. But by challenging yourself in this way you’ll grow too. And prospects will notice this. More than engineering might, and power at the keyboard, your perspective of what’s happening in computing generally, and what is on the horizon is invaluable to customers.

Related: 30 questions to ask a serverless fanboy

5. Learn & Share

Writing howtos is a great challenge too. By forcing yourself to teach something, you in turn learn the material better. You become better at executing, and formulating solutions.

As you share knowledge, you’ll also learn from others. As the disqus.com comments on my site can attest. Sure you get much of this same value from having an active account on Reddit.com, but your own real estate carries even more weight for your personal brand.

Related: Why you should always be publishing

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Is Alex Hudson right that software architecture is failing?

via GIPHY

I read Hacker News aka Ycombinator’s popular top 100. I never fail to find useful, surprising & stimulating reading there.

I recently stumbled on Alex Hudson’s software architecture is failing.

It’s very good, I recommend reading it.

Join 38,000 others and follow Sean Hull on twitter @hullsean.

But why did it grab my attention, you might ask? Perhaps I’m a naysayer. But I do find there is a lot of hype, and a lot of sex in software today. It’s as though the shiniest, newest, coolest toys are the ones getting the spotlight.

So when I find an alternative view, I sit up and take notice.

1. Are we making systems too complex?

Right out of the gates, Alex makes a great point:

“We’re not delivering quickly enough!”. “Our systems are too complex to maintain!”. “The application we delivered last year is completely legacy now but it’s too difficult to replace!”.

Our industry’s obsession with the newest & coolest toys, means we’re building things that don’t last very long. A real & ongoing problem.

Related: Why does Reddit CTO Martin Weiner advocate boring tech?

2. Smaller enterprises

One thing Alex pointed out that really struck a nerve was this:

For those in tech who are not working at Facebook/Google/Amazon, we’re simply not talking enough about what systems at smaller enterprises look like.

I couldn’t agree more. As a profession, we watch closely at what the big guys are doing. And that’s useful to a point. But for many smaller companies, to use such architectures would be over engineering in the extreme. Not to mention extremely costly!

Related: How I use terraform & composer to automate wordpress on AWS

3. Not bleeding & far from the edge

Another choice quote from Alex’s piece:


“It’s totally legacy, and no-one maintains it – it just sits there working, except for the occasions it doesn’t. The problem is replacing it is so hard, it’s got great performance, and the business doesn’t want to spend time replacing something working”. This is the problem being ahead of the curve – the definition of “success” (it works great, it’s reliable, it’s performant, we don’t need to think about it) looks a hell of a lot like the definition of “legacy”.

We know the term bleeding edge because it’s tough being out there trail blazing. Here I agree that sometimes legacy is also boring, yet eminently reliable.

Related: 30 questions to ask a serverless fanboy

4. Reduce, reuse, recycle

Should we build it or should we buy it? Here’s what Alex says:


I think we’re often getting the build/buy decision wrong. Software development should be the tool of last resort: “we’re building this because it doesn’t exist in the form we need it”.

Well said. Sure we should consider integration costs & testing. And using a service brings other things to balance. But it means we don’t have to own that code.

Better to focus on our business core competency.

Related: Is Amazon about to disrupt your data warehouse?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

How do I migrate my skills to the cloud?

via GIPHY


Hi, I’m currently an IT professional and I’m training for AWS Solutions Architect – Associate exam. My question is how to gain some valuable hands-on experience without quitting my well-paying consulting gig I currently have which is not cloud based. I was thinking, perhaps I could do some cloud work part time after I get certified.

Join 38,000 others and follow Sean Hull on twitter @hullsean.


I work in the public sector and the IT contract prohibits the agency from engaging any cloud solutions until the current contract expires in 2019. But I can’t just sit there without using these new skills – I’ll lose it. And if I jump ship I’ll loose $$$ because I don’t have the cloud experience.


Hi George,

Here’s what I’d suggest:

1. Setup your AWS account

A. open aws account, secure with 2FA & create IAM roles

First things first, if you don’t already have one, go signup. Takes 5 minutes & a credit card.

From there be sure to enable two factor authentication. Then stop using your root account! Create a new IAM user with permissions to command line & API. Then use that to authenticate. You’ll be using the awscli python package.

Also: Is Amazon too big to fail?

2. Automatic deployments

B. plugin a github project
C. setup CI & deployment
D. get comfy with Ansible

Got a pet project on github? If not it’s time to start one. 🙂

You can also alternatively use Amazon’s own CodeCommit which is a drop-in replacement for github and works fine too. Get your code in there.

Next setup codedeploy so that you can deploy that application to your EC2 instance with one command.

But you’re not done yet. Now automate the spinup of the EC2 instance itself with Ansible. If you’re comfortable with shell scripts, or other operational tools, the learning curve should be pretty easy for you.

Read: Is AWS too complex for small dev teams? The growing demand for Cloud SRE

3. Clusters

E. play around with kubernetes or docker swarm

Both of these technologies allow you to spinup & control a fleet of containers that are running on a fixed set of EC2 instances. You may also use Amazon ECS which is a similar type of offering.

Related: How to deploy on EC2 with Vagrant

4. Version your infrastructure

F. use terraform or cloudformation to manage your aws objects
G. put your terraform code into version control
H. test rollback & roll foward infrastructure changes

Amazon provides CloudFormation as it’s foundational templating system. You can use JSON or YAML. Basically you can describe every object in your account, from IAM users, to VPCs, RDS instances to EC2, lambda code & on & on all inside of a template file written in JSON.

Terraform is a sort of cloud-agnostic version of the same thing. It’s also more feature rich & has got a huge following. All reasons to consider it.

Once you’ve got all your objects in templates, you can checkin these files into your git or CodeCommit repository. Then updating infrastructure is like updating any other pieces of code. Now you’re self-documenting, and you can roll-forward & backward if you make a mistake!

Related: How I use terraform & composer to automate wordpress on AWS

5. Learn serverless

I. get familiar with lambda & use serverless framework

Building applications & deploying only code is the newest paradigm shift happening in cloud computing. On Amazon you have Lambda, on Google you have Cloud Functions.

Related: 30 questions to ask a serverless fanboy

6. Bonus: database skills

J. Learn RDS – MySQL, Postgres, Aurora, Oracle, SQLServer etc

For a bonus page on your resume, dig into Amazon Relational Database Service or RDS. The platform supports various databases, so try out the ones you know already first. You’ll find that there are a few surprises. I wrote Is upgrading RDS like a sh*t storm that will not end?. That was after a very frustrating weekend upgrading a customers production RDS instance. 🙂

Related: Is Amazon about to disrupt your data warehouse?

7. Bonus: Data warehousing

K. Redshift, Spectrum, Glue, Quicksight etc

If you’re interested in the data side of the house, there is a *LOT* happening at AWS. From their spectrum technology which allows you to keep most of your data in S3 and still query it, to Glue which provides an ETL as a service offering.

You can also use a world-class columnar storage database called Redshift. This is purpose built for reporting & batch jobs. It’s not going to meet your transactional web-backend needs, but it will bring up those Tableau reports blazingly fast!

Related: Is Amazon about to disrupt your data warehouse?

8. Now go find that cloud deployment job!


With the above under your belt there’s plenty of work for you. There is tons of demand right now for this stuff.

Did you do learn all that? You’ve now got very very in-demand skills. The recruiters will be chomping at the bit. Update those buzzwords (I mean keywords). This will help match you with folks looking for someone just like you!

Related: Why I don’t work with recruiters

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

What’s the *real* way to deploy on Google Cloud?

via GIPHY

I was talking to a customer recently and they asked about deployments. They wanted to do things the real way. Here’s a snippet…

I’m helping out a company called Blue Marble and they are getting ready to deploy a new POS system. The app has been built using a Node.js back-end and Google Cloud Datastore for storage. The current dev build is hosted on AWS and connects to Google for the data bits.

Join 38,000 others and follow Sean Hull on twitter @hullsean.


For prod launch, they are interested in migrating to the “real” way of deployment on Google for everything.

They are pressed on time and looking for someone who can jump in quickly. Are you available? Do you have Google Cloud expertise?

Here’s what I said.

Cultural hurdles


Yep, I’ve have used Bigquery & GCE.

What are they looking for specifically? Full deployment automation? Multiple deploys per day?

I’ve found that sometimes the biggest hurdle to fully automated deploys can be cultural issues.

In other words yes you can automate your deployment so it is push button, get all the artifacts & moving parts automated. Then deploy without much intervention. But to go from that to the team having *faith* in the system, that is a challenge.

Also: Why would I help a customer that’s not paying?

Unit testing


Once the process has been streamlined, a lot often still needs to happen around unit & smoke tests.

If the team isn’t already in the habit of building tests for each bit of code, this may take some time. Also building tests can be an art in itself. What are the edge cases? What values are out of bounds?

Consider for example odd vulnerabilities that show up when hackers type SQL code into fields that devs were expecting. Sanity checking anyone?

Read: Is AWS too complex for small dev teams? The growing demand for Cloud SRE

Integraton testing

What makes this all even more complicated is integration testing. Today many application use various third party APIs, service-based authentication, and even web-based databases like Firebase. So these things can complicate testing.

Related: How to build an operational datastore on Amazon Redshift with S3

Getting there

Although your project, startup or business may be pressed for time, that may not change the realities of development. Your team has to become culturally ready to be completely agile. Many teams choose a middle ground of automating much of the deployment process, but still having a person in the loop just in case.

Same with testing. Sure automating can make you more agile & more efficient. But you’ll never automate out creative thinking, problem solving & ownership of the product.

Related: Why did Flatiron School fail?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Is maintenance as sexy as innovation?

via GIPHY

A recent NYT piece on our aging american infrastructure got me thinking. It seems that roads, bridges, airports & city sewer systems are all in need of repair. Sadly as budgets to maintain these systems in good repair are often short, they become larger problems to fix as their status becomes critical.

Join 37,000 others and follow Sean Hull on twitter @hullsean.

“Americans have an impoverished and immature conception of technology, one that fetishizes innovation as a kind of art and demeans upkeep as a mere drudgery.”

I’m not sure this is an American-only phenomenon. However I do see it a lot with technology companies & startups.

1. Do we have to manage ops in the cloud?

The cloud has enabled infrastructure automation in some pretty phenomenal ways. Code pipelines can deliver changes to a repo, through automated unit testing, and out to customers all without human intervention. This makes teams more agile, and ultimately businesses faster & more profitable.

We might be distracted enough to stop worrying about operations altogether. After all Amazon knows how to manage broken servers & alert us right? I write do we have to manage operations in the cloud previously, as this sentiment seems to be growing.

Modern applications have a ton of interdependencies. Even with decent integration testing, the full stack is complex, and requires monitoring. Co-tenancy can complicate your performance tuning efforts as neighboring customers may directly affect your application. Third party services may be delivered from smaller or less experienced companies, whose SLA may be limiting besides. And hey if Amazon goes down, I can just tell my customers it was their fault, right?

Also: Is Amazon too big to fail?

2. Do you know Dustin Moskovitz?

Chances are I’m guessing you’ll say no. He was part of the original Facebook team alongside Zuckerberg. You don’t know his name? He had the sexy job of, you guessed it maintenance! He was the operations guy. Did he write the application code? More than likely he knew that code very well as he had to fix & maintain it. Along with the infrastructure to scale & support Facebook’s massive growth.

Read: Is AWS too complex for small dev teams? The growing demand for Cloud SRE

3. Is a little technical debt ok?

Ward Cunningham has an excellent interview about technical debt. Is a little bit ok? Maybe. But each amount is kicking the can down the road. As the NYT article on maintenance makes clear, you can move the responsibility on to the next administration, the next term, or someone else, but eventually you’ll have a critical problem on your hands, which will be much more expensive to fix.

Related: How to build an operational datastore on Amazon Redshift with S3

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters