What can we take away from Amazon’s US-EAST outage


AWS had a few outages in December. It’s interesting to note that when you’re as big as AWS you get a lot of media attention no matter what happens.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

It’s worthwhile to consider how many eggs are in one basket. Is multi-cloud an option? If I stick with a single cloud, am I redundant enough to weather it when USE1-AZ4 goes down? Or 3 or 2? Have we tested that scenario with all hands on deck? Have we tested weird edge cases, and isolated our dependencies?

1. Regions & Availability Zones matter

Amazon’s cloud offering is a globe spanning service. They call their data centers availability zones (AZs), and there are multiple of them in each region. In fact Amazon’s documentation lists 26 worldwide regions and a total of 84 availability zones.

Where are your data sits, has never mattered more. You want it to be close to your customers, so response is fast. CloudFront can provide redundancy for assets, but your application still has to sit somewhere. EC2 servers, S3, Route53, EBS, RDS, File Gateway, each of these myriad services has different region and AZ patterns.

To make it all just a little more interesting, new services are typically deployed in a few popular regions first, so every region has a different mix of the total soup of AWS offerings.

Read: How do we search for senior engineers?

2. Even if you’re not *using* a region, you might be vulnerable

US-East-1 was the first region in the AWS service offering. So it has the oldest equipment and wiring. It’s also the default, so more customers are deployed there, which means more competition, slower API response etc.

Also AWS has strange dependencies still built into it. Some resources are stored in regions, not globally. So even if you don’t have EC2 instances in US-East-1 you may have other resources you need access to.

Read: Should every engineer try a spell in consulting?

3. Have multiple types of backups and recovery paths

Even if you’ve double and triple checked your IAS rebuild scripts. Even if you believe you’ve got DR dialed in and locked down. Test once, test twice, and triple check.

What’s more have multiple different ways to rebuild. For example on my iheavy.com site, I have an article-by-article dump of the database that WordPress can create. That’s XML, and works across different versions of WP so even if I lost everything, I still have the text! I have copies of that sent to private s3 bucket, and to my local laptop.

Next I have database backups, sent to a private S3 bucket, and also copies to my laptop.

Then I have Infrastructure as code scripts in Ansible, and Terraform, that will rebuild the components of my site quickly and easily. Those I test regularly.

Sure you’re business is much more complicated than my little blog, but the concepts remain the same!

Read: Is it time to diversify your cloud?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

How do we search for senior engineers?


I recently stumbled upon Mike McQuaid’s blog and I was intrigued. He wrote an article which caught my attention immediately: Stop requiring specific technology experience for senior-plus engineers.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

Mike asks some really probing questions. For example how do we know most of the “getting up to speed” time won’t be spent on company culture, or learning your deployment system? Does a senior engineer spend more time on coding or on mentoring, interfacing with non-technical teams and other collaborations?

The article got my juices flowing. Here are some of my thoughts…

1. Why is sourcing people so hard?

For one thing people rarely match their resume. Some people are perform like rock stars but don’t interview like one. And some people interview *very* well, but don’t perform so great. Why is this?

Work ethic, salesmanship, personality and so much else comes into play.

What’s more if you’re building a new startup, you probably love it very much. So you may feel everyone you interview will also be scrambling to work there. But senior engineers likely have a *lot* of options.

Read: Should every engineer try a spell in consulting?

2. Can a conversational approach help?

Conversation is a great way to get past some of these limitations. Though admittedly it’s probably the resume and keyword search that brought them to your desk.

Asking the candidate to tell stories about problems they’ve encountered, and how they solved them is a great start. It also allows one to dig in. Find where the person becomes impassioned. These are likely topics they care a lot about. Which areas are those and do they match the ones your company needs?

Remember passion and hunger can’t be taught. If you can speak to managers and colleagues of this candidate all the better, to get a 360 view of who they are and how they work together on a team.

Read: Is it time to diversify your cloud?

3. What is the big picture really saying about this person?

I think passion speaks volumes. For years I specialized in scalability. Why? Because I’d be at all of these startup companies trying to go fast. And they weren’t able to. I got excited about finding the ways, because this meant I could make a difference for them.

When it comes to technologies and stacks engineers can get very territorial at times. It’s natural I guess. Let’s not forget that a wide field of experience gives you a birds eye perspective.

And time, teaches what books cannot.

Related: Can I secretly deploy a key to production?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Three big things I learned from 30 years of consulting


Ok truth be told, it’s only been 25 years, but that didn’t flow as nicely in a title. What can I say?

Join 35,000 others and follow Sean Hull on twitter @hullsean.

But seriously, thinking through those decades, I felt like piecing together the big themes. While many folks work at one firm for 2+ years, two decades would bring experience at ten firms.

In that time I’ve worked at 5-6 per year, bringing my total to over 100 firms. This has given me some perspective that’s unique and some useful patterns emerge.

1. Should have, could have, would have

In quite a number of companies, I’ve been brought in as the second wave. What do I mean by this? Well the first founders, builders and creators have left, and their technology still stands. So it is not our job to sift through the documentation and maintain those systems or expand.

What challenges happen with this?

A. Managing and reducing complexity

When you are handed an infrastructure with myriad servers, databases, programming languages, frameworks, libraries, security keys, and assets, you’re thinking, how can I simplify all of this. That is your number one concern.

So for the folks enabling microservices these days, please please please consider the future. Does that mean microservices is a non-starter? I don’t know. What I do know is the cambrian explosion of each dev gets whatever they want, is dangerous. By not standardizing on languages, frameworks, databases, or anything, you allow the inevitable creep of complexity.

B. Looking for documentation

When it’s quitting time, you won’t be able to document things. Push for this early and often. You won’t always get it, but try try try. Code commits can enforce comments. Inline, infrastructure, and automation go a long way too.

C. Managing performance and support

As complex systems age, performance often suffers. Packages and libraries need to be patched, or are no longer maintained. That’s when the big boring technologies really become your friends, because they’re still around, and still doing their jobs!

All this points back to reducing complexity and even god forbid using boring technology.

Read: Should every engineer try a spell in consulting?

2. Look for low hanging fruit

Low hanging fruit are like the elephant in the room. They’re big and can have a big impact. But sometimes nobody seems to notice them. Your job is to think critically and notice them.

For example the age-old battle with ORMs. They’re still finding their way into web applications mostly because nobody wants to learn SQL. Hey they solve the problem right now, the performance problems won’t occur until I’m long gone. No! I love what Justin Etheredge wrote – relational databases aren’t dinosaurs, they’re sharks. Wow, that packs so much truth into one sound bite!

If you can remove technologies, do it. If you can do the job with 5 servers instead of 10 smaller ones, on 5 different stacks, do it. Keep it simpler!

Read: Is it time to diversify your cloud?

3. Good data is gold

Whether it’s state agencies, fighting with the federal government about who is doing what correctly in the pandemic, or business teams making fair forcasts of the market for your minimum viable product.

All of these require clean data. Data that has been scrubbed of inaccuracies, messy state fields, missing names, round off errors, and the rest. It’s a dirty job, but it has to be done.

Being serious about how *good* your data is, where it is lacking, where it is downright untrustworthy, these are all disciplines, that everyone in the organization must participate in.

A stubborn nose for finding the truth will help. A Lt. Columbo sense of curiosity. And persistent diligence. All contribute to this end goal.

Related: Can I secretly deploy a key to production?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Are you ready to try consulting?


Well it’s been a minute since I blogged, but we’re back! I had a hack, and the site was down for a time, but it feels good to be putting the proverbial pen to paper again.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

I was flipping through yCombinator News recently, as it’s where I get most of my interesting information. The filter is just excellent.

I ran into this piece by Forrest Brazeal Every engineer should do a stint in consulting.

Being one of my ongoing mantras over the years, the headline caught my interest right away.

1. Running a business helps you understand your boss

A stint in independent consultant requires one to wear a *lot* of hats. Don’t know where your next meal is coming from? You’d better hit the streets and find work. That is start networking. Meeting people out there in the world. Talking, sharing your story, your knowledge, how you solve problems, and what you can contribute to those businesses.

You also have to constantly evaluate costs. How much time do I spend learning a new skill? Will that skill land me a specific role? What other skills would help in the mix? What problems are startups trying to solve? How can I save them money, but fixing problem X?

By managing your own tradeoffs you begin to see some of the tradeoffs your boss must juggle. Hiring new people is a cost, but longer term benefit. Creating the right atmosphere, creates a workplace where people stick around. And solving interesting problems. draws the talent.

Read: Is AWS too complex?

2. Running a business helps you value your time

If you’re working in a fulltime role all the time, you always make the same paycheck. So taking long lunches, or working through lunch benefits you the same. Putting in overtime too, is hard to translate directly.

In consulting, you may often work by the hour. So both you and your employer are slicing up your time into dollar amounts. Yes I can work at 2am, it costs $x/hr. If that price is higher than your day rate, your employer will evaluate how much it’s worth to him, and so will you.

A regular habit of this, allows you to get better perspective on pay packages, bonuses, and nebulous stock options that may only have a gamblers chance to pay dirt.

Read: How to lock down cloud systems

3. Running a business teaches you to solve problems

Yes an engineer builds things, and in so doing solves many many technical problems. And consulting also asks you to solve challenging problems such as:

o How can I frame what I do in a way that speaks to my bosses day-to-day challenges?
o How can I find new business? Who could use my skills?
o How can I price and package my services?
o How do I make sure me & my customer are both happy?
o How can I communicate better with my team or CTO?
o How do I ensure I get paid in a timely fashion?
o How do I create longer term growth and bring more customers to my business?

Related: How do I migrate my skills to the cloud?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Is it time to diversify your AWS cloud?


Is it time to diversify your AWS cloud?

If you missed the news, last week AWS had a major outage in it’s east coast datacenters.

What does this news mean for your business?

Join 35,000 others and follow Sean Hull on twitter @hullsean.

If you’re like a lot of customers – and I mean a *lot* of them, you probably have resources deployed in us-east-1. And if you haven’t done it right you probably had an outage last week.

1. How does AWS organize their cloud?

Amazon Web Services, not to be confused with the bookseller side of the business, provides on-demand compute power to customers. Want to run an application on the internet? Chances are you’re looking to AWS to host those servers.

They’ve designed a brilliant global network of datacenters, which are organized into regions, and within those regions individual buildings or datacenters themselves. Inside those buildings are rows and rows and rows of interchangeable servers, that you configure without ever visiting the building.

Those regions are named based on where they are in the world. us-east-1 (north virginia) us-east-2 (ohio), us-west-1 (Oregon) and so on. There are many many regions globally. Within those you have individual datacenters or availability zones. For example in northern virginia you have us-east-1a, us-east-1b, us-east-1c and so on.

Read: Is AWS too complex?

2. What happened?

AWS had an outage of the US-EAST region. Customers deployed in that region were affected. Applications went offline, maybe data was lost.

As it turns out North Virginia is the oldest region. That’s where it all began for AWS.

Want a bit more history, reach Why Amazon’s data centers are hidden in spy country.

Read: How to lock down cloud systems

3. Which customers were affected?

There are two ways you could have been affected by this outage.

1. You deployed in the default region and figured that was a safe bet

2. You deployed *only* in one region

These are two no-nos of cloud deployments, but a lot of customers still do it. Perhaps out of ignorance, or out of pressure to get it done.

Related: How do I migrate my skills to the cloud?

4. How can I mitigate these types of outages?

The right way to deploy is to use devops automation for starters. This means everything is in code. Want to move your full stack to another AZ or region? Easy just change one variable, and redeploy. A short while later all your resources will be moved over, and you’ll be back up and running. That’s powerful.

You can also use various types of DNS tricks. Cross-zone load balancing, is very powerful, and allows you to deploy some assets in different zones, and therefore avoid outages affecting individual zones.

Related: What mistakes did you make when starting as a consultant?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

How can we secretly deploy this key to production?


I worked with a client some time back that was in a very unfortunate position indeed. Through various management shuffling, the COO was the new sheriff in town. Which is fine and normal.

Except in this case the entire operations was managed across the sea.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

Now in many or most cases that would be fine. The new COO asks for the root credentials, works with devops to get new hires on board, and move forward. But these were not normal circumstances.

No, here the turf war began in earnest. Threats were even lobbed about what damage could be done.

1. Get a handle on all the systems

Upon starting the engagement, I attempted to inventory everything. What aws logins & IAM accounts were there? What servers were deployed? What networks, and security groups were setup like?

I followed many of the suggestions I outlined here: locking down cloud systems.

However we hit some hurdles, because I did not have my key deployed on production servers. I also could not update the deployed keys. So we were stuck in a holding pattern.

Read: Is AWS too complex?

2. Can you deploy this key in secret?

As we moved through the mess, and still could not gain control of systems, I received a strange request.

Can you deploy your key to production in secret?

Well of course what I’m thinking at this point is, I sure hope not. If a new engineer could bypass the devops team and get the keys changed on the servers, that would amount to a giant security hole. It was as it turned out not possible to do.

Related: How do I migrate my skills to the cloud?

3. Git is not a good way sidestep human communication

What I realized is that we were trying to use git to deploy a key, which would have communicated our intentions to the devops team anyway. What’s more it would have communicated it in a strange way. Why is this happening, and what are *they* trying to do over there in New York?

Better to communicate the normal way, hey we hired this new engineer and we would like to give him access to these production servers. End of story.

Related: What mistakes did you make when starting as a consultant?

4. How things got resolved

We attempted to get the key installed on systems, which did precipitate a conversation about, who is this guy Sean and what’s his role? All of that stalled for a while.

In the end a new CTO was hired, and the devops lead in Odessa reported to this new guy. This was a savvy move since the new CTO didn’t yet have a relationship good or bad with the loose cannon in Ukraine.

As we stepped forward carefully, we got a hold of credentials, and then planned an early morning cutover. While multiple engineers were poised at our keyboards, the devops lead in Ukraine was being fired. So we were locking him out a the same time.

Related: How to hire a developer that doesn’t suck?

5. No technology can sidestep hard conversations

It all went smoothly, and a lot was learned.

What I learned was that when strange requests surface, usually it means some bigger conversation needs to happen.

No technology is can sidestep those hard conversations.

Related: 30 questions to ask a serverless fanboy

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

How not to handle “we’re not paying” emails…


I saw this thread recently posted on Hacker News. “We’re not paying invoices due to the pandemic” – How to handle these.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

What shocked me is what Patrick McKenzie (@patio11) suggested:

“The right response is probably ‘I have received your letter. The invoice is good. I demand payment by the date on the invoice, under the terms of our contract'”

Wow. Powerful words. Mildly threatenting.

Here’s what I would suggest…

1. I demand payment under the terms of our agreement

I really don’t want to ever receive letter demanding anything. It’s unpleasant. It feels threatening. Why would you send such a missive to your client or customer?

As a consultant or freelancer, you should understand that you are *NOT* an employee. As such you are not protected by the law in the way payroll employees are. You should understand that the first ones who get paid are payroll, next would be rent & utilities, and lastly would be contractors or vendors.

What’s a better alternative. First keep good communication. Let your client know we are all struggling, and we can work through this together.

I typically send a message with subject “a gentle reminder: invoice xyz due on date abc”. Encourage any kind of communication, and keep it open. Don’t be afraid to remind your customer again in two weeks. If you don’t hear back, ask again just to make sure the messages are getting through.

Patience wins the day.

Read: How to avoid legal problems in consulting?

2. I’ve got bills to pay, this is serious

Talking from your own point of view, is not helpful. What’s more it encourages your customer to also see things from a selfish vantage point. Instead speak from their point of view. Explain that you understand where they are coming from, and let’s work together to get a portion of the bill paid. 50%, 25%, 10%. Any amount is good because it means an action has been taken. It confirms the invoice is valid.

As far as those bills go. If you’re a freelancer, and running out of money, you’re doing it wrong. When you freelance there is overhead, for health insurance, vacation & sick leave etc. If you are not making the rates that cover these costs, it doesn’t make sense to be freelance in the first place. In fact it means you’re not billing right, packaging right and selling right.

Related: When clients don’t pay

3. Remember this is a test for everyone

The current pandemic is a challenge for everyone. Small and large businesses, freelancers, deli, laundry owners, restaurants & music venues. I can go one. Please keep this in mind before you go off and send some email to your non-paying client, that you will later regret.

Keep your perspective, take a deep breath, and be the bigger man. Take the high road, and keep your center. We will all get through this together.

Related: Why do people leave consulting?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Why is there a company whose only purpose is to help me with my AWS bill?


I know everyone is talking about the pandemic at the moment, I thought I would sidestep the topic, and continue to discuss what I know best. Which is public cloud!

Join 35,000 others and follow Sean Hull on twitter @hullsean.

Of late, I see more and more discussions on CTO forums, and in the news on surprise AWS bills. Did you hear what happened NASA’s cloud deployment fail?

So I thought I’d provide some history, and a few ideas…

1. Oracle days

Believe it or not this has happened before. In the days when I did a lot of Oracle work, I remember it was terrible how tough their licensing was. And companies were pretty much hamstrung because they needed the technology to run their business.

Given the demand, new firms like Miro Consulting stepped in to help. Their business is specifically built to help customers with out-of-control Oracle bills.

Read: How to hire a developer that doesn’t suck

2. Repeating refrain

What’s this got to do with Amazon Web Services you say?

Just today I discovered a firm that promises to help you lower your aws bill by 20-40%.

Promises aside, if there is a business doing this, there must be money to be made in it. So that speaks to a sizeable market opportunity.

Related: When you have to take the fall

3. AWS has a webinar on the topic

Hey if you’re thinking you can manage it yourself, why not look to AWS for assistance on lowering your bill.

Something tells me there’s a conflict of interest there, but I digress.

Point is with this topic becoming of growing interest, I think you’ll see more businesses stepping into this niche.

Related: How do I migrate my skills to the cloud?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

I’m building a startup. What should I watch out for?


I’ve worked with close to 200 startups over the years. So I’ve seen a lot of stuff. Some of the get-rich-quick managed to get broke even quicker. And others saw great engineers leave behind a clunky maze of an architecture with little or no documtation.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

Here are some thoughts on what to be mindful of…

1. Keep the stack simple

Are you asking yourself questions like these:

o if we build x,y,z it will save us time later
o we need kubernetes because everyone is using it
o let’s deploy on Amazon’s Cloud, everyone is there

If so you may have sidestepped keeping the stack simple. If you are building a proof of concept, mvp, chances are you do not need to build all the bells and whistles.

Relentlessly apply the discipline to keep things simple.

Read: Are pioneers and process people different breeds?

2. Use boring technology

You may be tempted to use some exotic new language. I know this new framework solves all the problems that have come before. No!

Let someone else live on the bleeding edge. I’m not suggesting to use COBOL, though there is still a surprising amount of that still around. What I’m saying is 5 year old tech is *not* legacy.

Not sure? Do a quick search on a job site for the language or technology. How many candidates do you get? If they are rarer you will pay more. And replacing your genius developer will be even harder.

But boring technology is not just about finding talent. It means many of the problems have already been solved. This is huge. There are proven ways to implement with the tried and true.

Related: Why is Martin Weiner special?

3. Beware of sharks in the water

Sharks can come in many forms. Investors that want to take 50%, a google or an Amazon that could rearchitect your solution in a week. These are real and present dangers.

Let’s not forget overspending on co-working spaces, and hidden costs around benefits, and surprising fees on your data or aws bill.

Related: How can 1% of something equal nothing?

4. Outsourcing – trust but verify

You may be able to hire one or two devs locally in new york, but if you need a larger team, you’ll likely look to outsourcing. There are tons of talented and brilliant engineers worldwide. So it makes sense to leverage this.

Beware though of handing all control to a team outside our shores. That’s because if you stumble into a legal trouble, you cannot use legal methods to resolve them. Some examples…

o what if you lost your aws keys and the outsource team holds you hostage?
o what if you are hacked, and the outsource team abandons the project?
o what if the outsource team tries to steal your IP?

I’m not suggesting here that outsource teams are any more likely to do these things than some firm locally. I think these events are rare in business. But if they happen local you can use the levers of the justice system to resolve. Remote may not lend to that.

Related: The art of resistance – when you have to be the bad guy

5. Relentlessly manage time

Your time is going to be short, and you’re trying to do the impossible. Use the tech wisely:

o use monitoring systems to alert you – no need to manually check things
o beware of slack and email – as they can destroy productivity
o automate where you can see real benefits
o keep the architecture simple!
o network for hiring people

Related: Is maintenance a forgotten art?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

What can we learn from NASA’s AWS fail?


I was just reading the Register, which is sort of the UK’s version of Slashdot, and they had a jaw dropping title. NASA moved 247 petabytes into AWS and then later learned about EGRESS costs

OMG! Face palm. Wow.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

To say this is a disaster is an understatement. Could it have been prevented? Not likely by 100% strategic thinking. I believe a certain amount of real-world testing & prototyping is the only way.

Here are my thoughts…

1. Expect hidden costs

Everytime I check there are more AWS services. Just now I did some googling, and the number stands at 170. Not only is it tough to keep up with all of those, but the offerings are constantly evolving, getting new features and so forth. That means the pricing and costs are evolving too.

All this means an infinitely complex web of interconnecting pieces, so it is near impossible to predict prices in advance. The solution? Prototype.

And this would have helped save NASA. Because you would build, feature test and load test too. In all of that would have come a small estimate of cost which would include EGRESS costs.

There are no guarantees in this game, but it is surely getting complicated.

Read: How can 1% of something equal nothing?

2. Vendor lock-in is not dead

With the receding of some of the big old world vendors like Oracle, many have forgotten the shark like tactics they used with startup companies.

The model was something like this. Send in the big guns, nicely dressed to get you on board. Finesse the sale. Offer deep discounts, and get the customer on Oracle. After a year, maybe too, start squeezing. You’d be surprised how much blood comes out of diamonds.

These days we feel more free to port our applications to different cloud vendors. Even if mostly everybody is on Amazon already. But this NASA story really highlights the great organizational cost to migrate to the cloud. You architect your application, you do cost planning, and so on. So once you’re there, it’s hard to unravel.

Related: Is Fred Wilson right about dealing in an honest, direct and transparent way?

3. New possible hacking vector

Since costs are tied to usage in the public cloud, this could have implications for hacking. If a bad actor wants to cause you harm, then can now just use your service more.

Don’t like company A? Write some bots to access them from obscure locations, and ramp up those egress costs. With all the complexity of the cloud, are most firms monitoring for this sort of thing? I don’t see it in my engagements.

Something similar happened to me. I wrote: when mailchimp fraudulently charged my credit card. It really happened. Do I think it was intentional? You’ll have to read the article to get my 360 degree take on it.

Related: What mistakes did you make when starting as a consultant?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters