Lost and forgotten nuggets of ideas and advice

via GIPHY

I’ve been blogging for so long, sometimes, I forget about all the old material I’ve written.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

And I was just recently digging through some of the old titles, and thought it would be fun to repost some good ones.

1. What is it, and why is it important?

Infrastructure provisioning, what is it and why is it important?

Root cause analysis – what is it and why is it important?

Zero downtime – what is it and why is it important?

Stress testing – what is it and why is it important?

Data spot checks – what is it and why is it important?

Service monitoring – what is it and why is it important?

Decoupling – what is it and why is it important?

Read: What did Matt Ranney discover scaling Uber to 1000 microservices?

2. Thought provoking

Is AWS too complex for small dev teams?

The myth of five nines – Why high availability is overrated

Why are generalists better at scaling the web?

How to hire a developer that doesn’t suck?

What 5 things are toxic to scalability?

Is there a 4 letter word dividing dev and ops?

Related: Can humility help you in your career?

3. Consulting

Can progress reports help engagements succeed?

How do you handle the onboarding of a new engagement?

Why I ask clients for a deposit

How to avoid legal problems in consulting?

How best to do discovery in cloud devops engagements

When you’re hired to solve a people problem

When you have to take the fall

When clients don’t pay

Read: What happened when I offered advice outside my pay grade?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

When can I count time as project time?

via GIPHY

I was on the train looking at a Monday.com ad. I realized that the project management space is not dead and buried, but continuing to evolve everyday. While many of us spent years using tools like Jira, along comes an upstate to make project tracking simpler.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

The trouble is, I never felt it was the project management software that made tracking difficult.

For me there was always endless nuance, around tasks and tracking.

1. Time spent commuting or thinking off the chair

For me, when I get deep into the weeds of a customer and their technology stack, I’m thinking about problems as soon as I wake up. How can I tag those resources so they are region independent? How can I create the right S3 bucket policy so the application can write, but the world can’t?

What’s more if you’re like a lot of people, there’s some slack going on after hours, and emails too. Some of this may get tracked but inevitably there are hours not tracked.

An amount of estimating will probably happen. And creating a team policy on how to handle this ambiguity, is probably a good plan. Each project and company will be different, in terms of where to draw the line.

Read: What did Matt Ranney discover scaling Uber to 1000 microservices?

2. Crossover between projects or even customers

Sometimes there is a task which requires a bit of research, for example what’s the exact syntax in Terraform to create an IAM role, and attach it to an instance? You may spend an hour digging in, and then experimenting, to make sure you have the code right.

Then you may use that same snippet on another stack that you’re building, for department B and the same company, agile team C, or another customer entirely.

When you think about it, you carry a decade of learning into each new customer you work at. And they get all that learning, which translates into efficiency. And that’s effectively free.

So there is all sorts of ambiguity. And in each case you have to make a judgement call, when you’re tracking time.

Related: Can humility help you in your career?

3. Tasks grow and change

As you continue to work on a project, some tasks have a tendency to themselves unwind. As you dig deeper, you find vast caves yet to be explored. More excavation that needs to be done.

From there subtasks may hatch from the parent, and on it goes. But this will surely blowup initial estimates, and makes tracking progress often more of an art than a science.

Read: What happened when I offered advice outside my pay grade?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

I’m interviewed in a new book Devops Paradox

I had the good fortune to be interviewed earlier in the year by Viktor Farcic. I’m excited to see the book finally released. You can pickup your own copy of Devops Paradox here.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

Our interview is pretty wide ranging, touching on aspects of database administration, cloud operations, automation and devops.

1. Defining Devops

Viktor Farcic: Moving on to a more general subject, how would you define DevOps? I’ve gotten a different answer from every single person I’ve asked.

Sean Hull: I have a lot of opinions about it actually. I wrote an article on my blog a few years ago called The Four-Letter Word Dividing Dev and Ops, with the implication being that the four-letter word might be a swear word, akin to the development team swearing at the operations team, and the operations team swearing at the development team. But the four-letter word I was referring to was “risk.”
To summarize my article, in my view the development and the operations teams of old were separate silos in business, and they had very different mandates. Developers are tasked with writing code to build a product and to answer the needs of the customers, while directly building change into and facilitating a more sophisticated product. So, their thinking from day to day is about change and answering the requirements of the products team.
On the other hand, the operations team’s mandate is stability. It’s, “I don’t want these systems going down at 2:00 a.m.” So, over the long term, the operations teams are thinking about being as conservative as possible and having fewer moving parts, less code, and less new technologies. The simpler your stack is, the more reliable it is and the more robust and less likely it is to fail. I think the traditional reason why developers and operations teams were separated into silos was because of those two very different mandates.
They’re two different ways of prioritizing your work and your priorities when you think about the business and the technology. However, the downside was that those teams didn’t really communicate very well, and they were often at each other’s throats, pushing each other in opposite directions. But to answer your question, “what is DevOps?” I think of DevOps as a cultural movement that has made efforts to allow those teams to communicate better, and that’s a really good thing.

Read: What happened when I offered advice outside my pay grade?

2. How to adapt to change

Viktor Farcic: I have the impression that the speed with which new things are coming is only increasing. How do you keep up with it, and how do companies you work with keep up with all that?

Sean Hull: I don’t think they do keep up. I’ve gone to a lot of companies where they’ve never used serverless. None of their engineers know serverless at all. Lambda, web tasks, and Google Cloud functions have been out for a while, but I think there are very few companies that are able to really take advantage of them. I wrote another article blog post called Is Amazon Web Services Too Complex for Small Dev Teams? where I sort of implied that it is.
I do find a lot of companies want the advantage of on-demand computing, but they really don’t have the in-house expertise yet to really take advantage of all the things that Amazon can do and offer. That’s exactly why people aren’t up to speed on the technology, as it’s just changing so quickly. I’m not sure what the answer is. For me personally, there’s definitely a lot of stuff that I don’t know. I know I’m stronger in Python than I am with Node.js. Some companies have Node.js, and you can write Lambda functions in Java, Node.js, Python, and Go. So, I think Amazon’s investment in new technology allows the platform to evolve faster than a lot of companies are able to really take advantage of it.

Read: What did Matt Ranney discover scaling Uber to 1000 microservices?

3. What does the future hold for Devops?

Viktor Farcic: I’m going to ask you a question now that I hate being asked, so you’re allowed not to answer. Where do you see the future, let’s say a year from now?

Sean Hull:
I see more fragmentation happening across the technology landscape, and I think that that is ultimately making things more fragile because, for example, with microservices, companies don’t think twice about having Ruby, Python, Node.js, and Java. They have 10 different stacks, so when you hire new people, either you have to ask them to learn all those stacks or you have to hire people with each of those individual areas of expertise. The same is true with all these different clouds with their own sets of features: there’s a fragmentation happening.
Let’s look at the iPhone as an example. Think about how complex application testing is for Android versus the iPhone. I mean, you have hundreds of different smartphones that run Android, all with different screen sizes, different hardware, different amounts of memory, and the underlying stuff. Some may even have some extra chips that others don’t have, so how do you test your application across all those different platforms?
When you have fragmentation like that, it means the applications end up not working as well. I think the same thing is happening across the technology spectrum today that happened 10 to 15 years ago, where for your database backend there was Oracle, SQL Server, MySQL, and Postgres. Maybe somebody who’s a DB2 enterprise customer uses DB2, but now there are hundreds of open source databases, graph databases, and DynamoDB versus Cassandra, and so on and so on. There’s no real deep expertise in any of those databases.
What ends up happening is you have cases like what happened with customers who were using MongoDB. They found out the hard way about all of the weird behaviors and performance problems it had, because there just weren’t people around with deep knowledge of what was happening behind the scenes, whereas in Oracle’s space, for example, there are career DBAs that are performance experts that specialize in Oracle internals, so you can hire somebody to solve particular problems in that space.
There aren’t, as far as I know, a lot of people with MongoDB internals expertise. You’d have to call MongoDB themselves; maybe they have a few engineers that they can send out, so what’s the future? I see a lot of fragmentation and complexity, and that makes the internet and internet applications more fragile, more brittle, and more prone to failure.

Related: Can humility help you in your career?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

How do we make domain decisions when coding software?

via GIPHY

I stumbled on an interesting discussion over at hacker news about when you have to make a decision in code outside of your domain of expertise.

This is a super interesting topic to me. As the world is more and more powered by computers and algorithms, this seemingly obscure philisophical corner of computing begins to affect us all.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

While many programmers are familiar with the experience of stumbling upon an edge case which wasn’t covered in the spec. The truth is you can’t go back and block on every single tiny detail.

Some of the bigger ones can trigger further discussion, but were it not for Falsehoods programmers believe about X the software would never go out the door.

Here’s some more food for thought…

1. Falsehoods programmers believe about names

o They won’t be a *bad* word. Whose bad? i’m not sure.
o All people have names.
o We only have to work with names from English.
o Names are ordered in a sensible manner.
o A name won’t have to change later.

You get the picture. As you start working with the minutae around names, you realize there is a lot of assumptions. Even when you weed out the big ones, there are still small assumptions.

Ultimately, you can’t get to the bottom. There will always be some assumptions, and as software evolves, changes will be required to match the real world. Living systems grow. Software systems must receive *updates*.

Read: What happened when I offered advice outside my pay grade?

2. Falsehoods programmers believe about selling products

o each product has a price
o each product has one price
o each product is one unit
o two decimal places is enough for all currencies
o all currencies have a unicode symbol

There are many more. The point is once you start digging, everything gets fuzzy. Products can be in stock and not in the warehouse. Single product can have multiple prices. It goes on and on!

Read: What did Matt Ranney discover scaling Uber to 1000 microservices?

3. Falsehoods programmers believe about addresses

o two districts will not have the same number
o postal codes will agree across jurisdictions
o a fixed number of characters will work for every address
o roads have names
o buildings are not numbered zero

Addresses are *really* messy. I remember being surprised when I moved to brooklyn. Streeteasy & Google put me in different neighborhoods. I asked my lawyer why this was. He said DOB and the post office use different databases. And google further samples from one or the other.

Never the twain shall meet!

Related: Can humility help you in your career?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

What do senior engineers do differently?

via GIPHY

I recently stumbled up on this piece at Pivotal, The art of interrupting software engineers. It caught my attention because i like to read from a different vantage from my own.

What is it like to interrupt us?

Join 35,000 others and follow Sean Hull on twitter @hullsean.

What I really got from the article though, was how different types of engineers will think about problems differently.

1. Under the hood

For junior engineers who are still a bit green, and new to working in industry, they’re downward facing. Focusing solely under the hood, they may not see how their work contributes to a product, or how it fits into the overall picture for the business.

“When asked about their progress on a story, they would make an effort to ensure I understood what was happening under the hood and what tradeoffs they were facing using a vocabulary I was familiar with. ”

For her it was technical competence that stood out – or at least not the only thing – but rather how they were strategizing, and communicating their problems, challenges, and progress.

Read: What happened when I offered advice outside my pay grade?

2. Communicate discoveries

A more intermediate engineer, would sometimes anticipate & communicate better.

“In some cases, those engineers would come to me before I even had a chance to enter the paranoid zone and give me a simple explanation of how the team had learned new information since they first estimated the complexity of a story. ”

She is also speaking of situational awareness. So not working in a vacuum, communicating & incorporating that new information as it becomes available.

Read: What did Matt Ranney discover scaling Uber to 1000 microservices?

3. Anticipate in advance

Senior engineers, she says would really stay ahead of the curve. They were even anticipating what might be a roadblock for her product delivery.


“Some of the very best practitioners would ask me in advance how urgently we should deliver a particular user story and what we were ready to give up in order to ship faster.”

Weighing tradeoffs, and prioritizing is a huge factor in velocity. If you can tame that beast, you’ll go very far indeed!

Related: Can humility help you in your career?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Viktor Farcic Interview excerpts

via GIPHY

I recently did an interview with Viktor Farcic all about operations, DBA & Devops. Here are some excerpts: What does Dev-Ops mean?

Join 35,000 others and follow Sean Hull on twitter @hullsean.

Continuing where I left off, I’ve included a few more highlights below. Enjoy!

1. Can I use a tool to migrate to the public cloud?

Viktor Farcic: I’ve seen quite a few of these tools that tell you if you buy our tool, we’re going to transfer whatever you have to the cloud. For example, Docker announced in the last DockerCon that they’re going to put in containers without a single change and everything will work. What do you think about that?

Sean Hull Salespeople often simplify things quite a bit in order to sell a product; in my experience, the devil is in the detail. It’s not to say that an automation tool like that might not be valuable and useful. It might be a good first step to getting your application in the cloud, and it might be an easier way than to rebuild everything one by one. But I doubt that it’s going to work magically just by one script.
EC2 instances, for example, have different performance characteristics, not only in terms of the disk I/O, memory, and CPU, but in smaller instances, they actually throttle the network access so you might spin up an instance and it just might not behave well. It might take time. In fact, all sorts of things could happen. You might have written MySQL scripts that assume you have root access to the server and then you rebuild that in an RDS and you get errors because you don’t have access to those resources on RDS. There’s a lot of things to consider.

Read: What happened when I offered advice outside my pay grade?

2. How do you adapt to change?

Viktor Farcic: I have the impression that the speed with which new things are coming is only increasing. How do you keep up with it, and how do companies you work with keep up with all that?

Sean Hull: I don’t think they do keep up. I’ve gone to a lot of companies where they’ve never used serverless. None of their engineers know serverless at all. Lambda, web tasks, and Google Cloud functions have been out for a while, but I think there are very few companies that are able to really take advantage of them. I wrote another article blog post called Is Amazon Web Services Too Complex for Small Dev Teams? where I sort of implied that it is.
I do find a lot of companies want the advantage of on-demand computing, but they really don’t have the in-house expertise yet to really take advantage of all the things that Amazon can do and offer. That’s exactly why people aren’t up to speed on the technology, as it’s just changing so quickly. I’m not sure what the answer is. For me personally, there’s definitely a lot of stuff that I don’t know. I know I’m stronger in Python than I am with Node.js. Some companies have Node.js, and you can write Lambda functions in Java, Node.js, Python, and Go. So, I think Amazon’s investment in new technology allows the platform to evolve faster than a lot of companies are able to really take advantage of it.
Read: What did Matt Ranney discover scaling Uber to 1000 microservices?

3. What is the future of Devops?

Viktor Farcic: I’m going to ask you a question now that I hate being asked, so you’re allowed not to answer. Where do you see the future, let’s say a year from now?

Sean Hull:
I see more fragmentation happening across the technology landscape, and I think that that is ultimately making things more fragile because, for example, with microservices, companies don’t think twice about having Ruby, Python, Node.js, and Java. They have 10 different stacks, so when you hire new people, either you have to ask them to learn all those stacks or you have to hire people with each of those individual areas of expertise. The same is true with all these different clouds with their own sets of features: there’s a fragmentation happening.
Let’s look at the iPhone as an example. Think about how complex application testing is for Android versus the iPhone. I mean, you have hundreds of different smartphones that run Android, all with different screen sizes, different hardware, different amounts of memory, and the underlying stuff. Some may even have some extra chips that others don’t have, so how do you test your application across all those different platforms?
When you have fragmentation like that, it means the applications end up not working as well. I think the same thing is happening across the technology spectrum today that happened 10 to 15 years ago, where for your database backend there was Oracle, SQL Server, MySQL, and Postgres. Maybe somebody who’s a DB2 enterprise customer uses DB2, but now there are hundreds of open source databases, graph databases, and DynamoDB versus Cassandra, and so on and so on. There’s no real deep expertise in any of those databases.
What ends up happening is you have cases like what happened with customers who were using MongoDB. They found out the hard way about all of the weird behaviors and performance problems it had, because there just weren’t people around with deep knowledge of what was happening behind the scenes, whereas in Oracle’s space, for example, there are career DBAs that are performance experts that specialize in Oracle internals, so you can hire somebody to solve particular problems in that space.
There aren’t, as far as I know, a lot of people with MongoDB internals expertise. You’d have to call MongoDB themselves; maybe they have a few engineers that they can send out, so what’s the future? I see a lot of fragmentation and complexity, and that makes the internet and internet applications more fragile, more brittle, and more prone to failure.

Related: Can humility help you in your career?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Consulting outside my pay grade

via GIPHY

I recently worked with a customer migrating to Azure. Part of the plan was to take advantage of various free credits available on Azure. That’s not a bad reason cost wise, as long as the engineering cost of migration is considered.

Then I dug further…

Join 35,000 others and follow Sean Hull on twitter @hullsean.

Sean: Why are you migrating to Azure?
Customer: We’re getting free credit from Microsoft. Also we must not have any services on AWS.
Sean: Why is that?
Customer: We are trying to close to a deal with Walmart. Of course they’re a big fish for us and we don’t want to lose them.
Sean: Ummm… let’s take a closer look at that.

o we need to migrate because our future customer is in a pissing match with AWS
o sales should not blindly drive tech decisions
o there will always be some things on AWS
o if you’re going to be non-technically pure then just say yes we have *everything* off of AWS and keep running your business
o if sales can lie, then go ahead and blow the smoke up there

You can’t be completely free of AWS

While the sales team thought that was a simple directive, they forgot a few details.

1. There is a financial cost to the business to migrate from AWS to Azure
2. There is a technical risk of migrating from AWS to Azure
3. There are hidden and unforeseen technical costs as things fail, or code unravels.
4. As you have THIRD PARTY apps, you are using AWS, such as Slack, Salesforce, Zoom, Dropbox, Jira and perhaps many others.
5. Internet traffic flows through aws networks.

Read: The myth of five nines – why high availability is overated

Sales & engineering have different mandates

Sales teams are driven in a different way than engineers. Their day-to-day is about closing deals, and so priorities are different. What’s more the grasp of technical minutiae may not always be as deep.

This is not a criticism, more an observation. So too engineers don’t understand the realities of sales, and closing deals.

That said you get into the danger zone when sales driven calculation pushes the business to make a bad decision.

The engineering marvel is what holds up the bridge. So it doesn’t make sense to skip the inspections, or leave out trusses because a customer said the bridge will look prettier that way. The engineers must have serious input on technology decisions.

That will then help the business make a balanced decision, one that considers the true costs involved.

Read: Relational Database – What is it and why is it important?

Engineering & Sales should be cooperative

One should not outweigh or outmuscle the other. Their needs to be a balance of concerns. A big chunk of business that doubles the underlying revenue may be worth large tech stack changes.

But there is also a risk that the house of cards comes tumbling down. Even if the reengineering risks are managed, the new debts owed by Rome could still take you down.

Related: Why generalists are better at scaling the web

How does pay grade come in?

In the above case I was hired to do a migration. I wasn’t hired to think strategically about the business.

So some might argue that is stepping outside the mandate or scope of work. That said I do try to offer opinions where they make sense. In this case it may be of value to the business.

However, since it is “outside my pay grade” I would offer it humbly, and gently. No sense shooting yourself in the foot. 🙂

Related: Why generalists are better at scaling the web

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

What does DEV OPS mean?

via GIPHY

I was recently interviewed by Victor Farcic. He asked me a lot of interesting questions.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

One really struck me that I thought was important, about the whole devops movement.

How do you define devops

Viktor Farcic: Moving on to a more general subject, how would you define DevOps? I’ve gotten a different answer from every single person I’ve asked.

Sean Hull: I have a lot of opinions about it actually. I wrote an article on my blog a few years ago called The Four-Letter Word Dividing Dev and Ops, with the implication being that the four-letter word might be a swear word, akin to the development team swearing at the operations team, and the operations team swearing at the development team. But the four-letter word I was referring to was “risk.”

Related: Why generalists are better at scaling the web

To summarize my article, in my view the development and the operations teams of old were separate silos in business, and they had very different mandates. Developers are tasked with writing code to build a product and to answer the needs of the customers, while directly building change into and facilitating a more sophisticated product. So, their thinking from day to day is about change and answering the requirements of the products team.
On the other hand, the operations team’s mandate is stability. It’s, “I don’t want these systems going down at 2:00 a.m.” So, over the long term, the operations teams are thinking about being as conservative as possible and having fewer moving parts, less code, and less new technologies. The simpler your stack is, the more reliable it is and the more robust and less likely it is to fail. I think the traditional reason why developers and operations teams were separated into silos was because of those two very different mandates.

Also: Walking the delicate balance of transparency

They’re two different ways of prioritizing your work and your priorities when you think about the business and the technology. However, the downside was that those teams didn’t really communicate very well, and they were often at each other’s throats, pushing each other in opposite directions. But to answer your question, “what is DevOps?” I think of DevOps as a cultural movement that has made efforts to allow those teams to communicate better, and that’s a really good thing.

Read: One time in 2013 I had to take the fall

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Can humility help engagements succeed?

via GIPHY

I was reading this article on Vox recently titled Intellectual Humility: the importance of knowing you might be wrong.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

It caught my attention, and I think we can expand on it a bit. Here are my thoughts.

1. Admitting when you’re wrong

Of course we’ve all had moments when we’re wrong. We make a proclamation, which turns out wrong. We measure something incorrectly. Or we forecast imprecisely.

It is hard to stand on the stage. The spotlight is on you. And when you do that you can be the object of criticism, and speculation. Just like everyone you may make mistakes, but when the spotlight is on you, it can weigh heavier.

That is exactly the time to be a bit humble, acknowledge your thought process, and where you went wrong. By standing up and admitting your mixup, you will come out the other side stronger.

Related: How can we keep cloud architectures simple

2. Admitting you might be wrong

This can be harder. As engineers we like to problem solve. We spend years exploring math & science, looking for the “truth”. The more one searches for it thought, sometimes the more illusive it can be.

Measurements are never exact. And theories and architectures often fail in the face of real world traffic. Applications fail. Servers fail. Outages happen. Customers especially paying ones will inevitably get angry, and this can backfire onto you.

Be prepared for the real world. It gets messy.

Also: What hidden things does a deposit reveal?

3. Allowing space for others to be wrong

This is a tricky one. You may know what others don’t, but it may take finesse to share that truth. You may have to sell your perspective, even while another perspective may be measurably wrong.

Be prepared to sometimes let things break a little. As hard as this is, it may allow for others to learn.

Like immunizing, sometimes failure can teach what words cannot.

Read: Can communication mixups sour an engagement?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Should we right size instances in the public cloud?

via GIPHY

I’m a big fan of Corey Quinn’s Last Week in AWS newsletter.

Recently he wrote a piece titled Right Sizing your instances is nonesense

Not to be outdone, a blogger Joe at Sunshower.io wrote a counterpoint piece Why Right Sizing instances is not nonsense.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

So what’s the verdict here? Is Corey wrong and Joe right? Or is Joe right and Corey wrong?

I would argue it depends. Corey’s piece emphasizes the big picture, essentially that technical changes can buy more trouble than the money they save. While

1. Corey emphasizes the 300 foot view

Corey’s article uses a broad brush, and in doing so some of his specifics are incorrect. For example the point about older versions of Operating Systems and hypervisor. In most cases your OS won’t know it’s running on a hypbervisor at all, so this seems a very rare edge case indeed.

That said his big picture conclusion seems spot on. Changing instance sizes can be a huge risk if you don’t do so regularly. With legacy apps, who knows how they might behave. In this you carry the same burden as changing instances at all. Your AMI may not be ready, or you may have had some manual steps to build the box again.

Related: How can we keep cloud architectures simple

2. Joe gets down in the weeds of technical specifics

The first thing about Joe’s post that caught my attention was to use the console to change the instance size. You shouldn’t be manually changing instances through the console to start with. What, everything is code, all the way down? Hehe…

Also his points about cost savings seemed cherry picked. If you can save that much money by changing instance sizes, it is well worth the cost to do regression, integration & disaster recovery tests to make sure it will all work. Get on it!

Also: What hidden things does a deposit reveal?

3. Use infrastructure as code, test & retest it

IAC is really the way to go. But that still doesn’t mean you’re out of the woods. Be cautious when changing instance sizes!

With code, there is testing. So change your Terraform code and then verify it works. Now just because you have a variable indicating the instance size, doesn’t mean changing it won’t break something.

Read: Can communication mixups sour an engagement?

4. Weigh the cost savings with the risk of breaking things

Changing an instance size and redeploying could break all manner of things. It’s possible you used a variable for the instance size in some places and hard coded it in others. Or made some weird reference in an autoscaling group.

It may be the AMI you’ve built works on one type of instance but not another. Or that your AMI is deployed in one region but not another. Or that your old instance size is available in us-east-2, but your new instance size is not yet available there. Yes the console wouldn’t have offered it, but your Terraform code didn’t know.

Check out: How I use 5 daily habits to stay on track

5. Put up some guiderails

As you’ll see further down in the comments, Joe suggests to put limits around instance size changes. That makes sense. After you’ve done testing, you’ll have an idea of what these limits should be. No instance size less than 30GB memory? No instance size greater than X. None in region Y. Etc.

It’ll require tweaking your terraform infrastructure code, so it’s not a free change. But it’ll pay dividends if your costs savings are in the thousands.

Also: Can daily notes help you work better with clients?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters