When I found gold in my customer archives

via GIPHY

I’m good at keeping notes. I’ve blogged about Can progress reports & daily notes help engagements succeed. I would give that an emphatic YES!

From helping with communication, to sharing arcane details about blocking issues, struggles & hurdles, notes can illuminate things that a CTO or manager may not otherwise be aware of.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

I was digging through mine archives recently, and found a bunch of notes on how to think about Terraform. In particular, how do you think about infrastructure as code? How do you architect to make it all work together?

1. A dead end started me backtracking

You’re going to dig your heels in by getting your application working. To do that you’ll spinup a vpn, private public subnet, bastion boxes, ECS hosts to deploy containers to, and an application load balancer endpoint. Getting that all working wasn’t terrible. We even included a prometheus node, to give us some monitoring visibility. We even added our jenkins server into the mix. Do you see where this is going?

At a certain point we of course needed to destroy the whole setup, but didn’t want to destroy the CI pipeline. Duh! And what about monitoring? Lose all that data each time no way!

Read: Infrastructure provisioning – what is it and why is it important?

2. Organize around VPCs

After dragging yourself through that, you see a bit better. It’s like standing at 20,000 feet.

Your vpn is a logical collection of instances. A box that holds your application, provides security, and gets created and destroyed with it. You can even see in your Terraform code, a subnet requires a VPN id within which to create it. And an instance requires a subnet within which to create it. For security reasons the application instances will sit within PRIVATE subnets, and only bastion box & load balancers will sit in PUBLIC subnets.

TO my mind that means each environment DEV, STAGE, PROD all get their own vpn. This also allows you to control who can access stage & production, as they have their own bastion access points.

Read: How to hire a developer that doesn’t suck

3. Build a utility VPC

What you’ll also see from the above story is that you need a place to have business wide, non-application services sit inside. Welcome the UTILITY VPC!

This can contain prometheus, ELK or other log collection service, your jenkins or other CI pipeline, and any other services that don’t logically fit within the application VPC.

Related: Why generalists are better at scaling the web

4. A VPC should be ok to destroy and rebuild in another region – in one-click

When you use infrastructure code, you want to test, create & destroy often. That shouldn’t disrupt anything. That means all state data should sit outside of those instances. Logging data, send it to logstash or cloudwatch. Application state, keep that inside of an RDS instance. And you’ve tested those backups right?

Speaking of RDS, I encountered problems with Amazon’s own backup & restore. For my money, I had a lot of problems and ended up writing a custom db dump script. That may require a custom restore to, so buyer beware. Here’s my story though… I tried to build infrastructure as code with Terraform and Amazon and it didn’t go as i expected.

You also may encounter issues when you move across regions, such as elastic IPs and so forth. And you’ll need to check and verify the code which creates and destroy S3 buckets and domain name certs. These areas gave me some hiccups, but you can work through it with diligence!

Read: Is zero downtime even possible on RDS?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

How crazy can Kubernetes get?

I was flipping through Reddit and found this hilarious post referencing a Scott Adams Dilbert strip on Kubernetes.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

What I found even funnier were the comments on the Reddit thread. Read on for fun!

1. Watch your memory

One engineer said he had a dockerized app running with 3GB of memory serving just 7 customers.

Not to be outdone another ops guy pipes in that he has one using 180Gb of memory serving just a few hundred customers.

Of course this is the internet, and along comes a guy who has an app using 1TB of memory, with only one user!

Optimization be damned!

Read: Infrastructure provisioning – what is it and why is it important?

2. Beware growing application complexity

As you dockerize your application, you can support multiple versions of software and packages. This can keep you flexible but also enable engineers to kick the can down the road. Technical debt is real!

What’s more each microservice *can* be on a different stack, using a different language and framework. But just because you *can* do something doesn’t mean you should.

Though docker & kubernetes will enable the above, keep in mind your team has to support it. Using some cool new language that hasn’t really achieved critical mass? Remember the engineer who championed that, and built your business crown jewels on top of that, will eventually leave. And when he or she does, you will be faced with the challenge of finding someone who knows the stuff!

Related: 6 Devops interview questions

3. What is a microserved monolith?

Well it’s not really a thing, except it sounds fun. And a bit absurd. If all those docker containers never get optimized, they probably have layer upon layer of useless stuff. Start with a smaller base image, dont include debug stuff, and extra layers. And cleanup after installing packages.

Here’s a more detailed howto optimizing Docker image sizes.

Read: Is zero downtime even possible on RDS?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

How to think like a senior engineer

via GIPHY

I just read this story
the Art of interrupting engineers. While much of what I read was pretty obvious from the engineer’s perspective, the product & project manager perspective was illustrative.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

There are definitely things that senior or more experienced engineers do differently. And those things can be learned.

Here are my thoughts…

1. Document all the things

By documenting all the little pieces you are working on you gain in a few ways. You communicate to management complexity they may not see. You buy yourself more time.

Finally you may even help yourself see implicit tasks more clearly. Whenever I hear myself or someone else utter the phrase “that should be easy”, I know I’m onto one of those mysterious tasks that seems simple but never is.

Be relentless. Break big tasks into smaller ones, and ticket each and every darn one!

Also: How can 1% of something equal nothing?

2. Communicate more and better

If you’re doing agile, chances are you’re probably joining a standup everyday.

Those are opportunities to share what is blocking.

o What tradeoffs are you struggling with?
o What technical debt is slowing you down?
o What new discoveries may require a rework of the timeline?

There is surely an art buried in communication. You want to be descriptive. You don’t want to come off complaining. You want to educate stakeholders. Beware of coming off as dismissive.

Related: Is maintenance sometimes a forgotten art?

3. Anticipate. Under promise & over deliver.

If you’ve gotten in the weeds with a particular API before, you’re likely to have a gut feeling about how new features and changes may go well or poorly. Or you may have dug through the comments. Maybe you didn’t find any!?

Or maybe the particular codebase sits on top of an interface or library you haven’t worked with before. So there will be a bit more of a learning curve. Whatever the case, try to promise less than you think you can really deliver on.

You can always finish extra tickets and over deliver later!

Read: What tools and technologies are devops engineers using today

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Why maintenance sometimes a forgotten art?

via GIPHY

Just finished reading the excellent
Why do people neglect maintenance?.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

With a wide ranging discussion, from cultural myths to cognitive biases, there are a lot of reasons why your organization may not be giving maintenance the attention it deserves.

Here are some thoughts…

1. Weighing short & long term tradeoffs

When looking at maintenance problems, we sometimes must weigh easy to implement quick fixes, versus the better though weightier longer term fix.

Often the longer term fix requires more downtime, and so is a harder pill to swallow. So sometimes taking that medicine is put off till later. Weighing how much that could cost you is not easy. But that is often the reality of maintenance and ops teams.

Read: How to hire a developer that doesn’t suck

2. Keeping it sexy

One interesting point they make in the article is around the culture of innovation. Since building new products and changing the world is sexy, some how the day-to-day realities of managing and maintaining that after delivery can take a back seat.

Just as we prioritize creating new features, and building a changed product, we must always weigh the costs of such change. As I wrote in the four letter word dividing dev and ops, different team members have different mandates. And that’s important.

While developers are mandated with bringing new features to life, ops are mandated with keeping things running at 4am. Software updates, maintenance, outages are what the ops team is worried about.

Related: Does Amazon have a dirty little secret?

3. Status symbols – dev versus ops

There is some interesting discussion by Andy Jess & Lee about how these different job roles can be viewed as low or high status in an organization.

In my article why we need techops I mention some of this narrative. Once at a keynote, I heard a sales guy advocate a product that would obviate your needing to hire ops. Yep, more money to hire Devs!

On the flip side at ops and DBA conferences, I’ve heard over and over the story of “some idiot developer that took down our production systems…”

You get it on both sides. When will teams ever work together?

Read: The art of resistance – when you have to be the bad guy

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Bye bye Kubernetes

via GIPHY

James Sanders at TechReplublic interviews Corey Quinn,
Is Kubernetes better for building a resume than a platform?

Corey is the snarky, observant, and talented author behind Last Week in AWS. If you haven’t subscribed you’re definitely missing out!

Join 35,000 others and follow Sean Hull on twitter @hullsean.

The observation about Kubernetes is not new. My friend & colleague over at Smash Company, Lawrence Krubner, wrote an exhaustive piece commenting on the flaws of docker, kubernetes and their eco-system.

It’s October 7th, 2019. I will revisit the question of Kubernetes demise again in six months. First thing in April 2020 we can see where my prediction stands!

1. Senior engineers are asking hard questions

The new generation of engineers, are more open to new solutions. They’re not bogged down by history, and legacy limitations. They’ll try new things, and embrace new technologies if they can solve a problem.

But senior engineers have been bitten. They’ve had the sorry job of untangling over engineered solutions. They’ve seen small firms live and die by wrong choices, and tech stack decisions. They’ve hit the wall on architectures, that were designed without enough perspective and foresight.

Read: What did Matt Ranney discover scaling Uber to 1000 microservices?

2. Trends come come and go

Does anyone out there remember Xen? Probably not. That’s because AWS squashed it like a little ant.

Kubernetes could go the same way. Not least because AWS has their own ECS.

Related: Can humility help you in your career?

3. Complexity is getting out of control

Kubernetes is sure great for doing blue-green deployments. And dockerizing all the things, helps developers and ops teams work together. After all just handing over the container asset, means ops can concentrate on deploying that. And kubernetes is great right up to this point.

But what about when kubernetes itself introduces application problems?

What about when kubernetes introduces performance and scalability problems?

And don’t even get me started about the crazy problems of running a database on Kubernetes.

Read: What happened when I offered advice outside my pay grade?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

What tools & tech are devops engineers using today

via GIPHY

I just stumbled upon Graham King’s blog, and I’m liking his writing. He wrote an excellent piece a developer goes to a DevOps conference.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

I’ve been to plenty of Unix & operations type conferences over the years, so topics don’t surprise me. But hearing about a developer’s experience brings a new perspective and some great insights.

1. Tools change but mindset stays the same

Some talk about Devops as doing away with operations. Those job roles just aren’t necessary anymore. Well maybe for a small firm, or maybe shops that have pushed 2-pizza agile to the max. But handing the operations duties to developers has limitations. As I mentioned here (the difference between dev and ops is a four letter word…) these different job roles have different mandates.

It’s like an architect can design a building, and it can be a very beautiful house. But a super or building manager keeps it running over the years. He or she knows what to look for in cracked roofs, knows how to keep rodents & pests at bay, knows how to repair and maintain & stay ahead of the game.

In that analogy, the architect is the developer, while the super or building manager is the operations team. They’re two different mindsets, rarely shared in one person.

Read: What did Matt Ranney discover scaling Uber to 1000 microservices?

2. Being on-call is a b*tch

I could write volumes about being on-call. Getting woken up in the middle of the night, because someone pushed broken code is no fun. What’s more broken can have different meanings.

Broken can be something QA should catch, like a button doesn’t work or there’s an issue with some browser. It could also be that some new product feature doesn’t work properly.

But from the ops perspective, broken could also be some new feature doesn’t scale. It makes a million API calls, or makes a servless call that times out. These types of broken are much harder to test for.

This is also why traditionally operations and development were two different teams. Because from the vantage of the business, they had different mandates.

Ops was mandated with stability. So they don’t want change. Change breaks things, and wakes you up at 3am.

Devs are mandated with features changes, and product improvement. So they naturally bring change to the table.

And between the two we search for balance. I wrote a piece that hit on exactly these points the difference between dev and ops is a four letter word…

Related: Can humility help you in your career?

3. The kingmaker tools

Kubernetes – you’ve heard of it, you’re probably using it. Devs package their app as a docker container, and ops push that container through CI/CD pipeline, and finally orchestrate & deploy with kubernetes. Seems like the *only* way to do things these days, right?

But some argue Docker may not be right for everyone and certainly this stack brings a *lot* of complexity for small organizations.

Related: Is AWS too complex for small dev teams?

Terraform I’m a big fan of this technology. Once you’ve captured your entire stack in code, you can version it, check it into git, and manage it like any other asset. That’s great, but there are so many other benefits. You can easily deploy that same stack in another region, or tweak it to create dev, stage and production. Cool stuff!

Related: I tried to build infrastructure as with Terraform and AWS. It didn’t go as I expected

Ansible All those BASH scripts you have sitting around? Check them into version control before it’s too late! One great thing about Ansible is with slight tweaks and can run those bash scripts almost as-is.

And for ops who already have experience with managing things by hand, you can get up to speed with Ansible, in a few days. The learning curve isn’t as tough as Puppet or Chef, and brings many or most of the benefits.

Packer Here’s another cool tool. Chances are all those AMI’s that Amazon has pre-baked, need tweaks for your setup. Now you could do all that work post spinup with Ansible. And that’s fine. But it’ll be slower, and possibly prone to breaking if the base AMI changes.

Enter Packer, another great tool from the folks who brought us Terraform, Hashicorp. This tool allows you to write yaml files that then build AMI’s. You can then use your pipeline and other automation tools to automate those as well. Cool !

Read: What happened when I offered advice outside my pay grade?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

I’m interviewed in a new book Devops Paradox

I had the good fortune to be interviewed earlier in the year by Viktor Farcic. I’m excited to see the book finally released. You can pickup your own copy of Devops Paradox here.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

Our interview is pretty wide ranging, touching on aspects of database administration, cloud operations, automation and devops.

1. Defining Devops

Viktor Farcic: Moving on to a more general subject, how would you define DevOps? I’ve gotten a different answer from every single person I’ve asked.

Sean Hull: I have a lot of opinions about it actually. I wrote an article on my blog a few years ago called The Four-Letter Word Dividing Dev and Ops, with the implication being that the four-letter word might be a swear word, akin to the development team swearing at the operations team, and the operations team swearing at the development team. But the four-letter word I was referring to was “risk.”
To summarize my article, in my view the development and the operations teams of old were separate silos in business, and they had very different mandates. Developers are tasked with writing code to build a product and to answer the needs of the customers, while directly building change into and facilitating a more sophisticated product. So, their thinking from day to day is about change and answering the requirements of the products team.
On the other hand, the operations team’s mandate is stability. It’s, “I don’t want these systems going down at 2:00 a.m.” So, over the long term, the operations teams are thinking about being as conservative as possible and having fewer moving parts, less code, and less new technologies. The simpler your stack is, the more reliable it is and the more robust and less likely it is to fail. I think the traditional reason why developers and operations teams were separated into silos was because of those two very different mandates.
They’re two different ways of prioritizing your work and your priorities when you think about the business and the technology. However, the downside was that those teams didn’t really communicate very well, and they were often at each other’s throats, pushing each other in opposite directions. But to answer your question, “what is DevOps?” I think of DevOps as a cultural movement that has made efforts to allow those teams to communicate better, and that’s a really good thing.

Read: What happened when I offered advice outside my pay grade?

2. How to adapt to change

Viktor Farcic: I have the impression that the speed with which new things are coming is only increasing. How do you keep up with it, and how do companies you work with keep up with all that?

Sean Hull: I don’t think they do keep up. I’ve gone to a lot of companies where they’ve never used serverless. None of their engineers know serverless at all. Lambda, web tasks, and Google Cloud functions have been out for a while, but I think there are very few companies that are able to really take advantage of them. I wrote another article blog post called Is Amazon Web Services Too Complex for Small Dev Teams? where I sort of implied that it is.
I do find a lot of companies want the advantage of on-demand computing, but they really don’t have the in-house expertise yet to really take advantage of all the things that Amazon can do and offer. That’s exactly why people aren’t up to speed on the technology, as it’s just changing so quickly. I’m not sure what the answer is. For me personally, there’s definitely a lot of stuff that I don’t know. I know I’m stronger in Python than I am with Node.js. Some companies have Node.js, and you can write Lambda functions in Java, Node.js, Python, and Go. So, I think Amazon’s investment in new technology allows the platform to evolve faster than a lot of companies are able to really take advantage of it.

Read: What did Matt Ranney discover scaling Uber to 1000 microservices?

3. What does the future hold for Devops?

Viktor Farcic: I’m going to ask you a question now that I hate being asked, so you’re allowed not to answer. Where do you see the future, let’s say a year from now?

Sean Hull:
I see more fragmentation happening across the technology landscape, and I think that that is ultimately making things more fragile because, for example, with microservices, companies don’t think twice about having Ruby, Python, Node.js, and Java. They have 10 different stacks, so when you hire new people, either you have to ask them to learn all those stacks or you have to hire people with each of those individual areas of expertise. The same is true with all these different clouds with their own sets of features: there’s a fragmentation happening.
Let’s look at the iPhone as an example. Think about how complex application testing is for Android versus the iPhone. I mean, you have hundreds of different smartphones that run Android, all with different screen sizes, different hardware, different amounts of memory, and the underlying stuff. Some may even have some extra chips that others don’t have, so how do you test your application across all those different platforms?
When you have fragmentation like that, it means the applications end up not working as well. I think the same thing is happening across the technology spectrum today that happened 10 to 15 years ago, where for your database backend there was Oracle, SQL Server, MySQL, and Postgres. Maybe somebody who’s a DB2 enterprise customer uses DB2, but now there are hundreds of open source databases, graph databases, and DynamoDB versus Cassandra, and so on and so on. There’s no real deep expertise in any of those databases.
What ends up happening is you have cases like what happened with customers who were using MongoDB. They found out the hard way about all of the weird behaviors and performance problems it had, because there just weren’t people around with deep knowledge of what was happening behind the scenes, whereas in Oracle’s space, for example, there are career DBAs that are performance experts that specialize in Oracle internals, so you can hire somebody to solve particular problems in that space.
There aren’t, as far as I know, a lot of people with MongoDB internals expertise. You’d have to call MongoDB themselves; maybe they have a few engineers that they can send out, so what’s the future? I see a lot of fragmentation and complexity, and that makes the internet and internet applications more fragile, more brittle, and more prone to failure.

Related: Can humility help you in your career?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

What do senior engineers do differently?

via GIPHY

I recently stumbled up on this piece at Pivotal, The art of interrupting software engineers. It caught my attention because i like to read from a different vantage from my own.

What is it like to interrupt us?

Join 35,000 others and follow Sean Hull on twitter @hullsean.

What I really got from the article though, was how different types of engineers will think about problems differently.

1. Under the hood

For junior engineers who are still a bit green, and new to working in industry, they’re downward facing. Focusing solely under the hood, they may not see how their work contributes to a product, or how it fits into the overall picture for the business.

“When asked about their progress on a story, they would make an effort to ensure I understood what was happening under the hood and what tradeoffs they were facing using a vocabulary I was familiar with. ”

For her it was technical competence that stood out – or at least not the only thing – but rather how they were strategizing, and communicating their problems, challenges, and progress.

Read: What happened when I offered advice outside my pay grade?

2. Communicate discoveries

A more intermediate engineer, would sometimes anticipate & communicate better.

“In some cases, those engineers would come to me before I even had a chance to enter the paranoid zone and give me a simple explanation of how the team had learned new information since they first estimated the complexity of a story. ”

She is also speaking of situational awareness. So not working in a vacuum, communicating & incorporating that new information as it becomes available.

Read: What did Matt Ranney discover scaling Uber to 1000 microservices?

3. Anticipate in advance

Senior engineers, she says would really stay ahead of the curve. They were even anticipating what might be a roadblock for her product delivery.


“Some of the very best practitioners would ask me in advance how urgently we should deliver a particular user story and what we were ready to give up in order to ship faster.”

Weighing tradeoffs, and prioritizing is a huge factor in velocity. If you can tame that beast, you’ll go very far indeed!

Related: Can humility help you in your career?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Viktor Farcic Interview excerpts

via GIPHY

I recently did an interview with Viktor Farcic all about operations, DBA & Devops. Here are some excerpts: What does Dev-Ops mean?

Join 35,000 others and follow Sean Hull on twitter @hullsean.

Continuing where I left off, I’ve included a few more highlights below. Enjoy!

1. Can I use a tool to migrate to the public cloud?

Viktor Farcic: I’ve seen quite a few of these tools that tell you if you buy our tool, we’re going to transfer whatever you have to the cloud. For example, Docker announced in the last DockerCon that they’re going to put in containers without a single change and everything will work. What do you think about that?

Sean Hull Salespeople often simplify things quite a bit in order to sell a product; in my experience, the devil is in the detail. It’s not to say that an automation tool like that might not be valuable and useful. It might be a good first step to getting your application in the cloud, and it might be an easier way than to rebuild everything one by one. But I doubt that it’s going to work magically just by one script.
EC2 instances, for example, have different performance characteristics, not only in terms of the disk I/O, memory, and CPU, but in smaller instances, they actually throttle the network access so you might spin up an instance and it just might not behave well. It might take time. In fact, all sorts of things could happen. You might have written MySQL scripts that assume you have root access to the server and then you rebuild that in an RDS and you get errors because you don’t have access to those resources on RDS. There’s a lot of things to consider.

Read: What happened when I offered advice outside my pay grade?

2. How do you adapt to change?

Viktor Farcic: I have the impression that the speed with which new things are coming is only increasing. How do you keep up with it, and how do companies you work with keep up with all that?

Sean Hull: I don’t think they do keep up. I’ve gone to a lot of companies where they’ve never used serverless. None of their engineers know serverless at all. Lambda, web tasks, and Google Cloud functions have been out for a while, but I think there are very few companies that are able to really take advantage of them. I wrote another article blog post called Is Amazon Web Services Too Complex for Small Dev Teams? where I sort of implied that it is.
I do find a lot of companies want the advantage of on-demand computing, but they really don’t have the in-house expertise yet to really take advantage of all the things that Amazon can do and offer. That’s exactly why people aren’t up to speed on the technology, as it’s just changing so quickly. I’m not sure what the answer is. For me personally, there’s definitely a lot of stuff that I don’t know. I know I’m stronger in Python than I am with Node.js. Some companies have Node.js, and you can write Lambda functions in Java, Node.js, Python, and Go. So, I think Amazon’s investment in new technology allows the platform to evolve faster than a lot of companies are able to really take advantage of it.
Read: What did Matt Ranney discover scaling Uber to 1000 microservices?

3. What is the future of Devops?

Viktor Farcic: I’m going to ask you a question now that I hate being asked, so you’re allowed not to answer. Where do you see the future, let’s say a year from now?

Sean Hull:
I see more fragmentation happening across the technology landscape, and I think that that is ultimately making things more fragile because, for example, with microservices, companies don’t think twice about having Ruby, Python, Node.js, and Java. They have 10 different stacks, so when you hire new people, either you have to ask them to learn all those stacks or you have to hire people with each of those individual areas of expertise. The same is true with all these different clouds with their own sets of features: there’s a fragmentation happening.
Let’s look at the iPhone as an example. Think about how complex application testing is for Android versus the iPhone. I mean, you have hundreds of different smartphones that run Android, all with different screen sizes, different hardware, different amounts of memory, and the underlying stuff. Some may even have some extra chips that others don’t have, so how do you test your application across all those different platforms?
When you have fragmentation like that, it means the applications end up not working as well. I think the same thing is happening across the technology spectrum today that happened 10 to 15 years ago, where for your database backend there was Oracle, SQL Server, MySQL, and Postgres. Maybe somebody who’s a DB2 enterprise customer uses DB2, but now there are hundreds of open source databases, graph databases, and DynamoDB versus Cassandra, and so on and so on. There’s no real deep expertise in any of those databases.
What ends up happening is you have cases like what happened with customers who were using MongoDB. They found out the hard way about all of the weird behaviors and performance problems it had, because there just weren’t people around with deep knowledge of what was happening behind the scenes, whereas in Oracle’s space, for example, there are career DBAs that are performance experts that specialize in Oracle internals, so you can hire somebody to solve particular problems in that space.
There aren’t, as far as I know, a lot of people with MongoDB internals expertise. You’d have to call MongoDB themselves; maybe they have a few engineers that they can send out, so what’s the future? I see a lot of fragmentation and complexity, and that makes the internet and internet applications more fragile, more brittle, and more prone to failure.

Related: Can humility help you in your career?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Consulting outside my pay grade

via GIPHY

I recently worked with a customer migrating to Azure. Part of the plan was to take advantage of various free credits available on Azure. That’s not a bad reason cost wise, as long as the engineering cost of migration is considered.

Then I dug further…

Join 35,000 others and follow Sean Hull on twitter @hullsean.

Sean: Why are you migrating to Azure?
Customer: We’re getting free credit from Microsoft. Also we must not have any services on AWS.
Sean: Why is that?
Customer: We are trying to close to a deal with Walmart. Of course they’re a big fish for us and we don’t want to lose them.
Sean: Ummm… let’s take a closer look at that.

o we need to migrate because our future customer is in a pissing match with AWS
o sales should not blindly drive tech decisions
o there will always be some things on AWS
o if you’re going to be non-technically pure then just say yes we have *everything* off of AWS and keep running your business
o if sales can lie, then go ahead and blow the smoke up there

You can’t be completely free of AWS

While the sales team thought that was a simple directive, they forgot a few details.

1. There is a financial cost to the business to migrate from AWS to Azure
2. There is a technical risk of migrating from AWS to Azure
3. There are hidden and unforeseen technical costs as things fail, or code unravels.
4. As you have THIRD PARTY apps, you are using AWS, such as Slack, Salesforce, Zoom, Dropbox, Jira and perhaps many others.
5. Internet traffic flows through aws networks.

Read: The myth of five nines – why high availability is overated

Sales & engineering have different mandates

Sales teams are driven in a different way than engineers. Their day-to-day is about closing deals, and so priorities are different. What’s more the grasp of technical minutiae may not always be as deep.

This is not a criticism, more an observation. So too engineers don’t understand the realities of sales, and closing deals.

That said you get into the danger zone when sales driven calculation pushes the business to make a bad decision.

The engineering marvel is what holds up the bridge. So it doesn’t make sense to skip the inspections, or leave out trusses because a customer said the bridge will look prettier that way. The engineers must have serious input on technology decisions.

That will then help the business make a balanced decision, one that considers the true costs involved.

Read: Relational Database – What is it and why is it important?

Engineering & Sales should be cooperative

One should not outweigh or outmuscle the other. Their needs to be a balance of concerns. A big chunk of business that doubles the underlying revenue may be worth large tech stack changes.

But there is also a risk that the house of cards comes tumbling down. Even if the reengineering risks are managed, the new debts owed by Rome could still take you down.

Related: Why generalists are better at scaling the web

How does pay grade come in?

In the above case I was hired to do a migration. I wasn’t hired to think strategically about the business.

So some might argue that is stepping outside the mandate or scope of work. That said I do try to offer opinions where they make sense. In this case it may be of value to the business.

However, since it is “outside my pay grade” I would offer it humbly, and gently. No sense shooting yourself in the foot. 🙂

Related: Why generalists are better at scaling the web

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters