Lost and forgotten nuggets of ideas and advice

via GIPHY

I’ve been blogging for so long, sometimes, I forget about all the old material I’ve written.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

And I was just recently digging through some of the old titles, and thought it would be fun to repost some good ones.

1. What is it, and why is it important?

Infrastructure provisioning, what is it and why is it important?

Root cause analysis – what is it and why is it important?

Zero downtime – what is it and why is it important?

Stress testing – what is it and why is it important?

Data spot checks – what is it and why is it important?

Service monitoring – what is it and why is it important?

Decoupling – what is it and why is it important?

Read: What did Matt Ranney discover scaling Uber to 1000 microservices?

2. Thought provoking

Is AWS too complex for small dev teams?

The myth of five nines – Why high availability is overrated

Why are generalists better at scaling the web?

How to hire a developer that doesn’t suck?

What 5 things are toxic to scalability?

Is there a 4 letter word dividing dev and ops?

Related: Can humility help you in your career?

3. Consulting

Can progress reports help engagements succeed?

How do you handle the onboarding of a new engagement?

Why I ask clients for a deposit

How to avoid legal problems in consulting?

How best to do discovery in cloud devops engagements

When you’re hired to solve a people problem

When you have to take the fall

When clients don’t pay

Read: What happened when I offered advice outside my pay grade?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

When can I count time as project time?

via GIPHY

I was on the train looking at a Monday.com ad. I realized that the project management space is not dead and buried, but continuing to evolve everyday. While many of us spent years using tools like Jira, along comes an upstate to make project tracking simpler.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

The trouble is, I never felt it was the project management software that made tracking difficult.

For me there was always endless nuance, around tasks and tracking.

1. Time spent commuting or thinking off the chair

For me, when I get deep into the weeds of a customer and their technology stack, I’m thinking about problems as soon as I wake up. How can I tag those resources so they are region independent? How can I create the right S3 bucket policy so the application can write, but the world can’t?

What’s more if you’re like a lot of people, there’s some slack going on after hours, and emails too. Some of this may get tracked but inevitably there are hours not tracked.

An amount of estimating will probably happen. And creating a team policy on how to handle this ambiguity, is probably a good plan. Each project and company will be different, in terms of where to draw the line.

Read: What did Matt Ranney discover scaling Uber to 1000 microservices?

2. Crossover between projects or even customers

Sometimes there is a task which requires a bit of research, for example what’s the exact syntax in Terraform to create an IAM role, and attach it to an instance? You may spend an hour digging in, and then experimenting, to make sure you have the code right.

Then you may use that same snippet on another stack that you’re building, for department B and the same company, agile team C, or another customer entirely.

When you think about it, you carry a decade of learning into each new customer you work at. And they get all that learning, which translates into efficiency. And that’s effectively free.

So there is all sorts of ambiguity. And in each case you have to make a judgement call, when you’re tracking time.

Related: Can humility help you in your career?

3. Tasks grow and change

As you continue to work on a project, some tasks have a tendency to themselves unwind. As you dig deeper, you find vast caves yet to be explored. More excavation that needs to be done.

From there subtasks may hatch from the parent, and on it goes. But this will surely blowup initial estimates, and makes tracking progress often more of an art than a science.

Read: What happened when I offered advice outside my pay grade?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

I’m interviewed in a new book Devops Paradox

I had the good fortune to be interviewed earlier in the year by Viktor Farcic. I’m excited to see the book finally released. You can pickup your own copy of Devops Paradox here.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

Our interview is pretty wide ranging, touching on aspects of database administration, cloud operations, automation and devops.

1. Defining Devops

Viktor Farcic: Moving on to a more general subject, how would you define DevOps? I’ve gotten a different answer from every single person I’ve asked.

Sean Hull: I have a lot of opinions about it actually. I wrote an article on my blog a few years ago called The Four-Letter Word Dividing Dev and Ops, with the implication being that the four-letter word might be a swear word, akin to the development team swearing at the operations team, and the operations team swearing at the development team. But the four-letter word I was referring to was “risk.”
To summarize my article, in my view the development and the operations teams of old were separate silos in business, and they had very different mandates. Developers are tasked with writing code to build a product and to answer the needs of the customers, while directly building change into and facilitating a more sophisticated product. So, their thinking from day to day is about change and answering the requirements of the products team.
On the other hand, the operations team’s mandate is stability. It’s, “I don’t want these systems going down at 2:00 a.m.” So, over the long term, the operations teams are thinking about being as conservative as possible and having fewer moving parts, less code, and less new technologies. The simpler your stack is, the more reliable it is and the more robust and less likely it is to fail. I think the traditional reason why developers and operations teams were separated into silos was because of those two very different mandates.
They’re two different ways of prioritizing your work and your priorities when you think about the business and the technology. However, the downside was that those teams didn’t really communicate very well, and they were often at each other’s throats, pushing each other in opposite directions. But to answer your question, “what is DevOps?” I think of DevOps as a cultural movement that has made efforts to allow those teams to communicate better, and that’s a really good thing.

Read: What happened when I offered advice outside my pay grade?

2. How to adapt to change

Viktor Farcic: I have the impression that the speed with which new things are coming is only increasing. How do you keep up with it, and how do companies you work with keep up with all that?

Sean Hull: I don’t think they do keep up. I’ve gone to a lot of companies where they’ve never used serverless. None of their engineers know serverless at all. Lambda, web tasks, and Google Cloud functions have been out for a while, but I think there are very few companies that are able to really take advantage of them. I wrote another article blog post called Is Amazon Web Services Too Complex for Small Dev Teams? where I sort of implied that it is.
I do find a lot of companies want the advantage of on-demand computing, but they really don’t have the in-house expertise yet to really take advantage of all the things that Amazon can do and offer. That’s exactly why people aren’t up to speed on the technology, as it’s just changing so quickly. I’m not sure what the answer is. For me personally, there’s definitely a lot of stuff that I don’t know. I know I’m stronger in Python than I am with Node.js. Some companies have Node.js, and you can write Lambda functions in Java, Node.js, Python, and Go. So, I think Amazon’s investment in new technology allows the platform to evolve faster than a lot of companies are able to really take advantage of it.

Read: What did Matt Ranney discover scaling Uber to 1000 microservices?

3. What does the future hold for Devops?

Viktor Farcic: I’m going to ask you a question now that I hate being asked, so you’re allowed not to answer. Where do you see the future, let’s say a year from now?

Sean Hull:
I see more fragmentation happening across the technology landscape, and I think that that is ultimately making things more fragile because, for example, with microservices, companies don’t think twice about having Ruby, Python, Node.js, and Java. They have 10 different stacks, so when you hire new people, either you have to ask them to learn all those stacks or you have to hire people with each of those individual areas of expertise. The same is true with all these different clouds with their own sets of features: there’s a fragmentation happening.
Let’s look at the iPhone as an example. Think about how complex application testing is for Android versus the iPhone. I mean, you have hundreds of different smartphones that run Android, all with different screen sizes, different hardware, different amounts of memory, and the underlying stuff. Some may even have some extra chips that others don’t have, so how do you test your application across all those different platforms?
When you have fragmentation like that, it means the applications end up not working as well. I think the same thing is happening across the technology spectrum today that happened 10 to 15 years ago, where for your database backend there was Oracle, SQL Server, MySQL, and Postgres. Maybe somebody who’s a DB2 enterprise customer uses DB2, but now there are hundreds of open source databases, graph databases, and DynamoDB versus Cassandra, and so on and so on. There’s no real deep expertise in any of those databases.
What ends up happening is you have cases like what happened with customers who were using MongoDB. They found out the hard way about all of the weird behaviors and performance problems it had, because there just weren’t people around with deep knowledge of what was happening behind the scenes, whereas in Oracle’s space, for example, there are career DBAs that are performance experts that specialize in Oracle internals, so you can hire somebody to solve particular problems in that space.
There aren’t, as far as I know, a lot of people with MongoDB internals expertise. You’d have to call MongoDB themselves; maybe they have a few engineers that they can send out, so what’s the future? I see a lot of fragmentation and complexity, and that makes the internet and internet applications more fragile, more brittle, and more prone to failure.

Related: Can humility help you in your career?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

How do we make domain decisions when coding software?

via GIPHY

I stumbled on an interesting discussion over at hacker news about when you have to make a decision in code outside of your domain of expertise.

This is a super interesting topic to me. As the world is more and more powered by computers and algorithms, this seemingly obscure philisophical corner of computing begins to affect us all.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

While many programmers are familiar with the experience of stumbling upon an edge case which wasn’t covered in the spec. The truth is you can’t go back and block on every single tiny detail.

Some of the bigger ones can trigger further discussion, but were it not for Falsehoods programmers believe about X the software would never go out the door.

Here’s some more food for thought…

1. Falsehoods programmers believe about names

o They won’t be a *bad* word. Whose bad? i’m not sure.
o All people have names.
o We only have to work with names from English.
o Names are ordered in a sensible manner.
o A name won’t have to change later.

You get the picture. As you start working with the minutae around names, you realize there is a lot of assumptions. Even when you weed out the big ones, there are still small assumptions.

Ultimately, you can’t get to the bottom. There will always be some assumptions, and as software evolves, changes will be required to match the real world. Living systems grow. Software systems must receive *updates*.

Read: What happened when I offered advice outside my pay grade?

2. Falsehoods programmers believe about selling products

o each product has a price
o each product has one price
o each product is one unit
o two decimal places is enough for all currencies
o all currencies have a unicode symbol

There are many more. The point is once you start digging, everything gets fuzzy. Products can be in stock and not in the warehouse. Single product can have multiple prices. It goes on and on!

Read: What did Matt Ranney discover scaling Uber to 1000 microservices?

3. Falsehoods programmers believe about addresses

o two districts will not have the same number
o postal codes will agree across jurisdictions
o a fixed number of characters will work for every address
o roads have names
o buildings are not numbered zero

Addresses are *really* messy. I remember being surprised when I moved to brooklyn. Streeteasy & Google put me in different neighborhoods. I asked my lawyer why this was. He said DOB and the post office use different databases. And google further samples from one or the other.

Never the twain shall meet!

Related: Can humility help you in your career?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

How do you hire the best talent in a competitive marketplace?

via GIPHY

Kapwing, a scrappy startup out of SF wrote this excellent piece how they hired 10 employees in the very tight bay area market. It caught my attention because the demand for great people has never been higher.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

Here are the ways they found great people…

1. Events

Meetups are always an excellent way to find great people. Tech networking, can be one part selling your ideas, projecting your prowess or career building for your own people. And meanwhile career forward, self-actuated employees are scoping you out too.

Check out: What did Matt Ranney discover scaling Uber to 1000 microservices?

2. Friend referrals

Connecting through your own people is always a good way to find talent. These referrals are also great because you already have a handle on the hires real-world experience and personality. Good for everybody.

Also: What happened when I offered advice outside my pay grade?

3. Press

As your startup grows, you get some press in TechCrunch or ReCode. This generates some household recognition in the tech community, or posts on HackerNews. From there you get inbound requests because smart people are on the prowl for a great company. Done & done!

Related: Can humility help you in your career?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

What do senior engineers do differently?

via GIPHY

I recently stumbled up on this piece at Pivotal, The art of interrupting software engineers. It caught my attention because i like to read from a different vantage from my own.

What is it like to interrupt us?

Join 35,000 others and follow Sean Hull on twitter @hullsean.

What I really got from the article though, was how different types of engineers will think about problems differently.

1. Under the hood

For junior engineers who are still a bit green, and new to working in industry, they’re downward facing. Focusing solely under the hood, they may not see how their work contributes to a product, or how it fits into the overall picture for the business.

“When asked about their progress on a story, they would make an effort to ensure I understood what was happening under the hood and what tradeoffs they were facing using a vocabulary I was familiar with. ”

For her it was technical competence that stood out – or at least not the only thing – but rather how they were strategizing, and communicating their problems, challenges, and progress.

Read: What happened when I offered advice outside my pay grade?

2. Communicate discoveries

A more intermediate engineer, would sometimes anticipate & communicate better.

“In some cases, those engineers would come to me before I even had a chance to enter the paranoid zone and give me a simple explanation of how the team had learned new information since they first estimated the complexity of a story. ”

She is also speaking of situational awareness. So not working in a vacuum, communicating & incorporating that new information as it becomes available.

Read: What did Matt Ranney discover scaling Uber to 1000 microservices?

3. Anticipate in advance

Senior engineers, she says would really stay ahead of the curve. They were even anticipating what might be a roadblock for her product delivery.


“Some of the very best practitioners would ask me in advance how urgently we should deliver a particular user story and what we were ready to give up in order to ship faster.”

Weighing tradeoffs, and prioritizing is a huge factor in velocity. If you can tame that beast, you’ll go very far indeed!

Related: Can humility help you in your career?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Viktor Farcic Interview excerpts

via GIPHY

I recently did an interview with Viktor Farcic all about operations, DBA & Devops. Here are some excerpts: What does Dev-Ops mean?

Join 35,000 others and follow Sean Hull on twitter @hullsean.

Continuing where I left off, I’ve included a few more highlights below. Enjoy!

1. Can I use a tool to migrate to the public cloud?

Viktor Farcic: I’ve seen quite a few of these tools that tell you if you buy our tool, we’re going to transfer whatever you have to the cloud. For example, Docker announced in the last DockerCon that they’re going to put in containers without a single change and everything will work. What do you think about that?

Sean Hull Salespeople often simplify things quite a bit in order to sell a product; in my experience, the devil is in the detail. It’s not to say that an automation tool like that might not be valuable and useful. It might be a good first step to getting your application in the cloud, and it might be an easier way than to rebuild everything one by one. But I doubt that it’s going to work magically just by one script.
EC2 instances, for example, have different performance characteristics, not only in terms of the disk I/O, memory, and CPU, but in smaller instances, they actually throttle the network access so you might spin up an instance and it just might not behave well. It might take time. In fact, all sorts of things could happen. You might have written MySQL scripts that assume you have root access to the server and then you rebuild that in an RDS and you get errors because you don’t have access to those resources on RDS. There’s a lot of things to consider.

Read: What happened when I offered advice outside my pay grade?

2. How do you adapt to change?

Viktor Farcic: I have the impression that the speed with which new things are coming is only increasing. How do you keep up with it, and how do companies you work with keep up with all that?

Sean Hull: I don’t think they do keep up. I’ve gone to a lot of companies where they’ve never used serverless. None of their engineers know serverless at all. Lambda, web tasks, and Google Cloud functions have been out for a while, but I think there are very few companies that are able to really take advantage of them. I wrote another article blog post called Is Amazon Web Services Too Complex for Small Dev Teams? where I sort of implied that it is.
I do find a lot of companies want the advantage of on-demand computing, but they really don’t have the in-house expertise yet to really take advantage of all the things that Amazon can do and offer. That’s exactly why people aren’t up to speed on the technology, as it’s just changing so quickly. I’m not sure what the answer is. For me personally, there’s definitely a lot of stuff that I don’t know. I know I’m stronger in Python than I am with Node.js. Some companies have Node.js, and you can write Lambda functions in Java, Node.js, Python, and Go. So, I think Amazon’s investment in new technology allows the platform to evolve faster than a lot of companies are able to really take advantage of it.
Read: What did Matt Ranney discover scaling Uber to 1000 microservices?

3. What is the future of Devops?

Viktor Farcic: I’m going to ask you a question now that I hate being asked, so you’re allowed not to answer. Where do you see the future, let’s say a year from now?

Sean Hull:
I see more fragmentation happening across the technology landscape, and I think that that is ultimately making things more fragile because, for example, with microservices, companies don’t think twice about having Ruby, Python, Node.js, and Java. They have 10 different stacks, so when you hire new people, either you have to ask them to learn all those stacks or you have to hire people with each of those individual areas of expertise. The same is true with all these different clouds with their own sets of features: there’s a fragmentation happening.
Let’s look at the iPhone as an example. Think about how complex application testing is for Android versus the iPhone. I mean, you have hundreds of different smartphones that run Android, all with different screen sizes, different hardware, different amounts of memory, and the underlying stuff. Some may even have some extra chips that others don’t have, so how do you test your application across all those different platforms?
When you have fragmentation like that, it means the applications end up not working as well. I think the same thing is happening across the technology spectrum today that happened 10 to 15 years ago, where for your database backend there was Oracle, SQL Server, MySQL, and Postgres. Maybe somebody who’s a DB2 enterprise customer uses DB2, but now there are hundreds of open source databases, graph databases, and DynamoDB versus Cassandra, and so on and so on. There’s no real deep expertise in any of those databases.
What ends up happening is you have cases like what happened with customers who were using MongoDB. They found out the hard way about all of the weird behaviors and performance problems it had, because there just weren’t people around with deep knowledge of what was happening behind the scenes, whereas in Oracle’s space, for example, there are career DBAs that are performance experts that specialize in Oracle internals, so you can hire somebody to solve particular problems in that space.
There aren’t, as far as I know, a lot of people with MongoDB internals expertise. You’d have to call MongoDB themselves; maybe they have a few engineers that they can send out, so what’s the future? I see a lot of fragmentation and complexity, and that makes the internet and internet applications more fragile, more brittle, and more prone to failure.

Related: Can humility help you in your career?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Consulting outside my pay grade

via GIPHY

I recently worked with a customer migrating to Azure. Part of the plan was to take advantage of various free credits available on Azure. That’s not a bad reason cost wise, as long as the engineering cost of migration is considered.

Then I dug further…

Join 35,000 others and follow Sean Hull on twitter @hullsean.

Sean: Why are you migrating to Azure?
Customer: We’re getting free credit from Microsoft. Also we must not have any services on AWS.
Sean: Why is that?
Customer: We are trying to close to a deal with Walmart. Of course they’re a big fish for us and we don’t want to lose them.
Sean: Ummm… let’s take a closer look at that.

o we need to migrate because our future customer is in a pissing match with AWS
o sales should not blindly drive tech decisions
o there will always be some things on AWS
o if you’re going to be non-technically pure then just say yes we have *everything* off of AWS and keep running your business
o if sales can lie, then go ahead and blow the smoke up there

You can’t be completely free of AWS

While the sales team thought that was a simple directive, they forgot a few details.

1. There is a financial cost to the business to migrate from AWS to Azure
2. There is a technical risk of migrating from AWS to Azure
3. There are hidden and unforeseen technical costs as things fail, or code unravels.
4. As you have THIRD PARTY apps, you are using AWS, such as Slack, Salesforce, Zoom, Dropbox, Jira and perhaps many others.
5. Internet traffic flows through aws networks.

Read: The myth of five nines – why high availability is overated

Sales & engineering have different mandates

Sales teams are driven in a different way than engineers. Their day-to-day is about closing deals, and so priorities are different. What’s more the grasp of technical minutiae may not always be as deep.

This is not a criticism, more an observation. So too engineers don’t understand the realities of sales, and closing deals.

That said you get into the danger zone when sales driven calculation pushes the business to make a bad decision.

The engineering marvel is what holds up the bridge. So it doesn’t make sense to skip the inspections, or leave out trusses because a customer said the bridge will look prettier that way. The engineers must have serious input on technology decisions.

That will then help the business make a balanced decision, one that considers the true costs involved.

Read: Relational Database – What is it and why is it important?

Engineering & Sales should be cooperative

One should not outweigh or outmuscle the other. Their needs to be a balance of concerns. A big chunk of business that doubles the underlying revenue may be worth large tech stack changes.

But there is also a risk that the house of cards comes tumbling down. Even if the reengineering risks are managed, the new debts owed by Rome could still take you down.

Related: Why generalists are better at scaling the web

How does pay grade come in?

In the above case I was hired to do a migration. I wasn’t hired to think strategically about the business.

So some might argue that is stepping outside the mandate or scope of work. That said I do try to offer opinions where they make sense. In this case it may be of value to the business.

However, since it is “outside my pay grade” I would offer it humbly, and gently. No sense shooting yourself in the foot. 🙂

Related: Why generalists are better at scaling the web

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

What does DEV OPS mean?

via GIPHY

I was recently interviewed by Victor Farcic. He asked me a lot of interesting questions.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

One really struck me that I thought was important, about the whole devops movement.

How do you define devops

Viktor Farcic: Moving on to a more general subject, how would you define DevOps? I’ve gotten a different answer from every single person I’ve asked.

Sean Hull: I have a lot of opinions about it actually. I wrote an article on my blog a few years ago called The Four-Letter Word Dividing Dev and Ops, with the implication being that the four-letter word might be a swear word, akin to the development team swearing at the operations team, and the operations team swearing at the development team. But the four-letter word I was referring to was “risk.”

Related: Why generalists are better at scaling the web

To summarize my article, in my view the development and the operations teams of old were separate silos in business, and they had very different mandates. Developers are tasked with writing code to build a product and to answer the needs of the customers, while directly building change into and facilitating a more sophisticated product. So, their thinking from day to day is about change and answering the requirements of the products team.
On the other hand, the operations team’s mandate is stability. It’s, “I don’t want these systems going down at 2:00 a.m.” So, over the long term, the operations teams are thinking about being as conservative as possible and having fewer moving parts, less code, and less new technologies. The simpler your stack is, the more reliable it is and the more robust and less likely it is to fail. I think the traditional reason why developers and operations teams were separated into silos was because of those two very different mandates.

Also: Walking the delicate balance of transparency

They’re two different ways of prioritizing your work and your priorities when you think about the business and the technology. However, the downside was that those teams didn’t really communicate very well, and they were often at each other’s throats, pushing each other in opposite directions. But to answer your question, “what is DevOps?” I think of DevOps as a cultural movement that has made efforts to allow those teams to communicate better, and that’s a really good thing.

Read: One time in 2013 I had to take the fall

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

What Matt Ranney learned scaling Uber to 1000 services

via GIPHY

I was recently watching Matt Ranney’s 2016 talk from GOTO conference. He’s an excellent speaker, not least because of his formidable experience at the helm of Uber.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

Jump to 11:45 to start out…

1. What’s the cost of many languages

As they embraced microservices, they discovered surprising problems around having so many languages.

For example it became hard to share code and furthermore it became hard to move between teems.

The upshot is that the fragmentation of languages siloed teams, and built up tribes around those languages, and expertise or interest in them.

Related: How can we keep cloud architectures simple

2. What’s the cost of using RPC for everything

In the microservices world, everything becomes a remote procedure call. For starters there are problems with HTTP and related semantics.

Also there are challenges with JSON. Since there aren’t any types, you don’t have a schema to lock you into formats. So you get weird behaviors down the line.

The upshot: Servers are not browsers. Indeed. Function calls are wayyyy better *within* a datacenter. That’s because you control both servers, so you can tune accordingingly.

Also: What hidden things does a deposit reveal?

3. Tuning across languages can be very challenging

Different languages have different tools available. Can your favorite tool work across all the languages you’ve decided to support in your microservices?

Even though you’re going to go with microservices, think about standardizing across those services. Use tools that work on all of them. And when building your services, make sure it outputs the same dashboard metrics.

Read: Can communication mixups sour an engagement?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters