Do simpler systems fail better?

via GIPHY

I was recently reading Greg Kogan’s blog Simple Systems have less downtime.

It really caught my attention.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

As a professional services consultant over the years, I’ve worked with almost 200 firms. And many of those required unraveling of complex systems. And systems that were no longer well understood after the first wave of builders have long since gone.

So this topic resonates strongly for me.

I believe if firms adopted these advice, I would have a lot less work over the years. Seriously.

1. Redundancy

Redundancy means backup systems. If your laptop fails, do you have a second one with all your up-to-date data? If us-east-1 fails, do you have a backup or live copy of your database in another region?

Redundancy isn’t just backup systems, it is backup people. If Jane who manages Salesforce gets in an accident, what will the business do to support the sales teams? If a system gets hacked and compromised, how can you restore the most recent data?

Complex systems fail in surprising ways. Having a plan B, and plan C, and for really essential services and plan D will save the day.

Take Greg’s example of a container ship:
o if the automatic system fails you can steer the thing manually. Wow!
o if other electronics fail, you can control the damn rudder by hand!

Incredible to think a ship that big is basically a giant sailboat when you disengaged the powered systems. That is truly a lesson for all of us startup engineers.

Read: How can 1% of something equal nothing?

2. Overlapping skillsets

If you only have one guy who knows how to use the database platform, that’s a problem. If you have only one woman who knows how to program in Rust, that’s a problem. If there’s only one person who knows how the reporting system works, and can make changes, that’s a problem.

Better to have overlapping job roles and skillsets. If you have a chance to adopt a new technology, make sure it’s rock solid one that is mainstream, and easy to hire for.

Related: Is Fred Wilson right about dealing in an honest, direct and transparent way?

3. Beware of technical debt

We’ve all heard the reasons.


o We don’t have the luxury to fix that now.

o We can’t afford the downtime.

o We have pressing features to ship.

But as technical debt piles on, so does complexity. And you’ll quickly end up end up carrying a larger burden than you realized.

As advocated by Kogan, rip and replace is often a more serious solution, and better for the firm. Yes you’ll have some downtime. Yes you’ll redirect team members temporarily. But you’ll solve the real problem, and will bring more simplicity to your architecture.

What’s more the pain of paying down the debt will make you think twice about borrowing in the future!

Related: What mistakes did you make when starting as a consultant?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

What are the top posts of the past year?

via GIPHY

I dug through google analytics to see which posts have been the most popular.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

In the past year, the following posts stuck out…

1. Tech

Bye Bye Kubernetes

How can we keep cloud architectures simple?

Are shared databases back in vogue?

What Matt Ranney learned scaling Uber

What does Devops mean?

How crazy can Kubernetes get?

When should I use ansible vs packer vs terraform?

Read: Are pioneers and process people different breeds?

2. Process

How I use 5 daily habits to stay on track

What happens when you offer consulting advice outside your pay grade?

What do senior engineers do differently?

What tools are devops engineers using today?

Do you fear you are an imposter?

What dothe best engineers do better?

Related: What happens when a bartender doesn’t get the job and files a lawsuit instead?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Are pioneers & process people different breeds?

via GIPHY

I was just reading Oliver Eidel’s blog. He had a great post, with some provocative ideas. Pioneers versus Process people.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

Now since I love to play devils advocate, after reading this thought provoking article, I thought I would write a bit of my take.

1. Are pioneers and process really characteristics of two different people?

As Oliver describes it, there are two completely different humans. Are they different breeds? Different species perhaps? Whatever the case, he argues that people either have one characteristic or the other.

To me the pioneer, is the engineering expression of creativity. They want to create new things, run with an idea, and see if they can make something happen. But even the most stodgy people have a bit of creativity in them, even if they don’t always express it. Yes I believe everyone has a tiny pioneer buried inside.

And so too, the pioneer, can should the need arise, buckle down and get tasks done. Yes they can be disciplined too, if you want to apply a different word.

Read: How can 1% of something equal nothing?

2. Do pioneer stage startups need more discipline?

I’ve worked at many a startup. I love them. Riding skateboards to work, and having dogs around the office is certainly a laid back downtown atmosphere, that appeals a lot more that the financial district buttoned up scene.

Also pioneer startups, often have a lot of leeway, to, as Oliver says “walking off in random directions, developing other great products”. This is how new ideas are hatched.

That said, many of these startups will one day fail. And it’s not for lack of idea, or market need. It’s often team related. That’s right, some of the best and coolest startups fail for lack of discipline. They’re too pioneering!

Related: Is Fred Wilson right about dealing in an honest, direct and transparent way?

3. Do process stage companies need more pioneering spirit?

I’ve also worked at quite a few larger, somewhat stodgy firms. These are so process heavy, that simple things can take forever. Everything requires a signoff. Deploying code can take weeks. Yes indeed they do things the old fashioned way, and that’s because they always have done that.

Could they use a pioneer or two to shake things up? Totally! Creative energy, drive and spirit could help them find faster ways to do things, and become unstuck when their old school process mentality is limiting their growth.

Related: What mistakes did you make when starting as a consultant?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

What happens when a bartender doesn’t get the job, but files a lawsuit?

via GIPHY

I stumbled on this interesting article, and thought I’d share it. Bartender doesn’t get job, but wins suit for consulting fees

Now some of you may have already made some prejudgements. But I ask that you hold your conclusions, and take a listen.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

Now let’s for a moment, distinguish big firm consultants, with independent consultants. I’m speaking about the latter, the freelancers of the world.

1. Time spent evaluating a consultant

If you’ve hired a consultant or freelancer before, you know you’ll spend time evaluating. You’ll talk about your business problem, and they’ll share how they can help. They may even start brainstorming with you. This alone can be valuable to a firm, as it can give them new ideas and new perspectives to dig into their problem.

Notice too, that while you are a fulltime employee, all that time on the phone, and at your desk, you are getting paid. While that freelancer, is *not* getting paid.

My point is not to complain here. Just merely to point out that there is a lot of work that happens before you are paying your consultant. Before they are even billing. Whether it is preparation, leveling up on knowledge, networking, business dinners, or prospecting.

So when you consider the cost of consulting, figure that there is 25-30% more time that they are *working* though it doesn’t show up on the invoice.

Related: A CTO must never do this

2. Straightforward or naive?

In the case of the bartender story, he did indeed spend time researching for the propsect. Whatever that may entail. There is a point where the consultant goes out on a limb. Some saavy is required to avoid misunderstandings, but ultimately both parties also need to be fair.

Although we all want to make an effort to be straightforward, despite what Fred Wilson says, parties are not always transparent. I would argue if you were too naive in this regard, you would not succeed in business.

Related: What mistakes did you make when starting as a consultant?

3. Avoid the legal route at all costs

Unlike the bartender, I personally would never choose legal arbitration. Is it ever really worth it? In the case of the bartender, he says $35/hr and 5.5 hours work, so $192.50. But how much time did he spend filing the suit? Paperwork, phone calls, emails, whatever. And then driving to the courthouse, cost of gas etc. No way this could come out cost-effective. To my mind many legal cases come down to ego. One party wanting to SHOW the other party they are wrong. Not worth it!

My feeling is always take the high road. Talk with them, and explain where you’re coming from. Also listen to them, and try to understand their perspective. There is always a middle ground that can be found.

Read: How to avoid legal problems in consulting

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Is banning facial recognition missing the point?

via GIPHY

I thought I would step out of my usual shoes this month and talk about something besides cloud computing. People sometimes ask my opinion on technology, as I know a thing or two about it.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

What are your thoughts on facial recognition? Will banning it solve the problem?

If you are not already familiar with Bruce Schneier you should be. He has been the single smartest person talking about data collection for the past twenty years. He wrote Database Nation, Secrets and Lies, and Beyond Fear. His thinking is non-obvious, insightful, deep and almost always spot on.

Here’s what Bruce Schneier has to say about banning facial recognition.

1. There are many ways to skin a cat

If you want to prevent what facial recognition can do, ban it, right? Well, turns out there are many other ways to do the same thing. You can identify people by their heart beat (think fitbit or apple watch), the way they walk, and of course good old fashioned fingerprints. And we leave those everywhere. What else?

Every phone broadcasts it’s ID which is the MAC address of it’s network interface. And if you have cameras without facial recognition, they can still identify using Iris scanning. Yep really.

Read: How do i migrate my skills to the cloud

2. Surveillance as a norm

When we say we don’t want facial recognition, we mean among other things that we don’t want anonymous identifying of people. But it also means we don’t want the later collection and identifying of people either.

Imagine you have a shoebox full of old photos. Photos at a beach, at a wedding, at tourist sites. Now you scan those into your computer, and you can identify all the people in the background. What a strange world we’ve built.

As Schneier points out, the larger question is what surveillance is okay and what is not? We as a society need to design rules and laws to outline how these technologies can and should be used for good, and to prevent their misuse and harm to people.

Related: 5 things toxic to scalability

3. The darkness of data brokering

The further collection of data by these large entities like facebook & google is more frightening still. Not for the data itself, but for it remaining completely unregulated. Government is still very behind what is happening at these giant companies.

Google knows things about your wife & husband that you don’t know. Google knows what the CEO of your competitor company is thinking and doing. Google knows your weaknesses, how and when you break the law. It’s hard to really grasp the scope. Every part of our online lives touches one of these companies. Even if you don’t use their services, you email people who do, and therefore are still known by them.

The laws we’ve built for the last century to prevent these types of abuses are mostly irrelevant to modern internet data companies. And as unregulated entities, they remain adversarial to citizens. We remain the product, not the customer.

Related: Did Disney have to fail?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

When should I use Ansible versus packer or Terraform?

via GIPHY

I was having a conversation with a colleague recently. We wee discussing devops, and the topic of Ansible came up as I was advocating it as a great too to get things done.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

Here’s what he had to say…

— quote —
I’ve tried using ansible a few times and this is what I found with it.

It is great for what it does. It’s wonderful to be able to spin up a new app or web server automatically. However what I have found for my needs is …

It is easier to build a piece of furniture than it is to explain all the steps required for someone else to build it. Or in order to replicated the steps automatically.

With cloud servers, it’s enough, for me that I’ve built it once. When I need to spin up another, I simply clone the working copy.

— unquote —

My thoughts below.

1. When is Terraform good

Terraform is a coss-platform infrastructure building tool. If you need an IAM user or S3 bucket, Terraform can create it. Need an ec2 instance of a particular type, deployed with an autoscaling group TF is a great tool for that.

With Terraform you can capture in code, everything about your application stack, so that you can standup a complete copy in another region, that’s powerful!

Read: How can 1% of something equal nothing?

2. When is packer right?

Packer is another useful tool that Devops can use to automate. Like AWS own EC2 Image Builder, it allows you to create the images that you boot your instances off of. Think of them as docker images for the server itself.

For example there are lots of dependencies your application requires, and you’ll install with your package manager. And there are services you want to start. You *could* use an ansible playbook to get these going, but better to build a new image that contains all the software you need on the box.

Packer easily sits into your CI pipeline, so you can have new software deployed and ready anytime.

The principal difference is that a new AMI requires you to spinup a new server. You can’t take action on a running server with this tool.

Related: Is Fred Wilson right about dealing in an honest, direct and transparent way?

3. When does Ansible make sense?

In particular here’s what my response was about Ansible itself.

— quote —

Absolutely. It’s an interesting balance to strike.

Because of course packer or EC2 image builder are very powerful and fit neatly into a CI pipeline. That said there are things ansible is nicely suited for too.

For example I want to distribute public keys onto specific servers. I have a yml file with the keys. I have a new developer starting, I have him or her git checkout branch, edit keys.yml, commit, push changes, then make a pull request. When the new keys.yml file gets merged, an ansible playbook kicks off to distribute the new set of keys to the relavant servers.

— unquote —

If you want to take actions on running servers, like deploying keys or other ongoing tweaks, that is where Ansible really shines.

Related: What mistakes did you make when starting as a consultant?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Should I join this new startup Delicious Data?

via GIPHY

I’ve been asked this before by folks.

Hey, you know technology, what stock picks would you recommend?

Join 35,000 others and follow Sean Hull on twitter @hullsean.

It’s a tough question, with a lot of intangibles. It’s no wonder people ask friends for advice. You have to think about what matters to you? Your free time? Your income? Your time to commute? What about the team you’re working with? Or what your job contributes to the world?

Many of those I can’t quantify for you. What you can quantify money, so it’s worth doing that!

1. What are their prospects for success?

When asked about the chances of a companies success, knowing the industry may be one small part. You also have to know how many competitors they have, and where they are along in the process. And it’s not just developing technology, but team dynamics that are huge. From what I hear VCs hire more for team than for idea.

What factors outside domain expertise come into play? Lots! The weather, financial markets, or the big guys like google or amazon coming into the market. They may not buy you, they may just replicate your idea. Then where are you?

Read: How to hack job search the smart way

2. How can I apply mathematics to money?

My answer is always the same, go for the S&P 500. If the S&P beats 90% of all stocks, then nine out of ten times you will win this way. That’s it, calculation done.

Yeah but how does that pertain to joining a startup?

How indeed. I still say invest in the index, not in one pony. So use that advice as you will.

Gambling on one company is something for gamblers. If you want to become a vc, that’s a different question. In that case you would do a lot of due diligence on team and idea, to be sure you’re putting your money in a smart place.

Can’t I do that as an employee? Yes sure, but the intangibles remain strong.

How can 1% of something equal nothing?.

Related: Is Fred Wilson right about dealing in an honest, direct and transparent way?

3. How does all this help me?

It leaves out the intangibles. Don’t count paper as part of your compensation package. If money is a key factor, divide the number of hours per year by your salary plus real benefits – health insurance and so forth – to come up with a real number. Compare that to other jobs.

The heck with these finance jobs that pay $200k and offer a $50k bonus, but ask you to work 90-100 hours per week. Why not get two $180k/yr jobs at 45 hours per week? You see the logic right?

And what else? Of course if you’re going to be commuting in to an office everyday, and joining the family, you want to have great coworkers. So make sure you like the place where you’re working. I don’t know how much this is worth to you, but I would say it’s quite valuable!

Related: What to do when prospects mislead you?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Does Amazon’s security work well for startups?

via GIPHY

I was sifting through my project & progress reports from former clients today. Something struck me loud and clear. It seems 4 out of 5 of them don’t implement VPC best practices.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

Which begs the question again and again, is the service just too damn complicated? I wrote about this topic before… Is aws a bit too complex for most or at least smaller dev teams?

1. No private subnets

What are those you ask? I really hope you’re not asking that.

The best practices way to deploy on amazon is using a vpc. This provides a logical grouping. You could have a dev, stage and prod vpc, and perhaps a utility one for other more permanent services.

Within that VPC, you want to have everything deployed in one or more private subnets. These are each mapped to a specific AZ in that region. The AZ mapps to a physical datacenter, a single building within that region. These private subnets have *NO route to the internet*.

How do you reach resources in the private subnet? You must be coming from the public subnet deployed within that same VPC. All the routing rules enforce this. The two types of resources that would be deployed in public subnet: load balancer for 80/443 traffic, and a jump or bastion box for ssh.

Read: How can 1% of something equal nothing?

2. Security groups with all ports open

Another thing that I see more often than you might guess is all ports open by some wildcard rule. *BAD*. We all know it’s bad, but it happens. And then it gets forgotten. We see developers doing it as a temporary fix to get something working and forget to later plug up the hole.

Even for security groups that don’t have this problem, they often allow port 22 from anywhere on the internet (0.0.0.0). This is unnecessary and rather reckless. Everyone should be coming from known source IPs. This can be an office network, or it can be some other trusted server on the internet. Or a block of IPs that you’ll always have assigned.

And of course don’t have your database port open. MySQL and Postgres don’t have particularly great protections here.

Related: Is Fred Wilson right about dealing in an honest, direct and transparent way?

3. No flowlogs enabled

Flowlogs allow you to log things at the packet level. Want to know about failed ssh attempts? Log that. What to know about other ports? Log that too.

If you are funneling all your connections through a jump box, then you can just enable flowlogs then you can configure your vpc flowlogs monitoring just for that box itself. You may also want to watch what’s happening with the load balancer too.

Flowlogs work at the network interface layer of your VPC, so you’ll need to understand VPCs in depth.

Related: What mistakes did you make when starting as a consultant?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Do you fear you are an imposter? Join the club

via GIPHY

I was reading another delicious hacker news thread, this time on a psychology question. How do you work with the fear of your own incompetence?

Join 35,000 others and follow Sean Hull on twitter @hullsean.

It’s a great question. I’ve had this suspicion for years, and it was only after stumbling on psychology books that I even knew it was a thing.

So how *do* you manage this fear?

1. Demonstrate that it is a fear

Fear is a funny thing. It can color reality. You may not even realize it’s happening. When it comes to imposter syndrome, prove yourself wrong. Do the work, and then step back and show yourself the evidence.

You’re a logical rational engineer, so you should be able to weigh the evidence, and see that you made a mistake.

Doing good work is not about perfectionism. It is about knowing you can execute, and delivering quality. That doesn’t not mean there are no imperfections. That means good enough. That means equal to or better than the team you’re working in.

That means you’re improving the bottom line for the firm you’re part of. Help them deliver new features, new code, new product. And help other team members do the same. That’s the name of the game.

Read: How can 1% of something equal nothing?

2. Look at your history

Whenever I have this feeling, I look at my own history. Then it makes me sorta chuckle. I have a list of twenty companies that I worked for recently, and they’ve all been really happy with my work.

How do I know I did good work? They paid me handsomely, paid me on time, and then recommended me to other colleagues.

That’s how I know I’m not an imposter. Am I perfect? Nope. Do I know everything? Nope? But I do good work, and I take ownership, admit when I’m wrong, and play well with others.

If you want to stand out, take a look at these two pieces:

Check out: What do the best engineers do better?

And this: How to think like a senior engineer

Those will help you on your way…

Related: Is Fred Wilson right about dealing in an honest, direct and transparent way?

3. Realize your perfectionism

I think a lot of engineers or bright people have this problem. They want everything to be perfect. They want to produce documents without spelling errors, and code without bugs. They want to deliver everything on time perfectly every time. And they want to feel they know everything.

But it doesn’t play to your benefit. People resent this type of thinking, and it’s unhealthy besides. Take a deep breath, realize we’re all working towards the same goal, and keep your eye on the ball. That means have a sense of humor. You’re probably *way* harder on yourself then others will ever be.

Related: What mistakes did you make when starting as a consultant?

4. Be easier on yourself and easier on others

As you begin to be “easier” on yourself, hopefully you’ll also be a little bit easier on others. Be patient with mistakes. Understand that people have a lot going on in their life. Notice that they are trying.

Sure even after you gain a sense of humor, there will be some people who are not trying, who don’t care or who are really incompetent. But have your default position be patience, and give them and yourself the benefit of the doubt.

Usually if said person is really that bad, others will also complain and the problem will come to management’s attention. It is their job, after all to manage the team as a whole, and keep it productive.

Have fun!

Related: Why did mailchimp fraudulently charge my credit card?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters