Categories
All Cloud Computing CTO/CIO Devops

When should I use Ansible versus packer or Terraform?

via GIPHY

I was having a conversation with a colleague recently. We wee discussing devops, and the topic of Ansible came up as I was advocating it as a great too to get things done.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

Here’s what he had to say…

— quote —
I’ve tried using ansible a few times and this is what I found with it.

It is great for what it does. It’s wonderful to be able to spin up a new app or web server automatically. However what I have found for my needs is …

It is easier to build a piece of furniture than it is to explain all the steps required for someone else to build it. Or in order to replicated the steps automatically.

With cloud servers, it’s enough, for me that I’ve built it once. When I need to spin up another, I simply clone the working copy.

— unquote —

My thoughts below.

1. When is Terraform good

Terraform is a coss-platform infrastructure building tool. If you need an IAM user or S3 bucket, Terraform can create it. Need an ec2 instance of a particular type, deployed with an autoscaling group TF is a great tool for that.

With Terraform you can capture in code, everything about your application stack, so that you can standup a complete copy in another region, that’s powerful!

Read: How can 1% of something equal nothing?

2. When is packer right?

Packer is another useful tool that Devops can use to automate. Like AWS own EC2 Image Builder, it allows you to create the images that you boot your instances off of. Think of them as docker images for the server itself.

For example there are lots of dependencies your application requires, and you’ll install with your package manager. And there are services you want to start. You *could* use an ansible playbook to get these going, but better to build a new image that contains all the software you need on the box.

Packer easily sits into your CI pipeline, so you can have new software deployed and ready anytime.

The principal difference is that a new AMI requires you to spinup a new server. You can’t take action on a running server with this tool.

Related: Is Fred Wilson right about dealing in an honest, direct and transparent way?

3. When does Ansible make sense?

In particular here’s what my response was about Ansible itself.

— quote —

Absolutely. It’s an interesting balance to strike.

Because of course packer or EC2 image builder are very powerful and fit neatly into a CI pipeline. That said there are things ansible is nicely suited for too.

For example I want to distribute public keys onto specific servers. I have a yml file with the keys. I have a new developer starting, I have him or her git checkout branch, edit keys.yml, commit, push changes, then make a pull request. When the new keys.yml file gets merged, an ansible playbook kicks off to distribute the new set of keys to the relavant servers.

— unquote —

If you want to take actions on running servers, like deploying keys or other ongoing tweaks, that is where Ansible really shines.

Related: What mistakes did you make when starting as a consultant?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Categories
All Consulting CTO/CIO Devops Hiring

Should I join this new startup Delicious Data?

via GIPHY

I’ve been asked this before by folks.

Hey, you know technology, what stock picks would you recommend?

Join 35,000 others and follow Sean Hull on twitter @hullsean.

It’s a tough question, with a lot of intangibles. It’s no wonder people ask friends for advice. You have to think about what matters to you? Your free time? Your income? Your time to commute? What about the team you’re working with? Or what your job contributes to the world?

Many of those I can’t quantify for you. What you can quantify money, so it’s worth doing that!

1. What are their prospects for success?

When asked about the chances of a companies success, knowing the industry may be one small part. You also have to know how many competitors they have, and where they are along in the process. And it’s not just developing technology, but team dynamics that are huge. From what I hear VCs hire more for team than for idea.

What factors outside domain expertise come into play? Lots! The weather, financial markets, or the big guys like google or amazon coming into the market. They may not buy you, they may just replicate your idea. Then where are you?

Read: How to hack job search the smart way

2. How can I apply mathematics to money?

My answer is always the same, go for the S&P 500. If the S&P beats 90% of all stocks, then nine out of ten times you will win this way. That’s it, calculation done.

Yeah but how does that pertain to joining a startup?

How indeed. I still say invest in the index, not in one pony. So use that advice as you will.

Gambling on one company is something for gamblers. If you want to become a vc, that’s a different question. In that case you would do a lot of due diligence on team and idea, to be sure you’re putting your money in a smart place.

Can’t I do that as an employee? Yes sure, but the intangibles remain strong.

How can 1% of something equal nothing?.

Related: Is Fred Wilson right about dealing in an honest, direct and transparent way?

3. How does all this help me?

It leaves out the intangibles. Don’t count paper as part of your compensation package. If money is a key factor, divide the number of hours per year by your salary plus real benefits – health insurance and so forth – to come up with a real number. Compare that to other jobs.

The heck with these finance jobs that pay $200k and offer a $50k bonus, but ask you to work 90-100 hours per week. Why not get two $180k/yr jobs at 45 hours per week? You see the logic right?

And what else? Of course if you’re going to be commuting in to an office everyday, and joining the family, you want to have great coworkers. So make sure you like the place where you’re working. I don’t know how much this is worth to you, but I would say it’s quite valuable!

Related: What to do when prospects mislead you?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Categories
All Consulting CTO/CIO Devops Software Development

Do you fear you are an imposter? Join the club

via GIPHY

I was reading another delicious hacker news thread, this time on a psychology question. How do you work with the fear of your own incompetence?

Join 35,000 others and follow Sean Hull on twitter @hullsean.

It’s a great question. I’ve had this suspicion for years, and it was only after stumbling on psychology books that I even knew it was a thing.

So how *do* you manage this fear?

1. Demonstrate that it is a fear

Fear is a funny thing. It can color reality. You may not even realize it’s happening. When it comes to imposter syndrome, prove yourself wrong. Do the work, and then step back and show yourself the evidence.

You’re a logical rational engineer, so you should be able to weigh the evidence, and see that you made a mistake.

Doing good work is not about perfectionism. It is about knowing you can execute, and delivering quality. That doesn’t not mean there are no imperfections. That means good enough. That means equal to or better than the team you’re working in.

That means you’re improving the bottom line for the firm you’re part of. Help them deliver new features, new code, new product. And help other team members do the same. That’s the name of the game.

Read: How can 1% of something equal nothing?

2. Look at your history

Whenever I have this feeling, I look at my own history. Then it makes me sorta chuckle. I have a list of twenty companies that I worked for recently, and they’ve all been really happy with my work.

How do I know I did good work? They paid me handsomely, paid me on time, and then recommended me to other colleagues.

That’s how I know I’m not an imposter. Am I perfect? Nope. Do I know everything? Nope? But I do good work, and I take ownership, admit when I’m wrong, and play well with others.

If you want to stand out, take a look at these two pieces:

Check out: What do the best engineers do better?

And this: How to think like a senior engineer

Those will help you on your way…

Related: Is Fred Wilson right about dealing in an honest, direct and transparent way?

3. Realize your perfectionism

I think a lot of engineers or bright people have this problem. They want everything to be perfect. They want to produce documents without spelling errors, and code without bugs. They want to deliver everything on time perfectly every time. And they want to feel they know everything.

But it doesn’t play to your benefit. People resent this type of thinking, and it’s unhealthy besides. Take a deep breath, realize we’re all working towards the same goal, and keep your eye on the ball. That means have a sense of humor. You’re probably *way* harder on yourself then others will ever be.

Related: What mistakes did you make when starting as a consultant?

4. Be easier on yourself and easier on others

As you begin to be “easier” on yourself, hopefully you’ll also be a little bit easier on others. Be patient with mistakes. Understand that people have a lot going on in their life. Notice that they are trying.

Sure even after you gain a sense of humor, there will be some people who are not trying, who don’t care or who are really incompetent. But have your default position be patience, and give them and yourself the benefit of the doubt.

Usually if said person is really that bad, others will also complain and the problem will come to management’s attention. It is their job, after all to manage the team as a whole, and keep it productive.

Have fun!

Related: Why did mailchimp fraudulently charge my credit card?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Categories
All Consulting CTO/CIO Devops Startups

What do the best engineers do better?

via GIPHY

I’m fascinated by this topic.

I recently found another thread on HN about it What do top engineers you know do that others don’t.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

As always for hacker news, there’s a feisty debate about what such character includes.

Here’s my take.

1. Tackle learning quickly

Whether it means getting up to speed on a new service that AWS has launched, building a new api for an application that has never been built before, or getting up to speed with a new platform. Learning is ongoing.

Top engineers can make this a seemless part of their daily routine. Getting going quickly with new concepts and technologies, means wading into the water at first, to gain the general lay of the land. Now you can talk intelligently about the features, limitations and challenges.

From there he or she can dive in quickly to the specific area required for the project, and move forward with that technology comfortably.

Related: How do I migrate my skills to the cloud?

2. Customer & product perspective

When building code, it’s easy to get mired in libraries, sorting algorithms, and API minutiae. And all of that is very important. But what are you building, and why are you building it?

Understanding your customer, what they do day-to-day is not always easy. It means using the product yourself, and also talking with sales teams regularly to hear what they are hearing.

Then pouring all this into your user stories. For top engineers it will inform their decisions, and help them communicate to product & project managers about what issues their encountering. Tradeoffs about features, coding, performance and technical debt can be better evaluated with more information.

Read: Is Fred Wilson honest about transparency?

3. Dig Deeper

Does your code run slowly? Have you tried to figure out why?

Is it related to:

o latency in production that doesn’t appear our your laptop
o untuned production database queries
o untuned connection pooling
o slow API calls
o weird kubernetes or orchestration issues
o web host issue with memory shortage
o web host issue with slow unoptimized code
o issues on the client side

Top engineers have seen applications slow down or fail in a myriad of ways. This allows them to imagine how a new application might be failing, and investigate those.

Related: What mistakes did you make when starting as a consultant?

4. Great communicators

In startups, your engineers need to communicate to many folks who don’t have an engineering background. Product & UI/UX folks probably are quite technical on their own. But what about sales teams who are dealing directly with customers? Or C-suite folks who watch the business bottom lines, but may not have the same low level technical understanding?

Great communicators can find the right metaphor to explain hurdles and holdups, technical debt, or the latest performance challenges. And explaining those in terms that resonate for others is incredibly valuable to the team and business velocity.

Related: Can Mailchimp fraudulently charge your credit card?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Categories
All Devops

When I found gold in my customer archives

via GIPHY

I’m good at keeping notes. I’ve blogged about Can progress reports & daily notes help engagements succeed. I would give that an emphatic YES!

From helping with communication, to sharing arcane details about blocking issues, struggles & hurdles, notes can illuminate things that a CTO or manager may not otherwise be aware of.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

I was digging through mine archives recently, and found a bunch of notes on how to think about Terraform. In particular, how do you think about infrastructure as code? How do you architect to make it all work together?

1. A dead end started me backtracking

You’re going to dig your heels in by getting your application working. To do that you’ll spinup a vpn, private public subnet, bastion boxes, ECS hosts to deploy containers to, and an application load balancer endpoint. Getting that all working wasn’t terrible. We even included a prometheus node, to give us some monitoring visibility. We even added our jenkins server into the mix. Do you see where this is going?

At a certain point we of course needed to destroy the whole setup, but didn’t want to destroy the CI pipeline. Duh! And what about monitoring? Lose all that data each time no way!

Read: Infrastructure provisioning – what is it and why is it important?

2. Organize around VPCs

After dragging yourself through that, you see a bit better. It’s like standing at 20,000 feet.

Your vpn is a logical collection of instances. A box that holds your application, provides security, and gets created and destroyed with it. You can even see in your Terraform code, a subnet requires a VPN id within which to create it. And an instance requires a subnet within which to create it. For security reasons the application instances will sit within PRIVATE subnets, and only bastion box & load balancers will sit in PUBLIC subnets.

TO my mind that means each environment DEV, STAGE, PROD all get their own vpn. This also allows you to control who can access stage & production, as they have their own bastion access points.

Read: How to hire a developer that doesn’t suck

3. Build a utility VPC

What you’ll also see from the above story is that you need a place to have business wide, non-application services sit inside. Welcome the UTILITY VPC!

This can contain prometheus, ELK or other log collection service, your jenkins or other CI pipeline, and any other services that don’t logically fit within the application VPC.

Related: Why generalists are better at scaling the web

4. A VPC should be ok to destroy and rebuild in another region – in one-click

When you use infrastructure code, you want to test, create & destroy often. That shouldn’t disrupt anything. That means all state data should sit outside of those instances. Logging data, send it to logstash or cloudwatch. Application state, keep that inside of an RDS instance. And you’ve tested those backups right?

Speaking of RDS, I encountered problems with Amazon’s own backup & restore. For my money, I had a lot of problems and ended up writing a custom db dump script. That may require a custom restore to, so buyer beware. Here’s my story though… I tried to build infrastructure as code with Terraform and Amazon and it didn’t go as i expected.

You also may encounter issues when you move across regions, such as elastic IPs and so forth. And you’ll need to check and verify the code which creates and destroy S3 buckets and domain name certs. These areas gave me some hiccups, but you can work through it with diligence!

Read: Is zero downtime even possible on RDS?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Categories
All Devops Scalability Startups

How crazy can Kubernetes get?

I was flipping through Reddit and found this hilarious post referencing a Scott Adams Dilbert strip on Kubernetes.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

What I found even funnier were the comments on the Reddit thread. Read on for fun!

1. Watch your memory

One engineer said he had a dockerized app running with 3GB of memory serving just 7 customers.

Not to be outdone another ops guy pipes in that he has one using 180Gb of memory serving just a few hundred customers.

Of course this is the internet, and along comes a guy who has an app using 1TB of memory, with only one user!

Optimization be damned!

Read: Infrastructure provisioning – what is it and why is it important?

2. Beware growing application complexity

As you dockerize your application, you can support multiple versions of software and packages. This can keep you flexible but also enable engineers to kick the can down the road. Technical debt is real!

What’s more each microservice *can* be on a different stack, using a different language and framework. But just because you *can* do something doesn’t mean you should.

Though docker & kubernetes will enable the above, keep in mind your team has to support it. Using some cool new language that hasn’t really achieved critical mass? Remember the engineer who championed that, and built your business crown jewels on top of that, will eventually leave. And when he or she does, you will be faced with the challenge of finding someone who knows the stuff!

Related: 6 Devops interview questions

3. What is a microserved monolith?

Well it’s not really a thing, except it sounds fun. And a bit absurd. If all those docker containers never get optimized, they probably have layer upon layer of useless stuff. Start with a smaller base image, dont include debug stuff, and extra layers. And cleanup after installing packages.

Here’s a more detailed howto optimizing Docker image sizes.

Read: Is zero downtime even possible on RDS?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Categories
All CTO/CIO Devops Software Development Startups

How to think like a senior engineer

via GIPHY

I just read this story
the Art of interrupting engineers. While much of what I read was pretty obvious from the engineer’s perspective, the product & project manager perspective was illustrative.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

There are definitely things that senior or more experienced engineers do differently. And those things can be learned.

Here are my thoughts…

1. Document all the things

By documenting all the little pieces you are working on you gain in a few ways. You communicate to management complexity they may not see. You buy yourself more time.

Finally you may even help yourself see implicit tasks more clearly. Whenever I hear myself or someone else utter the phrase “that should be easy”, I know I’m onto one of those mysterious tasks that seems simple but never is.

Be relentless. Break big tasks into smaller ones, and ticket each and every darn one!

Also: How can 1% of something equal nothing?

2. Communicate more and better

If you’re doing agile, chances are you’re probably joining a standup everyday.

Those are opportunities to share what is blocking.

o What tradeoffs are you struggling with?
o What technical debt is slowing you down?
o What new discoveries may require a rework of the timeline?

There is surely an art buried in communication. You want to be descriptive. You don’t want to come off complaining. You want to educate stakeholders. Beware of coming off as dismissive.

Related: Is maintenance sometimes a forgotten art?

3. Anticipate. Under promise & over deliver.

If you’ve gotten in the weeds with a particular API before, you’re likely to have a gut feeling about how new features and changes may go well or poorly. Or you may have dug through the comments. Maybe you didn’t find any!?

Or maybe the particular codebase sits on top of an interface or library you haven’t worked with before. So there will be a bit more of a learning curve. Whatever the case, try to promise less than you think you can really deliver on.

You can always finish extra tickets and over deliver later!

Read: What tools and technologies are devops engineers using today

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Categories
All Cloud Computing CTO/CIO Devops

Why maintenance sometimes a forgotten art?

via GIPHY

Just finished reading the excellent
Why do people neglect maintenance?.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

With a wide ranging discussion, from cultural myths to cognitive biases, there are a lot of reasons why your organization may not be giving maintenance the attention it deserves.

Here are some thoughts…

1. Weighing short & long term tradeoffs

When looking at maintenance problems, we sometimes must weigh easy to implement quick fixes, versus the better though weightier longer term fix.

Often the longer term fix requires more downtime, and so is a harder pill to swallow. So sometimes taking that medicine is put off till later. Weighing how much that could cost you is not easy. But that is often the reality of maintenance and ops teams.

Read: How to hire a developer that doesn’t suck

2. Keeping it sexy

One interesting point they make in the article is around the culture of innovation. Since building new products and changing the world is sexy, some how the day-to-day realities of managing and maintaining that after delivery can take a back seat.

Just as we prioritize creating new features, and building a changed product, we must always weigh the costs of such change. As I wrote in the four letter word dividing dev and ops, different team members have different mandates. And that’s important.

While developers are mandated with bringing new features to life, ops are mandated with keeping things running at 4am. Software updates, maintenance, outages are what the ops team is worried about.

Related: Does Amazon have a dirty little secret?

3. Status symbols – dev versus ops

There is some interesting discussion by Andy Jess & Lee about how these different job roles can be viewed as low or high status in an organization.

In my article why we need techops I mention some of this narrative. Once at a keynote, I heard a sales guy advocate a product that would obviate your needing to hire ops. Yep, more money to hire Devs!

On the flip side at ops and DBA conferences, I’ve heard over and over the story of “some idiot developer that took down our production systems…”

You get it on both sides. When will teams ever work together?

Read: The art of resistance – when you have to be the bad guy

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Categories
All Cloud Computing CTO/CIO Devops

Bye bye Kubernetes

via GIPHY

James Sanders at TechReplublic interviews Corey Quinn,
Is Kubernetes better for building a resume than a platform?

Corey is the snarky, observant, and talented author behind Last Week in AWS. If you haven’t subscribed you’re definitely missing out!

Join 35,000 others and follow Sean Hull on twitter @hullsean.

The observation about Kubernetes is not new. My friend & colleague over at Smash Company, Lawrence Krubner, wrote an exhaustive piece commenting on the flaws of docker, kubernetes and their eco-system.

It’s October 7th, 2019. I will revisit the question of Kubernetes demise again in six months. First thing in April 2020 we can see where my prediction stands!

1. Senior engineers are asking hard questions

The new generation of engineers, are more open to new solutions. They’re not bogged down by history, and legacy limitations. They’ll try new things, and embrace new technologies if they can solve a problem.

But senior engineers have been bitten. They’ve had the sorry job of untangling over engineered solutions. They’ve seen small firms live and die by wrong choices, and tech stack decisions. They’ve hit the wall on architectures, that were designed without enough perspective and foresight.

Read: What did Matt Ranney discover scaling Uber to 1000 microservices?

2. Trends come come and go

Does anyone out there remember Xen? Probably not. That’s because AWS squashed it like a little ant.

Kubernetes could go the same way. Not least because AWS has their own ECS.

Related: Can humility help you in your career?

3. Complexity is getting out of control

Kubernetes is sure great for doing blue-green deployments. And dockerizing all the things, helps developers and ops teams work together. After all just handing over the container asset, means ops can concentrate on deploying that. And kubernetes is great right up to this point.

But what about when kubernetes itself introduces application problems?

What about when kubernetes introduces performance and scalability problems?

And don’t even get me started about the crazy problems of running a database on Kubernetes.

Read: What happened when I offered advice outside my pay grade?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Categories
All CTO/CIO Devops

What tools & tech are devops engineers using today

via GIPHY

I just stumbled upon Graham King’s blog, and I’m liking his writing. He wrote an excellent piece a developer goes to a DevOps conference.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

I’ve been to plenty of Unix & operations type conferences over the years, so topics don’t surprise me. But hearing about a developer’s experience brings a new perspective and some great insights.

1. Tools change but mindset stays the same

Some talk about Devops as doing away with operations. Those job roles just aren’t necessary anymore. Well maybe for a small firm, or maybe shops that have pushed 2-pizza agile to the max. But handing the operations duties to developers has limitations. As I mentioned here (the difference between dev and ops is a four letter word…) these different job roles have different mandates.

It’s like an architect can design a building, and it can be a very beautiful house. But a super or building manager keeps it running over the years. He or she knows what to look for in cracked roofs, knows how to keep rodents & pests at bay, knows how to repair and maintain & stay ahead of the game.

In that analogy, the architect is the developer, while the super or building manager is the operations team. They’re two different mindsets, rarely shared in one person.

Read: What did Matt Ranney discover scaling Uber to 1000 microservices?

2. Being on-call is a b*tch

I could write volumes about being on-call. Getting woken up in the middle of the night, because someone pushed broken code is no fun. What’s more broken can have different meanings.

Broken can be something QA should catch, like a button doesn’t work or there’s an issue with some browser. It could also be that some new product feature doesn’t work properly.

But from the ops perspective, broken could also be some new feature doesn’t scale. It makes a million API calls, or makes a servless call that times out. These types of broken are much harder to test for.

This is also why traditionally operations and development were two different teams. Because from the vantage of the business, they had different mandates.

Ops was mandated with stability. So they don’t want change. Change breaks things, and wakes you up at 3am.

Devs are mandated with features changes, and product improvement. So they naturally bring change to the table.

And between the two we search for balance. I wrote a piece that hit on exactly these points the difference between dev and ops is a four letter word…

Related: Can humility help you in your career?

3. The kingmaker tools

Kubernetes – you’ve heard of it, you’re probably using it. Devs package their app as a docker container, and ops push that container through CI/CD pipeline, and finally orchestrate & deploy with kubernetes. Seems like the *only* way to do things these days, right?

But some argue Docker may not be right for everyone and certainly this stack brings a *lot* of complexity for small organizations.

Related: Is AWS too complex for small dev teams?

Terraform I’m a big fan of this technology. Once you’ve captured your entire stack in code, you can version it, check it into git, and manage it like any other asset. That’s great, but there are so many other benefits. You can easily deploy that same stack in another region, or tweak it to create dev, stage and production. Cool stuff!

Related: I tried to build infrastructure as with Terraform and AWS. It didn’t go as I expected

Ansible All those BASH scripts you have sitting around? Check them into version control before it’s too late! One great thing about Ansible is with slight tweaks and can run those bash scripts almost as-is.

And for ops who already have experience with managing things by hand, you can get up to speed with Ansible, in a few days. The learning curve isn’t as tough as Puppet or Chef, and brings many or most of the benefits.

Packer Here’s another cool tool. Chances are all those AMI’s that Amazon has pre-baked, need tweaks for your setup. Now you could do all that work post spinup with Ansible. And that’s fine. But it’ll be slower, and possibly prone to breaking if the base AMI changes.

Enter Packer, another great tool from the folks who brought us Terraform, Hashicorp. This tool allows you to write yaml files that then build AMI’s. You can then use your pipeline and other automation tools to automate those as well. Cool !

Read: What happened when I offered advice outside my pay grade?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters