What mistakes did you make when starting as a consultant ?

via GIPHY

I was recently reading a Hacker News thread on mistakes made in consulting. While most of the discussion I didn’t agree with folks *at all*, it did get me thinking about my own lessons.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

Since these threads in HN often end up in debate, I decided to blog my answers instead. Here they are!

1. Not getting a deposit

I wrote before about When clients don’t pay. It doesn’t happen often, but it does on occasion.

Most of the time, you have a solid relationship because you’re referred by someone who you have also worked with. And a bad actor will get a bad rep among others in the network. This also by the way keeps consultants honest too, as your reputation is at stake with every new customer.

But there have been a few. And there are other good reasons Why I ask clients for a deposit.

It can be a nominal amount of a few hundred dollars. But it gets you into the finance system, provides a small hurdle that the client must also jump over, and generally provides good will to both parties. It says “we’re serious”.

And that’s important!

Read: What I learned from 10 years of blogging

2. Giving away free advice

In the first few years of consulting, I worked for so many great companies, that I figured they were all great. Then along comes one shady shop, for whom I was called in to help with scaling an application.

For a couple of hours we met face to face to discuss their challenges. What I found was an application that had grown beyond it’s original ambitions. I suggested they evaluate the product, provide new service levels to their customers, at different price points. And then for the higher paying customers, build out new hardware just for them.

It was a great solution, if I do say so myself. The customer thought so too. They went and implemented it themselves, without hiring me! I think they even offered to pay me a couple of hours for the meeting.

Suffice it to say I was pissed. There wasn’t much I could do because we didn’t have any agreement in place at that stage. I had no idea people would do stuff like this.

But I learned the hard way. Sad to say it’s the few bad actors that make us all play more carefully in business. Buyer beware!

Related: 6 Devops interview questions

3. Billing hourly

Billing hourly is how many start out. Some think of it as an industry standard. Lawyers do it, hey why not?

But hourly billing can be very confusing to customers. If you work 10 hours to solve a problem one week and 60 the next, they will find these invoices confusing and frustrating. What’s more on the 10 hour end of the spectrum you’ll be answering questions why the work was so “easy” and on the 60 hour end, what did you do wrong?

Customers often just want a problem solved. They want you and they want your availability. And the value of that may vary quite a lot. If you can figure out a way to bill weekly or monthly such that the client is happy with the value you’re providing, this will simplify your life immeasurably. Customers will be happier, and so will you.

I also recommend keeping daily notes and providing progress reports to your client.

Read: High availability what is it and why is it important?

4. Don’t step on toes

As a consultant, you will often be working with a team of fulltime folks. Not always, but sometimes there can be a tiny bit of resentment. Maybe because you’re outside opinion is threatening, or maybe for a million other reasons.

So I recommend treading carefully. Try to reassure your teamates that you aren’t trying to outshine them. Inevitably you’ll be in meetings with folks smarter than you. Certainly they will know more as they have boots on the ground, but sometimes, they are just plain & simple smarter than you. 🙂

Feel things out, and don’t step on toes. Some need to be right. Let them be that.

Read: High availability what is it and why is it important?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Mailchimp fraudulently charged my credit card for spambot activity… Really!

via GIPHY

Wait seriously, you ask? Isn’t Mailchimp in the business of identifying, and protecting us from spam? Uh, yes indeed they are.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

I’m still in disbelief myself. And while I got the problem cleared up in the end, I really have to share the story.

1. The precipitating event – a charge to my credit card

I love when I pay for a service, and their method of communicating with me is to charge my credit card. Of course I pay attention to when someone is taking my money, and I perk up.

At first I thought they were raising the prices again. They’ve done that recently, so I thought it was odd.

But sure enough I got an email with the following message:

Your account has been adjusted to another billing tier. Old plan $22.49, new plan $31.49.

Wait huh? I add about 2-3 new subscribers per day. How could this be?

Read: What I learned from 10 years of blogging

2. After digging I found spam emails

After looking at my list, I found that I had added 600 new subscribers last week in three days. How is that even possible? I wasn’t mentioned on BBC. That must be spam, I thought.

So I emailed support. They sent me all sorts of links, but didn’t seem to understand the issue. So I emailed back again and they said they were working on it.

Related: 6 Devops interview questions

3. Mailchimp communication – a warning

This *warning* is problematic. For one thing is it buried through various menus and pages. Only because I was looking for spam did I find it.

Plus Mailchimp doesn’t take responsibility.

In fact they kind of imply that I’m a bad actor here. Seriously? Is that how you communicate with your customers?


Warning

We noticed a 0.55% abuse rate on your campaign “Welcome Message”. This is above industry standards, so we strongly recommend you review your collection process, audience management, and sending frequency.

Internet service providers set strict limits on unsubscribe rates, undeliverable mail, and abuse complaints. Mailchimp is required to observe these limits. If your emails continue to generate high rates of unsubscribes, bounces, or abuse complaints, we may need to review or restrict your account. Please take the opportunity to address this now.

Read: High availability what is it and why is it important?

4. Can’t get someone on the phone

I did some google searching because I could not find the phone number. Turns out you *CANNOT* call Mailchimp. A lot of these services internet companies are going this route. Sure it saves them lots of money, but the customer service goes straight to the trash.

So I begrudgingly jump on a chat session. It took

Read: Service Monitoring – what is it and why is it important?

5. The chat transcript in full

Sean Hull
I've been hacked.

THEN MAILCHIMP CHARGED ME!

This is strange.

Does mailchimp protect me?

Mailchimp Support
We apologize for keeping you waiting and appreciate your patience. Our operators are busy at the moment. One of our agents will be with you as soon as possible.
Sean Hull
Thank you ... waiting patiently.
Mailchimp Support
We didn't forget about you. We apologize for keeping you waiting and appreciate your patience. Our agents are busy at the moment. One of our agents will be with you as soon as possible.
Sean Hull
thank you robot person...
how is the progress?
7 more! :)
4 more!
we are almost there!
Neo joined the chat
Sean Hull
hi neo
Neo
Hey there Sean, thanks for reaching out to Mailchimp support. Give me just a moment while I pull up your account.
Sean Hull
ok thank you
Neo
Alright Sean, what is the exact issue you are facing within your account?
Sean Hull
mailchimp charged me for fake subscribers.
if you look at my email list, you'll see it typically grows by 2 or 3 maximum per day
recently a hacker dumped 200+ per day into my list.
Mailchimp didn't monitor things, and then CHARGED ME to my credit card.
Does mailchimp protect me?
hi Neo, are you still there?
Neo
I'm still with you Sean. One of the main ways that Mailchimp prevents spam signups is through the use of ReCAPTCHA. This is a setting you can add to your embedded form from the "Audience name and defaults" page, which you can read more about here: https://mailchimp.com/help/about-fake-signups/#How_we_prevent_it
Sean Hull
ok. that is helpful. for the time being i enabled double opt-in.
but I also see that mailchimp has a WARNING.
about recent activity on my account, and possibly shutting it down. do you see that?
Neo
Are you referring to the "Account issue" that is referenced in the bar at the top of the screen?
Sean Hull
Warning

We noticed a 0.55% abuse rate on your campaign "Welcome Message". This is above industry standards, so we strongly recommend you review your collection process, audience management, and sending frequency.

Internet service providers set strict limits on unsubscribe rates, undeliverable mail, and abuse complaints. Mailchimp is required to observe these limits. If your emails continue to generate high rates of unsubscribes, bounces, or abuse complaints, we may need to review or restrict your account. Please take the opportunity to address this now.
This is what it says...
so to explain more...
first off this is fraudulent activity.
so I'm concerned that mailchimp would just charge my account, without warning of some problem.
and further, it seems that mailchimp *monitors* to INCREASE BILLING and monitors to DISABLE YOUR ACCOUNT, but they don't monitor to protect their customers.
Is that correct? Because if there is some type of monitoring I can enable, that would certainly be very helpful.
Also is it possible to DISABLE AUTO PAYMENT on my credit card?
Neo
For the sake of clarity, let's tackle your questions one at a time. I'm getting some more information for you at the moment and will follow up with you shortly. Thank you for your patience.
Sean Hull
thank you Neo.
you're awesome !
do i need *both* double-opt-in and RECAPTCHA? or is RECAPTCHA enough?
Neo
It certainly couldn't hurt to use both. Using double op-t in will help ensure higher engagement rates overall and less likely to present warnings such as what you've seen. Here is some more information on double opt-in: About Double Opt-In: https://eepurl.com/dyij4v
Sean Hull
i mean the warning is a mistake from mailchimp isn't it?
because these automated systems just sent that because of the hacking.
i feel mailchimp should be protecting me, so I'm confused by that.
I am a paying subscriber of the service. and the price has gone up in recent months. so i think we can agree there should be protection from spambots.
is it possible to disable AUTOPAY on my credit card? BC i don't want to get further fraudulent charges from mailchimp, because of a spam problem.
does that make sense?

Neo
The method of protecting your account would be through tools provided to avoid spam signups, which would be double opt-in and ReCAPTCHA. Additionally, our teams, such as Compliance and Billing, would be happy to look into your account with you to help resolve any issues you may be experiencing.

I would also like to let you know that I understand the situation you are facing is frustrating, so I will be submitting feedback on your behalf internally.
Sean Hull
anyway, could you scan my account for further spam signups? I think around july 19th there was a bump in signups of 80 people. I tried the segment method but couldn't find which day they were from.
Neo
Sure thing, allow me a moment to take a look.
Sean Hull
thank you Neo, i do appreciate that.
i mean at the end of the day I'm not an email spam expert, so that's why I pay for a service like mailchimp. to avoid problems and run a really clean list.
so for example that RECAPTCHA thing should be ON BY DEFAULT. probably that would have avoided all this to begin with.
also mailchimp should SCAN FOR SPAM FIRST. not charge customers first, then realize there is spam and charge back. Because that is a fraudulent charge. which i find super frustrating. I do realize these are all automated systems. But mailchimp should be more sophisticated to protect good customers like me.
Neo
I can certainly see how that is frustrating and would be useful to users such as yourself, so I would highly recommend leaving feedback on the matter via the "Feedback" tab at the right side of your screen.
Additionally, I’ll certainly be routing your concerns and feedback to our internal teams.
Sean Hull
Is that part of "chat comments"?
i left a good review of your help :)
okay thx again Neo.
I'll see if I can find that feedback tab
have a nice night :)

Neo
You can find it within your Mailchimp account on any page on the right side of the screen. Please don't hesitate to reach back out if you have any further issues, and have a great rest of your night.
Sean Hull
thx
laters

The remedy as you can see above was the enable recaptcha and also double-opt-in. That was fairly easy once I knew where to look.

From there I created a "segment" which is a collection of emails. And I selected the date range for the three days where I got spambot hit. And then clicked UNSUBSCRIBE for all those.

Why didn't Mailchimp do this for me? Read more to find out why I think they don't automatically fix this.

Read: How do I migrate my skills to the cloud

6. What Mailchimp did wrong

o They monitored the account to increase a SALE.
o They monitored my account to warn me about shutdown.
o They did not warn me about RECAPTCHA.
o They did not alert me when I got aberrant signups. When you see a 100x increase in signups, it doesn't take a rocket scientist to see that's a hacker of some kind.

o Then Mailchimp fraudulently charged my account!

Read: How to hire a developer that doesn't suck?

7. The sinister side

This charge could be innocent. A left over part of an automation system that hasn't evolved with the spambots. But I wonder.

From the help forums, they are clearly AWARE of the problem.

And the company *could* go and FORCE enable RECAPTHA for all lists. They could then email customers about the change, and for those who this poses and problem, the user could then go and manually DISABLE it.

They haven't done that. And certainly RECAPTCHA was not enabled by default.

Please don't call me paranoid when I say that there is a *huge* revenue stream to be had for users who fail to notice, this, get charged, and don't even know their own negligence. Think of all that revenue. My list has under 1500 subscribers, but others have 5000 or 10,000. Imagine how easily a *higher billing tier* could get overlooked.

Yes folks, it's dark.

And I'm not happy with Mailchimp now.

I'm happy to pay for a service, when it is done well. But I'm not a fan of these dirty tricks.

Read: 5 things toxic to Scalability

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don't work with recruiters

Did Disney have to fail?

via GIPHY

Well it was a big day for Disney a couple of weeks ago when they launched the much anticipated Disney+ streaming service.

And of course as widely reported, they had a really bad outage.

We’ve heard the old refrain before. We never saw traffic like this before. We were absolutely buried.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

Indeed, Disney, in their own microcosm never saw traffic like that before. But does cloud computing know how to solve this problem? Has the software, techops and devops profession solved these problems?

Answer: Yes. Just look at Facebook, Google, Netflix or a million other high traffic sites.

Interestingly, I got a letter from a recruiter over at Disney just the other day!

Hi Sean,

I wanted to reach out again to see if you’ve had an opportunity to think about exploring a future role with Disney Streaming Services?

The company is in the midst of a massive engineering expansion, as Disney+ launched today in their US, Canadian and Dutch markets. While phased with natural issues, given the pure scope of their launch, paired with unforeseen customer engagement, the team is continuing to actively to build out their engineering team in order to ensure seamless launches in Western Europe and the Asian Pacific countries in late 2019/ early 2020.

While you may not actively be on the market, I would be interested in having a preliminary chat to see if joining and impacting one of the largest scale technical projects the international video streaming industry has ever seen, could align with future career goals?

Looking forward to your response!

**Name redacted** 

Related: Why generalists are better at scaling the web

1. Management maturity

When I was reading the article above I stopped at this line:

“This is a new ballgame for Disney”

And that tells the whole story doesn’t it? Just because the industry has solved a problem, just because there are technologists out there who know how to do this, doesn’t mean they work at Disney! And further doesn’t mean Disney’s management is ready for their new reality.

But they learn quickly!

Read: Infrastructure provisioning – what is it and why is it important?

2. Streaming is a solved

The challenge of streaming content on the internet is not new. The pipes are there, the cloud can scale seemlessly. But yes there are a lot of moving parts. That’s what testing is for. And for a launch like this, one could easily launch a million test clients, using aws regions and zones around the world. And point them all at your new streaming service. Don’t want to spend the money? Scale down your stack, and send a proportional amount of traffic.

If you’re not seeing the autoscaling happen quickly enough, spinup *lots* of spare compute before launch time. That’s another option. You can always scale back after the flash hits.

And yes flash sales, daily deals or deal-a-day sites are another example. ideeli and the first unicorn Gilt Groupe

Sites like these deal with an explosion of traffic in a short period each day. Typically 90% of their traffic occurs in half to one hour of the day. So it’s a real herd that pummels the site.

Related: 6 Devops interview questions

3. Have doubts? Ask experts

To my mind, if a problem is solved, there’s no excuse to fail. But then it happens again at Airbnb and again at Dropbox and again many others.

Yes we can monitor. Yes we can test. Yes we can automate. Yes we can react. But still systems fail.

As a small pitch, I’ve helped companies like Hollywood Reporter (100m uniques per month), AppNexus, ideeli and SoulCycle scale their systems for hypergrowth. It’s not easy but it can be done!

Read: Is zero downtime even possible on RDS?

4. Good problem to have?

Oscar Wilde said…

The only thing worse than being talked about is not being talked about.

And if today’s political climate is any indication, their is some prescient irony buried there.

So even though Disney+ and many other big names have failed…

Perhaps it’s a good problem to have?

Read: How to hire a developer that doesn’t suck

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

 

When I found gold in my customer archives

via GIPHY

I’m good at keeping notes. I’ve blogged about Can progress reports & daily notes help engagements succeed. I would give that an emphatic YES!

From helping with communication, to sharing arcane details about blocking issues, struggles & hurdles, notes can illuminate things that a CTO or manager may not otherwise be aware of.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

I was digging through mine archives recently, and found a bunch of notes on how to think about Terraform. In particular, how do you think about infrastructure as code? How do you architect to make it all work together?

1. A dead end started me backtracking

You’re going to dig your heels in by getting your application working. To do that you’ll spinup a vpn, private public subnet, bastion boxes, ECS hosts to deploy containers to, and an application load balancer endpoint. Getting that all working wasn’t terrible. We even included a prometheus node, to give us some monitoring visibility. We even added our jenkins server into the mix. Do you see where this is going?

At a certain point we of course needed to destroy the whole setup, but didn’t want to destroy the CI pipeline. Duh! And what about monitoring? Lose all that data each time no way!

Read: Infrastructure provisioning – what is it and why is it important?

2. Organize around VPCs

After dragging yourself through that, you see a bit better. It’s like standing at 20,000 feet.

Your vpn is a logical collection of instances. A box that holds your application, provides security, and gets created and destroyed with it. You can even see in your Terraform code, a subnet requires a VPN id within which to create it. And an instance requires a subnet within which to create it. For security reasons the application instances will sit within PRIVATE subnets, and only bastion box & load balancers will sit in PUBLIC subnets.

TO my mind that means each environment DEV, STAGE, PROD all get their own vpn. This also allows you to control who can access stage & production, as they have their own bastion access points.

Read: How to hire a developer that doesn’t suck

3. Build a utility VPC

What you’ll also see from the above story is that you need a place to have business wide, non-application services sit inside. Welcome the UTILITY VPC!

This can contain prometheus, ELK or other log collection service, your jenkins or other CI pipeline, and any other services that don’t logically fit within the application VPC.

Related: Why generalists are better at scaling the web

4. A VPC should be ok to destroy and rebuild in another region – in one-click

When you use infrastructure code, you want to test, create & destroy often. That shouldn’t disrupt anything. That means all state data should sit outside of those instances. Logging data, send it to logstash or cloudwatch. Application state, keep that inside of an RDS instance. And you’ve tested those backups right?

Speaking of RDS, I encountered problems with Amazon’s own backup & restore. For my money, I had a lot of problems and ended up writing a custom db dump script. That may require a custom restore to, so buyer beware. Here’s my story though… I tried to build infrastructure as code with Terraform and Amazon and it didn’t go as i expected.

You also may encounter issues when you move across regions, such as elastic IPs and so forth. And you’ll need to check and verify the code which creates and destroy S3 buckets and domain name certs. These areas gave me some hiccups, but you can work through it with diligence!

Read: Is zero downtime even possible on RDS?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

How crazy can Kubernetes get?

I was flipping through Reddit and found this hilarious post referencing a Scott Adams Dilbert strip on Kubernetes.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

What I found even funnier were the comments on the Reddit thread. Read on for fun!

1. Watch your memory

One engineer said he had a dockerized app running with 3GB of memory serving just 7 customers.

Not to be outdone another ops guy pipes in that he has one using 180Gb of memory serving just a few hundred customers.

Of course this is the internet, and along comes a guy who has an app using 1TB of memory, with only one user!

Optimization be damned!

Read: Infrastructure provisioning – what is it and why is it important?

2. Beware growing application complexity

As you dockerize your application, you can support multiple versions of software and packages. This can keep you flexible but also enable engineers to kick the can down the road. Technical debt is real!

What’s more each microservice *can* be on a different stack, using a different language and framework. But just because you *can* do something doesn’t mean you should.

Though docker & kubernetes will enable the above, keep in mind your team has to support it. Using some cool new language that hasn’t really achieved critical mass? Remember the engineer who championed that, and built your business crown jewels on top of that, will eventually leave. And when he or she does, you will be faced with the challenge of finding someone who knows the stuff!

Related: 6 Devops interview questions

3. What is a microserved monolith?

Well it’s not really a thing, except it sounds fun. And a bit absurd. If all those docker containers never get optimized, they probably have layer upon layer of useless stuff. Start with a smaller base image, dont include debug stuff, and extra layers. And cleanup after installing packages.

Here’s a more detailed howto optimizing Docker image sizes.

Read: Is zero downtime even possible on RDS?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

How to think like a senior engineer

via GIPHY

I just read this story
the Art of interrupting engineers. While much of what I read was pretty obvious from the engineer’s perspective, the product & project manager perspective was illustrative.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

There are definitely things that senior or more experienced engineers do differently. And those things can be learned.

Here are my thoughts…

1. Document all the things

By documenting all the little pieces you are working on you gain in a few ways. You communicate to management complexity they may not see. You buy yourself more time.

Finally you may even help yourself see implicit tasks more clearly. Whenever I hear myself or someone else utter the phrase “that should be easy”, I know I’m onto one of those mysterious tasks that seems simple but never is.

Be relentless. Break big tasks into smaller ones, and ticket each and every darn one!

Also: How can 1% of something equal nothing?

2. Communicate more and better

If you’re doing agile, chances are you’re probably joining a standup everyday.

Those are opportunities to share what is blocking.

o What tradeoffs are you struggling with?
o What technical debt is slowing you down?
o What new discoveries may require a rework of the timeline?

There is surely an art buried in communication. You want to be descriptive. You don’t want to come off complaining. You want to educate stakeholders. Beware of coming off as dismissive.

Related: Is maintenance sometimes a forgotten art?

3. Anticipate. Under promise & over deliver.

If you’ve gotten in the weeds with a particular API before, you’re likely to have a gut feeling about how new features and changes may go well or poorly. Or you may have dug through the comments. Maybe you didn’t find any!?

Or maybe the particular codebase sits on top of an interface or library you haven’t worked with before. So there will be a bit more of a learning curve. Whatever the case, try to promise less than you think you can really deliver on.

You can always finish extra tickets and over deliver later!

Read: What tools and technologies are devops engineers using today

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

How can 1% of something equal nothing?

via GIPHY

I just read this painful story
My company sold for $100 million and I got zilch. How can that be?.

I felt apalled. I was a little sick to my stomach actually.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

Is it possible to work for years, putting blood sweat & tears into a firm, and end up with nothing?

I’ve been working with startups since 1996. While I wish this was an aberration, I’ve seen it happen a half dozen times over the years.

Here are my thoughts…

1. Have a lawyer review legalese

If your intuition tells you this additional compensation is worth something, then use logic to prove it.

The first step is to have a lawyer review the fine print. They know things you don’t. Since this isn’t your domain of expertise, you’ll want someone who can see around the corners, to give you fair advice.

Related: Is maintenance sometimes a forgotten art?

2. Have a financial expert review it

Ask a financial expert to review your compensation package. Think of it like buying a house, you’re commiting to something big, with a slow long term return.

What are the chances statistically that the company will succeed? What are the chances it may fail? If it does get bought, what are the chances the fine print will eat up your portion? When in doubt be conservative with your guesses.

Also: Is Kubernetes going bye bye?

3. Use your math skills

Compare these numbers to other financial investments, such as real estate, or the S&P 500. Use your math skills to choose the best place for your money.

Be relentlessly logical. It’s very easy to get emotional or be irrational when you are dealing with money. But remember it is just math in the end. Evaluate. Test. And execute rationally.

Read: What tools and technologies are devops engineers using today

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Why maintenance sometimes a forgotten art?

via GIPHY

Just finished reading the excellent
Why do people neglect maintenance?.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

With a wide ranging discussion, from cultural myths to cognitive biases, there are a lot of reasons why your organization may not be giving maintenance the attention it deserves.

Here are some thoughts…

1. Weighing short & long term tradeoffs

When looking at maintenance problems, we sometimes must weigh easy to implement quick fixes, versus the better though weightier longer term fix.

Often the longer term fix requires more downtime, and so is a harder pill to swallow. So sometimes taking that medicine is put off till later. Weighing how much that could cost you is not easy. But that is often the reality of maintenance and ops teams.

Read: How to hire a developer that doesn’t suck

2. Keeping it sexy

One interesting point they make in the article is around the culture of innovation. Since building new products and changing the world is sexy, some how the day-to-day realities of managing and maintaining that after delivery can take a back seat.

Just as we prioritize creating new features, and building a changed product, we must always weigh the costs of such change. As I wrote in the four letter word dividing dev and ops, different team members have different mandates. And that’s important.

While developers are mandated with bringing new features to life, ops are mandated with keeping things running at 4am. Software updates, maintenance, outages are what the ops team is worried about.

Related: Does Amazon have a dirty little secret?

3. Status symbols – dev versus ops

There is some interesting discussion by Andy Jess & Lee about how these different job roles can be viewed as low or high status in an organization.

In my article why we need techops I mention some of this narrative. Once at a keynote, I heard a sales guy advocate a product that would obviate your needing to hire ops. Yep, more money to hire Devs!

On the flip side at ops and DBA conferences, I’ve heard over and over the story of “some idiot developer that took down our production systems…”

You get it on both sides. When will teams ever work together?

Read: The art of resistance – when you have to be the bad guy

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Bye bye Kubernetes

via GIPHY

James Sanders at TechReplublic interviews Corey Quinn,
Is Kubernetes better for building a resume than a platform?

Corey is the snarky, observant, and talented author behind Last Week in AWS. If you haven’t subscribed you’re definitely missing out!

Join 35,000 others and follow Sean Hull on twitter @hullsean.

The observation about Kubernetes is not new. My friend & colleague over at Smash Company, Lawrence Krubner, wrote an exhaustive piece commenting on the flaws of docker, kubernetes and their eco-system.

It’s October 7th, 2019. I will revisit the question of Kubernetes demise again in six months. First thing in April 2020 we can see where my prediction stands!

1. Senior engineers are asking hard questions

The new generation of engineers, are more open to new solutions. They’re not bogged down by history, and legacy limitations. They’ll try new things, and embrace new technologies if they can solve a problem.

But senior engineers have been bitten. They’ve had the sorry job of untangling over engineered solutions. They’ve seen small firms live and die by wrong choices, and tech stack decisions. They’ve hit the wall on architectures, that were designed without enough perspective and foresight.

Read: What did Matt Ranney discover scaling Uber to 1000 microservices?

2. Trends come come and go

Does anyone out there remember Xen? Probably not. That’s because AWS squashed it like a little ant.

Kubernetes could go the same way. Not least because AWS has their own ECS.

Related: Can humility help you in your career?

3. Complexity is getting out of control

Kubernetes is sure great for doing blue-green deployments. And dockerizing all the things, helps developers and ops teams work together. After all just handing over the container asset, means ops can concentrate on deploying that. And kubernetes is great right up to this point.

But what about when kubernetes itself introduces application problems?

What about when kubernetes introduces performance and scalability problems?

And don’t even get me started about the crazy problems of running a database on Kubernetes.

Read: What happened when I offered advice outside my pay grade?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

What tools & tech are devops engineers using today

via GIPHY

I just stumbled upon Graham King’s blog, and I’m liking his writing. He wrote an excellent piece a developer goes to a DevOps conference.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

I’ve been to plenty of Unix & operations type conferences over the years, so topics don’t surprise me. But hearing about a developer’s experience brings a new perspective and some great insights.

1. Tools change but mindset stays the same

Some talk about Devops as doing away with operations. Those job roles just aren’t necessary anymore. Well maybe for a small firm, or maybe shops that have pushed 2-pizza agile to the max. But handing the operations duties to developers has limitations. As I mentioned here (the difference between dev and ops is a four letter word…) these different job roles have different mandates.

It’s like an architect can design a building, and it can be a very beautiful house. But a super or building manager keeps it running over the years. He or she knows what to look for in cracked roofs, knows how to keep rodents & pests at bay, knows how to repair and maintain & stay ahead of the game.

In that analogy, the architect is the developer, while the super or building manager is the operations team. They’re two different mindsets, rarely shared in one person.

Read: What did Matt Ranney discover scaling Uber to 1000 microservices?

2. Being on-call is a b*tch

I could write volumes about being on-call. Getting woken up in the middle of the night, because someone pushed broken code is no fun. What’s more broken can have different meanings.

Broken can be something QA should catch, like a button doesn’t work or there’s an issue with some browser. It could also be that some new product feature doesn’t work properly.

But from the ops perspective, broken could also be some new feature doesn’t scale. It makes a million API calls, or makes a servless call that times out. These types of broken are much harder to test for.

This is also why traditionally operations and development were two different teams. Because from the vantage of the business, they had different mandates.

Ops was mandated with stability. So they don’t want change. Change breaks things, and wakes you up at 3am.

Devs are mandated with features changes, and product improvement. So they naturally bring change to the table.

And between the two we search for balance. I wrote a piece that hit on exactly these points the difference between dev and ops is a four letter word…

Related: Can humility help you in your career?

3. The kingmaker tools

Kubernetes – you’ve heard of it, you’re probably using it. Devs package their app as a docker container, and ops push that container through CI/CD pipeline, and finally orchestrate & deploy with kubernetes. Seems like the *only* way to do things these days, right?

But some argue Docker may not be right for everyone and certainly this stack brings a *lot* of complexity for small organizations.

Related: Is AWS too complex for small dev teams?

Terraform I’m a big fan of this technology. Once you’ve captured your entire stack in code, you can version it, check it into git, and manage it like any other asset. That’s great, but there are so many other benefits. You can easily deploy that same stack in another region, or tweak it to create dev, stage and production. Cool stuff!

Related: I tried to build infrastructure as with Terraform and AWS. It didn’t go as I expected

Ansible All those BASH scripts you have sitting around? Check them into version control before it’s too late! One great thing about Ansible is with slight tweaks and can run those bash scripts almost as-is.

And for ops who already have experience with managing things by hand, you can get up to speed with Ansible, in a few days. The learning curve isn’t as tough as Puppet or Chef, and brings many or most of the benefits.

Packer Here’s another cool tool. Chances are all those AMI’s that Amazon has pre-baked, need tweaks for your setup. Now you could do all that work post spinup with Ansible. And that’s fine. But it’ll be slower, and possibly prone to breaking if the base AMI changes.

Enter Packer, another great tool from the folks who brought us Terraform, Hashicorp. This tool allows you to write yaml files that then build AMI’s. You can then use your pipeline and other automation tools to automate those as well. Cool !

Read: What happened when I offered advice outside my pay grade?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters