Categories
All Cloud Computing Consulting CTO/CIO Security Startups

How can we secretly deploy this key to production?

via GIPHY

I worked with a client some time back that was in a very unfortunate position indeed. Through various management shuffling, the COO was the new sheriff in town. Which is fine and normal.

Except in this case the entire operations was managed across the sea.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

Now in many or most cases that would be fine. The new COO asks for the root credentials, works with devops to get new hires on board, and move forward. But these were not normal circumstances.

No, here the turf war began in earnest. Threats were even lobbed about what damage could be done.

1. Get a handle on all the systems

Upon starting the engagement, I attempted to inventory everything. What aws logins & IAM accounts were there? What servers were deployed? What networks, and security groups were setup like?

I followed many of the suggestions I outlined here: locking down cloud systems.

However we hit some hurdles, because I did not have my key deployed on production servers. I also could not update the deployed keys. So we were stuck in a holding pattern.

Read: Is AWS too complex?

2. Can you deploy this key in secret?

As we moved through the mess, and still could not gain control of systems, I received a strange request.

Can you deploy your key to production in secret?

Well of course what I’m thinking at this point is, I sure hope not. If a new engineer could bypass the devops team and get the keys changed on the servers, that would amount to a giant security hole. It was as it turned out not possible to do.

Related: How do I migrate my skills to the cloud?

3. Git is not a good way sidestep human communication

What I realized is that we were trying to use git to deploy a key, which would have communicated our intentions to the devops team anyway. What’s more it would have communicated it in a strange way. Why is this happening, and what are *they* trying to do over there in New York?

Better to communicate the normal way, hey we hired this new engineer and we would like to give him access to these production servers. End of story.

Related: What mistakes did you make when starting as a consultant?

4. How things got resolved

We attempted to get the key installed on systems, which did precipitate a conversation about, who is this guy Sean and what’s his role? All of that stalled for a while.

In the end a new CTO was hired, and the devops lead in Odessa reported to this new guy. This was a savvy move since the new CTO didn’t yet have a relationship good or bad with the loose cannon in Ukraine.

As we stepped forward carefully, we got a hold of credentials, and then planned an early morning cutover. While multiple engineers were poised at our keyboards, the devops lead in Ukraine was being fired. So we were locking him out a the same time.

Related: How to hire a developer that doesn’t suck?

5. No technology can sidestep hard conversations

It all went smoothly, and a lot was learned.

What I learned was that when strange requests surface, usually it means some bigger conversation needs to happen.

No technology is can sidestep those hard conversations.

Related: 30 questions to ask a serverless fanboy

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Categories
All Cloud Computing CTO/CIO Devops

What does DEV OPS mean?

via GIPHY

I was recently interviewed by Victor Farcic. He asked me a lot of interesting questions.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

One really struck me that I thought was important, about the whole devops movement.

How do you define devops

Viktor Farcic: Moving on to a more general subject, how would you define DevOps? I’ve gotten a different answer from every single person I’ve asked.

Sean Hull: I have a lot of opinions about it actually. I wrote an article on my blog a few years ago called The Four-Letter Word Dividing Dev and Ops, with the implication being that the four-letter word might be a swear word, akin to the development team swearing at the operations team, and the operations team swearing at the development team. But the four-letter word I was referring to was “risk.”

Related: Why generalists are better at scaling the web

To summarize my article, in my view the development and the operations teams of old were separate silos in business, and they had very different mandates. Developers are tasked with writing code to build a product and to answer the needs of the customers, while directly building change into and facilitating a more sophisticated product. So, their thinking from day to day is about change and answering the requirements of the products team.
On the other hand, the operations team’s mandate is stability. It’s, “I don’t want these systems going down at 2:00 a.m.” So, over the long term, the operations teams are thinking about being as conservative as possible and having fewer moving parts, less code, and less new technologies. The simpler your stack is, the more reliable it is and the more robust and less likely it is to fail. I think the traditional reason why developers and operations teams were separated into silos was because of those two very different mandates.

Also: Walking the delicate balance of transparency

They’re two different ways of prioritizing your work and your priorities when you think about the business and the technology. However, the downside was that those teams didn’t really communicate very well, and they were often at each other’s throats, pushing each other in opposite directions. But to answer your question, “what is DevOps?” I think of DevOps as a cultural movement that has made efforts to allow those teams to communicate better, and that’s a really good thing.

Read: One time in 2013 I had to take the fall

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters