I worked with a client some time back that was in a very unfortunate position indeed. Through various management shuffling, the COO was the new sheriff in town. Which is fine and normal.
Except in this case the entire operations was managed across the sea.
Join 35,000 others and follow Sean Hull on twitter @hullsean.
Now in many or most cases that would be fine. The new COO asks for the root credentials, works with devops to get new hires on board, and move forward. But these were not normal circumstances.
No, here the turf war began in earnest. Threats were even lobbed about what damage could be done.
1. Get a handle on all the systems
Upon starting the engagement, I attempted to inventory everything. What aws logins & IAM accounts were there? What servers were deployed? What networks, and security groups were setup like?
I followed many of the suggestions I outlined here: locking down cloud systems.
However we hit some hurdles, because I did not have my key deployed on production servers. I also could not update the deployed keys. So we were stuck in a holding pattern.
Read: Is AWS too complex?
2. Can you deploy this key in secret?
As we moved through the mess, and still could not gain control of systems, I received a strange request.
Can you deploy your key to production in secret?
Well of course what I’m thinking at this point is, I sure hope not. If a new engineer could bypass the devops team and get the keys changed on the servers, that would amount to a giant security hole. It was as it turned out not possible to do.
3. Git is not a good way sidestep human communication
What I realized is that we were trying to use git to deploy a key, which would have communicated our intentions to the devops team anyway. What’s more it would have communicated it in a strange way. Why is this happening, and what are *they* trying to do over there in New York?
Better to communicate the normal way, hey we hired this new engineer and we would like to give him access to these production servers. End of story.
4. How things got resolved
We attempted to get the key installed on systems, which did precipitate a conversation about, who is this guy Sean and what’s his role? All of that stalled for a while.
In the end a new CTO was hired, and the devops lead in Odessa reported to this new guy. This was a savvy move since the new CTO didn’t yet have a relationship good or bad with the loose cannon in Ukraine.
As we stepped forward carefully, we got a hold of credentials, and then planned an early morning cutover. While multiple engineers were poised at our keyboards, the devops lead in Ukraine was being fired. So we were locking him out a the same time.
5. No technology can sidestep hard conversations
It all went smoothly, and a lot was learned.
What I learned was that when strange requests surface, usually it means some bigger conversation needs to happen.
No technology is can sidestep those hard conversations.