Categories
All Cloud Computing Consulting CTO/CIO Security Startups

How can we secretly deploy this key to production?

via GIPHY

I worked with a client some time back that was in a very unfortunate position indeed. Through various management shuffling, the COO was the new sheriff in town. Which is fine and normal.

Except in this case the entire operations was managed across the sea.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

Now in many or most cases that would be fine. The new COO asks for the root credentials, works with devops to get new hires on board, and move forward. But these were not normal circumstances.

No, here the turf war began in earnest. Threats were even lobbed about what damage could be done.

1. Get a handle on all the systems

Upon starting the engagement, I attempted to inventory everything. What aws logins & IAM accounts were there? What servers were deployed? What networks, and security groups were setup like?

I followed many of the suggestions I outlined here: locking down cloud systems.

However we hit some hurdles, because I did not have my key deployed on production servers. I also could not update the deployed keys. So we were stuck in a holding pattern.

Read: Is AWS too complex?

2. Can you deploy this key in secret?

As we moved through the mess, and still could not gain control of systems, I received a strange request.

Can you deploy your key to production in secret?

Well of course what I’m thinking at this point is, I sure hope not. If a new engineer could bypass the devops team and get the keys changed on the servers, that would amount to a giant security hole. It was as it turned out not possible to do.

Related: How do I migrate my skills to the cloud?

3. Git is not a good way sidestep human communication

What I realized is that we were trying to use git to deploy a key, which would have communicated our intentions to the devops team anyway. What’s more it would have communicated it in a strange way. Why is this happening, and what are *they* trying to do over there in New York?

Better to communicate the normal way, hey we hired this new engineer and we would like to give him access to these production servers. End of story.

Related: What mistakes did you make when starting as a consultant?

4. How things got resolved

We attempted to get the key installed on systems, which did precipitate a conversation about, who is this guy Sean and what’s his role? All of that stalled for a while.

In the end a new CTO was hired, and the devops lead in Odessa reported to this new guy. This was a savvy move since the new CTO didn’t yet have a relationship good or bad with the loose cannon in Ukraine.

As we stepped forward carefully, we got a hold of credentials, and then planned an early morning cutover. While multiple engineers were poised at our keyboards, the devops lead in Ukraine was being fired. So we were locking him out a the same time.

Related: How to hire a developer that doesn’t suck?

5. No technology can sidestep hard conversations

It all went smoothly, and a lot was learned.

What I learned was that when strange requests surface, usually it means some bigger conversation needs to happen.

No technology is can sidestep those hard conversations.

Related: 30 questions to ask a serverless fanboy

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Categories
All Business Consulting CTO/CIO Scalability

Why I can’t raise the bar at every firm

Screen-shot-2012-08-02-at-1.28.35-PM

Join 17,000 others and follow Sean Hull on twitter @hullsean. Also check out: Who is Sean Hull?

It may seem counterintuitive. If I am not the best solution provider, why on earth would I highlight it?

I believe by pointing those cases out, I also underline the clients and problems that I’m particularly well suited too, and for which I can really provide value. Read on!

1. People Problems

Sometimes, you’re hired to solve a particular problem which is framed as a technical one. Some process needs to be reworked, recoded or retooled. It’s framed as a technical problem, yet as things unfold the client already has the expertise in-house to solve & write the code. What then?

It may be that the right people aren’t communicating, project managers aren’t seeing the issues, or part of the human systems are gummed up. We can’t raise the technical bar, but we can help getting those folks talking.

I wrote about this before in When You’re Hired to Solve a People Problem.

Also: Why are oil spills & financial instability related to datacenter outages?

2. ORM Usage & Technical Debt

If you’ve read my blog you know I am not very fond of Object Relational Modelers.

I would also argue as Ward Cunningham does so elloquently that technical debt can be a real and pressing problem.

Here we would help identify and frame the problem, though the work of raising the bar technically involves the longer process of retooling & refactoring your code base.

Related: Why database choices are tricky

3. Where Commodity & Offshore Works

Some firms are already making use of odesk or offshoring resources, where you might pay as little as $150/day. If you have a very technical manager or CTO, such a solution may work well for you.

At the other end of the spectrum are the high priced senior consultants from firms like Oracle, Percona or Pythian. Yes they may set you back as much as $3500/day.

In those cases a scalability & performance review may make sense. Here’s how.. Although specialists are necessary, remember to ask yourself Why generalists are better at scaling the web.

We sit in the sweet spot between the two options. With low overhead, our prices are more affordable. At the same time you’re getting a whole lot more than a commodity solution. We’ll communicate in plain language with folks at every technical level. And for many firms that in itself is a value add.

Check this: Does Oracle Aim to Kill MySQL?

4. Existing team did their homework

Believe it or not, I’ve gone into consulting engagements where the existing team has really really done their homework.

In those cases it becomes much harder to raise the bar technically. In those cases I can help when existing team missed something. But more importantly, I can validate a correct setup, or identify technical debt.

Having an outside perspective then, can provide reassurance. As I see ten to fifteen new environments per year, I’ve seen hundreds in the past decade & a half. That’s helpful perspective in itself.

Read: Does Oracle Aim to Kill MySQL?

5. Availability & Uptime Are Already High

I wrote in depth about high availability in the Myth of Five Nines

At the end of the day, availability can only approach perfection, not actually reach it. That’s a property of complex systems. If your uptime is already extremely high, again we can validate your environment, review and provide & summarize findings. But we may not be able to raise the bar.

If that’s you, it’s a good problem to have!

Also: Why AirBNB Didn’t Have to Fail

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters