Tag: people

Categories
All Cloud Computing CTO/CIO

How do we lock down cloud systems from disgruntled engineers?

CommitStrip.com

I worked at a customer last year, on a short term assignment. A brilliant engineer had built their infrastructure, automated deployments, and managed all the systems. Sadly despite all the sleepless nights, and dedication, they hadn’t managed to build up good report with management.

Join 32,000 others and follow Sean Hull on twitter @hullsean.

I’ve seen this happen so many times, and I do find it a bit sad. Here’s an engineer who’s working his butt off, really wants the company to succeed. Really cares about the systems. But doesn’t connect well with people, often is dismissive, disrespectful or talks down to people like they’re stupid. All of this burns bridges, and there’s a lot of bad feelings between all parties.

How do you manage the exit process? Here’s a battery of recommendations for changing credentials & logins so that systems can’t be accessed anymore.

1. Lock out API access

You can do this by removing the administrator role or any other role their IAM user might have. That way you keep the account around *just in case*. This will also prevent them from doing anything on the console, but you can see if they attempt any logins.

Also: Is AWS too complex for small dev teams?

2. Lock out of servers

They may have the private keys for various serves in your environment. So to lock them out, scan through all the security groups, and make sure their whitelisted IPs are gone.

Are you using a bastion box for access? That’s ideal because then you only have one accesspoint. Eliminate their login and audit access there. Then you’ve covered your bases.

Related: Does Amazon eat it’s own dogfood?

3. Update deployment keys

At one of my customers the outgoing op had setup many moving parts & automated & orchestrated all the deployment processes beautifully. However he also used his personal github key inside jenkins. So when it went to deploy, it used those credentials to get the code from github. Oops.

We ended up creating a company github account, then updating jenkins with those credentials. There were of course other places in the capistrano bits that also needed to be reviewed.

Read: Is aws a patient that needs constant medication?

4. Update dashboard logins

Monitoring with NewRelic or Nagios? Perhaps you have a centralized dashboard for your internal apps? Or you’re using Slack?

Also: Is Amazon too big to fail?

5. Audit Non-key based logins

Have some servers outside of AWS in a traditional datacenter? Or even servers in AWS that are using usernames & passwords? Be sure to audit the full list of systems, and change passwords or disable accounts for the outgoing sysop.

Also: When hosting data on Amazon turns bloodsport?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Categories
All Book review iHeavy Newsletter

30 Days of Aspirin for your Ailing Management Headaches

Take Two & Call Me In the Morning

I like a book like Gerry Czarnecki’s even if I can’t pronounce his name. He’s laid out things for readers, a very busy bunch who are going to have trouble finding time in their day to read, but can sorely use the advice. Organized into 30 daily bites of probably 15-20 minutes, you can dig through on your commute to work, or on your lunch break.

Also check out Who Moved My Cheese a business self-help classic.

Czarnecki should know. As a 2nd Army Lieutenant then later heading up board of directors at organizations large and small he’s seen a lot. He digs up the best stories, and offers up lessons straight out of his experiences.

[quote]
On interviews – Be careful that you do not let the resume dominate your conversation. You will gain better insight if you listen to what the candidates want to talk about or what they think you want to talk about. – Gerry Czarnecki
[/quote]

Some of the topics you’ll touch on…

o dealing with mistaken hires that don’t work out
o finding a mentor
o career moves & pivots
o setting expectations with new hires
o hitting peak performance
o filling your bus with rock stars

For startups an all-time favorite of mine is REWORK – 37 Signals guys. These guys have taken the lean methodology and built a real business by being efficient. The pages resonate for me as a small business owner. I’ve found many of the same lessons work in the real world for me.

[mytweetlinks]

Also read
Thank you for arguing by Jay Heinrichs. With some of the best advice on rhetoric, sales, presentations & persuasion, I re-read bits of the book often.

Get some in your inbox: Exclusive monthly Scalable Startups. We share tips and special content. Here’s a sample

Categories
All Consulting iHeavy Newsletter

When You're Hired to Solve a People Problem

Also find Sean Hull’s ramblings on twitter @hullsean.

A good five years ago I worked for a firm in online education. Among various products they provided through their website, they were struggling with a process to get content churned out more quickly. The bottleneck was slowing down their business, and limiting the new products they could offer.

Help Us Publish, Please…

Among a number of things I was asked to look at, one particularly vexing problem surrounded publishing. Adding new products had become a cumbersome & difficult process. It took days sometimes weeks. For obvious reasons the stake holders wanted to wrestle this process out of the hands of engineering, and place it were it arguably belonged in the hands of the business units.

[quote]
When you’re hired to solve specific technical problems it only figures that you go looking for software solutions. But sometimes the problems turn out to involve the people and processes of an organization. Getting them unstuck is one of the biggest challenges an professional services consultant can face. But it is also one of the most rewarding to solve.
[/quote]

Bumping into Fear, Uncertainty & Doubt

As I dug into the meat of the problem I began to work closely with the database administrator. He was a very smart gentleman & friendly in his own way. But he also spoke with a very thick accent and brusqueness about his manner that proved difficult at times. After working together for some awhile, however I began to win him over, and he started to trust me.

Looking for a top-flight database administrator? Here’s our interview guide for recruiters, managers and candidates alike

It became apparent that he was rather resistant to handing over the keys to the publish process to non-technical folks in other departments. Having handled his share of outages, and bungling screw ups, which sometimes fall on operations during some of the least hospitable hours on the dial, I could understand his concern. What’s more he knew the code which had grown unwieldy.

If I were to use a polite euphemism I would call it spaghetti code.

Management, Managers & Trouble Brewing

Around then the CTO decided to send a manager to sniff around. Unfortunately the manager in question was a very hands off type. His edict was simply to get this done in two weeks, and proceeded to go on vacation. Upon his return when things were still hitting snags, things started to go south.

Despite AWS failures firms like AirBNB and Reddit didn’t have to go down.

Though some of the process had been automated, I refused to move the changes into full push-button automation without first testing on dev environments. Of course those requests had fallen on deaf ears.

Problem comes to a head

Next the hands off manager escalated things upstream, of course adding his own spin on the situation. Shortly thereafter I’m called into the CTOs office only to get royally chewed out. A serious smack down which I’ll admit came almost out of nowhere.

A related article which readers also found quite popular: A CTO Must Never Do This

Oh, honestly I’m not complaining. On some level this is the job of the consultant. To act as the third party, wise or unbiased second opinion, and even punching bag at times.

Once things calmed down, I explained the situation from top to bottom. Yes there was messy code, and yes the process was complex, but it could of course be automated. What really stood in the way was a very resistant engineer who currently owned the process.

As much of the Sandy recovery continues, Devops can learn real lessons from the hurricane.

The CTO for his part concurred, having had trouble communicating with the engineer himself, and really not liking him much. He then appointed a proper project manager to oversee redoing the publish process from scratch.

A Plea for Cooperation

If I were to do it all again, for my part I’d sniff out the people dynamics more carefully. It’s often the case the companies have the engineering talent in house to solve a particular problem, but not the will or knowledge to put it into play.

Is your business growing? Having trouble scaling? Here’s how we do a performance review. It’s the first step on your way to hyper growth.

To managers & CTOs I’d encourage where possible to look for people, process and communication issues. Try to ferret out when something is an engineering problem, or whether it is one of people, silos and territory.

Want more? Grab our Scalable Startups monthly for more tips and special content. Here’s a sample