Also find Sean Hull’s ramblings on twitter @hullsean.
A good five years ago I worked for a firm in online education. Among various products they provided through their website, they were struggling with a process to get content churned out more quickly. The bottleneck was slowing down their business, and limiting the new products they could offer.
Help Us Publish, Please…
Among a number of things I was asked to look at, one particularly vexing problem surrounded publishing. Adding new products had become a cumbersome & difficult process. It took days sometimes weeks. For obvious reasons the stake holders wanted to wrestle this process out of the hands of engineering, and place it were it arguably belonged in the hands of the business units.
When you’re hired to solve specific technical problems it only figures that you go looking for software solutions. But sometimes the problems turn out to involve the people and processes of an organization. Getting them unstuck is one of the biggest challenges an professional services consultant can face. But it is also one of the most rewarding to solve.
Bumping into Fear, Uncertainty & Doubt
As I dug into the meat of the problem I began to work closely with the database administrator. He was a very smart gentleman & friendly in his own way. But he also spoke with a very thick accent and brusqueness about his manner that proved difficult at times. After working together for some awhile, however I began to win him over, and he started to trust me.
Looking for a top-flight database administrator? Here’s our interview guide for recruiters, managers and candidates alike
It became apparent that he was rather resistant to handing over the keys to the publish process to non-technical folks in other departments. Having handled his share of outages, and bungling screw ups, which sometimes fall on operations during some of the least hospitable hours on the dial, I could understand his concern. What’s more he knew the code which had grown unwieldy.
If I were to use a polite euphemism I would call it spaghetti code.
Management, Managers & Trouble Brewing
Around then the CTO decided to send a manager to sniff around. Unfortunately the manager in question was a very hands off type. His edict was simply to get this done in two weeks, and proceeded to go on vacation. Upon his return when things were still hitting snags, things started to go south.
Despite AWS failures firms like AirBNB and Reddit didn’t have to go down.
Though some of the process had been automated, I refused to move the changes into full push-button automation without first testing on dev environments. Of course those requests had fallen on deaf ears.
Problem comes to a head
Next the hands off manager escalated things upstream, of course adding his own spin on the situation. Shortly thereafter I’m called into the CTOs office only to get royally chewed out. A serious smack down which I’ll admit came almost out of nowhere.
A related article which readers also found quite popular: A CTO Must Never Do This
Oh, honestly I’m not complaining. On some level this is the job of the consultant. To act as the third party, wise or unbiased second opinion, and even punching bag at times.
Once things calmed down, I explained the situation from top to bottom. Yes there was messy code, and yes the process was complex, but it could of course be automated. What really stood in the way was a very resistant engineer who currently owned the process.
As much of the Sandy recovery continues, Devops can learn real lessons from the hurricane.
The CTO for his part concurred, having had trouble communicating with the engineer himself, and really not liking him much. He then appointed a proper project manager to oversee redoing the publish process from scratch.
A Plea for Cooperation
If I were to do it all again, for my part I’d sniff out the people dynamics more carefully. It’s often the case the companies have the engineering talent in house to solve a particular problem, but not the will or knowledge to put it into play.
Is your business growing? Having trouble scaling? Here’s how we do a performance review. It’s the first step on your way to hyper growth.
To managers & CTOs I’d encourage where possible to look for people, process and communication issues. Try to ferret out when something is an engineering problem, or whether it is one of people, silos and territory.