Also find Sean Hull’s ramblings on twitter @hullsean.
One of the biggest jobs in operations is monitoring. There are so many servers, databases, webservers, search servers, backup servers. Each has lots of moving parts, lots that can go wrong. Typically if you have monitoring, and react to that monitoring, you’ll head off bigger problems later.
A problem is brewing
We, myself & the operations team started receiving alerts for one server. Space was filling up. Anyone can relate to this problem. You fill up your dropbox, or the drive on your laptop and all sorts of problems will quickly bubble to the surface.
Also check out – Why generalists are better at scaling the web.
As we investigated over the coming days, a complicated chain of processes and backups were using space on this server. Space that didn’t belong to them.
Dinner boils over
What happened next was inevitable. The weekly batch jobs kicked off and failed for lack of space. Those processes were not being monitored. Business units then discovered missing data in their reports and a firestorm of emails ensued.
Why weren’t these services being monitored, they wanted to know.
Time to shoot the messenger
Having recently seen a changing of the guard, and a couple of key positions left vacant, it was clear that the root problem was communication.
Looking for talent? Why is it so hard to find a mythical MySQL DBA or devops expert these days?
I followed up the group emails, explaining in polite tone that we do in fact have monitoring in place, but that it seemed a clear chain of command was missing, and this process fell through the cracks.
I quickly received a response from the CTO requesting that I not send “these types of emails” to the team and to direct issues directly to him.
You might also like: A CTO Must Never Do This
A consultants job
As the sands continued to shift, a lead architect did emerge, one who took ownership of the products overall. Acting as a sort of life guard with a higher perch from which to watch, we were able to escalate important issues & he would then prioritize the team accordingly.
Are you a startup grappling with scalability? Keep in mind these 5 things toxic to scalability
Sometimes things have to break a little first.
What’s more a consultants job isn’t necessarily to lead the pack, nor to force management to act. A consultant’s job is to provide the best advice possible & to raise issues to the decision makers. And yes sometimes it means being a bit of a fall guy.
Those are the breaks of the game.