As always for hacker news, there’s a feisty debate about what such character includes.
Here’s my take.
1. Tackle learning quickly
Whether it means getting up to speed on a new service that AWS has launched, building a new api for an application that has never been built before, or getting up to speed with a new platform. Learning is ongoing.
Top engineers can make this a seemless part of their daily routine. Getting going quickly with new concepts and technologies, means wading into the water at first, to gain the general lay of the land. Now you can talk intelligently about the features, limitations and challenges.
From there he or she can dive in quickly to the specific area required for the project, and move forward with that technology comfortably.
When building code, it’s easy to get mired in libraries, sorting algorithms, and API minutiae. And all of that is very important. But what are you building, and why are you building it?
Understanding your customer, what they do day-to-day is not always easy. It means using the product yourself, and also talking with sales teams regularly to hear what they are hearing.
Then pouring all this into your user stories. For top engineers it will inform their decisions, and help them communicate to product & project managers about what issues their encountering. Tradeoffs about features, coding, performance and technical debt can be better evaluated with more information.
Does your code run slowly? Have you tried to figure out why?
Is it related to:
o latency in production that doesn’t appear our your laptop
o untuned production database queries
o untuned connection pooling
o slow API calls
o weird kubernetes or orchestration issues
o web host issue with memory shortage
o web host issue with slow unoptimized code
o issues on the client side
Top engineers have seen applications slow down or fail in a myriad of ways. This allows them to imagine how a new application might be failing, and investigate those.
In startups, your engineers need to communicate to many folks who don’t have an engineering background. Product & UI/UX folks probably are quite technical on their own. But what about sales teams who are dealing directly with customers? Or C-suite folks who watch the business bottom lines, but may not have the same low level technical understanding?
Great communicators can find the right metaphor to explain hurdles and holdups, technical debt, or the latest performance challenges. And explaining those in terms that resonate for others is incredibly valuable to the team and business velocity.
Meetups are always an excellent way to find great people. Tech networking, can be one part selling your ideas, projecting your prowess or career building for your own people. And meanwhile career forward, self-actuated employees are scoping you out too.
Connecting through your own people is always a good way to find talent. These referrals are also great because you already have a handle on the hires real-world experience and personality. Good for everybody.
As your startup grows, you get some press in TechCrunch or ReCode. This generates some household recognition in the tech community, or posts on HackerNews. From there you get inbound requests because smart people are on the prowl for a great company. Done & done!
What I really got from the article though, was how different types of engineers will think about problems differently.
1. Under the hood
For junior engineers who are still a bit green, and new to working in industry, they’re downward facing. Focusing solely under the hood, they may not see how their work contributes to a product, or how it fits into the overall picture for the business.
“When asked about their progress on a story, they would make an effort to ensure I understood what was happening under the hood and what tradeoffs they were facing using a vocabulary I was familiar with. ”
For her it was technical competence that stood out – or at least not the only thing – but rather how they were strategizing, and communicating their problems, challenges, and progress.
A more intermediate engineer, would sometimes anticipate & communicate better.
“In some cases, those engineers would come to me before I even had a chance to enter the paranoid zone and give me a simple explanation of how the team had learned new information since they first estimated the complexity of a story. ”
She is also speaking of situational awareness. So not working in a vacuum, communicating & incorporating that new information as it becomes available.
I shoot for five tasks on my todo list. Be sure they are small, 15-30 minute tasks because things have a way of ballooning. If what you’re doing takes longer, break it down into smaller pieces. This keeps you moving, and always making progress.
You might be tempted to have more items. But chances are you’ll spend an hour on emails and time on phone calls, and other distractions. And there will be preemptive tasks that suddenly require your attention. So keeping this list small, allows you to hit close to 100% success.
Sure there will be days when you’re *more* productive. It doesn’t hurt to pull some items off the long term list. 🙂
Smokers have an easy time with this. And perhaps coffee drinkers. If you’re anyone else, you may get into the habit of staying in your chair. Don’t. Regular breaks promote creative thinking, and physically moving helps get the mind in motion too.
Sometimes when I work in a coffeeshop I don’t bring my charger. That way I’m forced to take a break when the battery runs low.
Pat yourself on the back when you complete all your tasks. If it’s 4pm, so be it. Jet a bit early. You know there will be other days when you’re working until 8pm too. Promise yourself something when you finish. A treat, or a stroll through the park, or an extra ten minutes to walk your dog, or a frosty IPA. Whatever it is, rewards help remind is we’ve done well.
If you’re in a FT role, you may do most of your socializing with coworkers. That’s fine, but be sure to go to some regular meetups too. And followup with people. Maybe even give a few talks now and then. Networking is the most surefire way to build your career and always be growing. And it’s a little bit each day that it takes to build lasting momentum.
The ways i have found, network, meetups, blog weekly and have a newsletter that you send out monthly. Add everyone you ever meet to your newsletter. Write interesting things & appeal to a broad audience. Some receiving your newsletter will not read it but they will see your name pop up in their inbox once a month.
And don’t forget to tell your story. And tell it well. Craft a memorable origin narrative. Practice & and add or remove things that resonate with people you meet. Even ask people, what do you think about my presentation? Any suggestions? Is it confusing, enticing, exciting?
In some organizations that are smaller, you get a chance to wear a lot of hats. You aren’t so specialized because there are fewer total team members. For example there may not be one person devoted to the database work, and one developer takes on that responsibility. While there is not devops team, another developer automates infrastructure.
Alternatively do you prefer more clearly defined job roles? That may be a larger org that has many more engineers. In that way you can own your own tiny slice, and focus just on that skillset or tool.
Both are valid of course, but they may be different types of orgs or companies at different stages in their development.
This is an interesting question. For me personally, I prefer to have the biggest business impact. If I can come into an organization and raise the bar, even if the bar wasn’t high to begin with, that is very satisfying. If I don’t get to use the coolest wiz-bang technologies that’s ok with me.
Alternatively there are some organizations that are facing much more challenging problems. These tend to be very hard technical problems, where the bar is already quite high. In those you may be surrounded by very talented engineers indeed, and the baseline for entry is already quite high.
Again both are valid, just a matter of what type of environment you thrive in.
Devops is in serious demand these days. At every meetup or tech event I attend, I hear a recruiter or startup founder talking about it. It seems everyone wants to see benefits of talented operations brought to their business.
That said the skill set is very broad, which explains why there aren’t more devs picking up the batton.
I thought it would be helpful to put together a list of interview questions. There are certainly others, but here’s what I came up with.
1. Explain the gitflow release process
As a devops engineer you should have a good foundation about software delivery. With that you should understand git very well, especially the standard workflow.
Although there are other methods to manage code, one solid & proven method is gitflow. In a nutshell you have two main branches, development & master. Developers checkout a new branch to add a feature, and push it back to development branch. Your stage server can be built automatically off of this branch.
Periodically you will want to release a new version of the software. For this you merge development to master. UAT is then built automatically off of the master branch. When acceptance testing is done, you deploy off of master to production. Hence the saying always ship trunk.
Bonus points if you know that hotfixes are done directly off the master branch & pushed straight out that way.
There are a lot of tools in the devops toolbox these days. One that is great at provisioning resources is Terraform. With it you can specify in declarative code everything your application will need to run in the cloud. From IAM users, roles & groups, dynamodb tables, rds instances, VPCs & subnets, security groups, ec2 instances, ebs volumes, S3 buckets and more.
You may also choose to use CloudFormation of course, but in my experience terraform is more polished. What’s more it supports multi-cloud. Want to deploy in GCP or Azure, just port your templates & you’re up and running in no time.
It takes some time to get used to the new workflow of building things in terraform rather than at the AWS cli or dashboard, but once you do you’ll see benefits right away. You gain all the advantages of versioning code we see with other software development. Want to rollback, no problem. Want to do unit tests against your infrastructure? You can do that too!
The four big choices for configuration management these days are Ansible, Salt, Chef & Puppet. For my money Ansible has some nice advantages.
First it doesn’t require an agent. As long as you have SSH access to your box, you can manage it with Ansible. Plus your existing shell scripts are pretty easy to port to playbooks. Ansible also does not require a server to house your playbooks. Simply keep them in your git repository, and checkout to your desktop. Then run ansible-playbook on the yaml file. Voila, server configuration!
Unit testing & integration testing are super import parts of continuous integration. As you automate your tests, you formalize how your site & code should behave. That way when you automate the deployment, you can also automate the test process. Let the software do the drudgework of making sure a new feature hasn’t broken anything on the site.
As you automate more tests, you accelerate the software development process, because you’re doing less and less manually. That means being more agile, and makes the business more nimble.
Docker a low overhead way to run virtual machines on your local box or in the cloud. Although they’re not strictly distinct machines, nor do they need to boot an OS, they give you many of those benefits.
Docker can encapsulate legacy applications, allowing you to deploy them to servers that might not otherwise be easy to setup with older packages & software versions.
Docker can be used to build test boxes, during your deploy process to facilitate continuous integration testing.
Docker can be used to provision boxes in the cloud, and with swarm you can orchestrate clusters too. Pretty cool!
Since devops brings a new process of continuous delivery to the organization, it involves some risk. Actually doing things the old way involves more risk in the long term, because things can and will break. With automation, you can recovery quicker from failure.
But this new world, requires a leap of faith. It’s not right for every organization or in every case, and you’ll likely strike a balance from what the devops holy book says, and what your org can tolerate. However inevitably communication becomes very important as you advocate for new ways of doing things.
My theory is that devops enables the business in a lot of profound ways. Sure it means one sysadmin can do much more, manage a fleet of servers, and support a large user base. But it goes much deeper than that.
Being able to standup your entire dev, qa, or production environment at the click of the button transforms software delivery dramatically. It means it can happen more often, more easily, and with less risk to the business. It means you can do things like blue/green deployments, rolling out featues without any risk to the production environment running in parallel.
What kind of chops does it take?
Strong generalist skills
For starters you’ll need a pragmatist mindset. Not fanatical about one technology, but open to the many choices available. And as a generalist, you start with a familiarity with a broad spectrum of skills, from coding, troubleshooting & debugging, to performance tuning & integration testing.
Stir into the mix good operating system fundamentals, top to bottom knowledge of Unix & Linux, networking, configuration and more. Maybe you’ve built kernels, compiled packages by hand, or better yet contributed to a few open source projects yourself.
You’ll be comfortable with databases, frontend frameworks, backend technologies & APIs. But that’s not all. You’ll need a broad understanding of cloud technologies, from GCP to AWS. S3, EC2, VPCs, EBS, webservers, caching servers, load balancing, Route53 DNS, serverless lambda. Add to all of that programmable infrastructure through CloudFormation or Terraform.
Although as a devop you probably won’t be doing frontend dev, you’ll need some cursory understanding of those. You should be competent at Python and perhaps Nodejs. Maybe Ruby & bash scripts. You’ll need to understand JSON & Yaml, CloudFormation & Terraform if you want to deliver IAC.
These are fundamental. But what does that mean? Ops mindset is born out of necessity. Having seen failures & outages, you prioritize around uptime. A simpler stack means fewer moving parts & less to manage. Do as Martin Weiner would suggest & use boring tech.
But you’ll also need to reason about all these components. That’ll come from dozens of debug & troubleshooting sessions you’ll do through years of practice.
Build systems like CircleCI, Jenkins or Gitlab offer a way to automate code delivery. And as their use becomes more widespread knowing them becomes de rigueur. But it doesn’t end there.
With deployments you’ll have a lot to choose from. At the very simplest a single target deploy, to all-at-once, minimum in service and rolling upgrades. But if you have completely automated your dev, qa & prod infra buildout, you can dive into blue/green deployments, where you make a completely knew infra for each deploy, test, then tear down the old.
I think if you’ve made it this far you will agree that the technical know-how is a broad spectrum of modern computing expertise. But you’ll also need excellent people skills to put all this into practice.
That’s because devops is also about organizational transformation. Yes devs & ops have to get up to speed on the tech, but the organization has to get on board too. Many entrenched orgs pay lip service to devops, but still do a lot of things manually. This is out of fear as much as it stands as technical debt.
But getting past that requires evangelizing, and advocating. For that a leader in the devops department will need superb people skills. They’ll communicate concepts broadly across the organization to win hearts and minds.
She went on to say how much has changed in the last decade. We talked about how the database administrator, as a career role, wasn’t really being hired for much these days. Things had changed. Evolved a lot.
How do you keep up with all the new technology, she asked?
I went on to talk about Amazon RDS, EC2, lambda & serverless as really exciting stuff. And lets not forget terraform (I wrote a howto on terraform), ansible, jenkins and all the other deployment automation technologies.
We talked about Redshift too. It seems to be everywhere these days and starting to supplant hadoop as the warehouse of choice for analytics.
It was a great conversation, and afterward I decided to summarize my thoughts. Here’s how I think automation and the cloud are impacting the dba role.
My career pivots
Over the years I’ve poured all those computer science algorithms, coding & hardware skills into a lot of areas. Tools & popular language change. Frameworks change. But solid deductive reasoning remains priceless.
o C++ Developer
Fresh out of college I was doing Object Oriented Programming on the Macintosh with Codewarrior & powerplant. C++ development is no joke, and daily coding builds strength in a lot of areas. Turns out he application was a database application, so I was already getting my feet wet with databases.
o Jack of all trades developer & Unix admin
One type of job role that I highly recommend early on is as a generalist. At a small startup with less than ten employees, you become the primary technology solutions architect. So any projects that come along you get your hands dirty with. I was able to land one of these roles. I got to work on Windows one day, Mac programming another & Unix administration & Oracle yet another day.
o Oracle DBA
The third pivot was to work primarily on Oracle. I attended Oracle conferences & my peers were Oracle admins. Interestingly, many of the Oracle “experts” came from more of a business background, not computer science. So to have a more technical foundation really made you stand out.
For the startups I worked with, I was a performance guru, scalability expert. Managers may know they have Oracle in the mix, but ultimately the end goal is to speed up the website & make the business run. The technical nuts & bolts of Oracle DBA were almost incidental.
o MySQL & Postgres
As Linux matured, so did a lot of other open source projects. In particular the two big Open Source databases, MySQL & Postgres became viable.
Suddenly startups were willing to put their businesses on these technologies. They could avoid huge fees in Oracle licenses. Still there were not a lot of career database experts around, so this proved a good niche to focus on.
o RDS & Redshift on Amazon Cloud
Fast forward a few more years and it’s my fifth career pivot. Amazon Web Services bursts on the scene. Every startup is deploying their applications in the cloud. And they’re using Amazon RDS their managed database service to do it. That meant the traditional DBA role was less crucial. Sure the business still needed data expertise, but usually not as a dedicated role.
Time to shift gears and pour all of that Linux & server building experience into cloud deployments & migrating to the cloud.
o Devops, data, scalability & performance
Now of course the big sysadmin type role is usually called an SRE or Devops role. SRE being site reliability engineer. New name but many of the same responsibilities.
Now though infrastructure as code becomes front & center. Tools like CloudFormation & Terraform, plus Ansible, Chef & Jenkins are all quite mature, and being used everywhere.
Checkout your infrastructure code from git, and run terraform apply. And minutes later you have rebuilt your entire stack from bare metal to fully functioning & autoscaling application. Cool!
However these days cloud, automation & microservices have brought a lot of madness too! Don’t believe me check out this piece on microservice madness.
With microservices you have more databases across the enterprise, on more platforms. How do you restore all at the same time? How do you do point-in-time recovery? What if your managed service goes down?
Migration scripts have become popular to make DDL changes in the database. Going forward (adding columns or tables) is great. But should we be letting our deployment automation roll *BACK* DDL changes? Remember that deletes data right? 🙂
What about database drop & rebuild? Or throwing databases in a docker container? No bueno. But we’re seeing this more and more. New performance problems are cropping up because of that.
What about when your database upgrades automatically? Remember when you use a managed service, it is build for 1000 users, not one. So if your use case is different you may struggle.
In my experience upgrading RDS was a nightmare. Database as a service upgrades lack visibility. You don’t have OS or SSH access so you can’t keep track of things. You just simply wait.
No longer do we have “zero downtime”. With amazon RDS you have guarenteed downtime upgrades. No seriously.
As the field of databases fragments, we are wearing many more hats. If you like this challenge & enjoy being a generalist, you may feel at home here. But it is a long way from one platform one skill set career path.
Also fragmented db platforms means more complex recovery. I can’t stress this enough. It would become practically impossible to restore all microservices, all their underlying databases & all systems to one single point in time, if you need to.
Let’s say you’re hiring a devops & you want to suss out their database knowledge? Or you’re hiring a professional services firm or freelance consultant. Whatever the case you’ll need to sift through for the best people. Here’s how.
Caching is a popular way to speed up access to your backend database. Put Amazon’s elasticache behind your webserver, and you can reduce load on your database by 90%. Nice!
The two types that amazon supports are Memcache & Redis. Memcache is historically more popular. These days Redis seems a clear winner. It’s faster, and can maintain your cached data between restarts. That will save you I promise!
When you’re doing large reports for your business intelligence team, you don’t want to bog down your backend relational database. Redshift is purpose built for this use case.
I’ve see a report that took over 8 hours in MySQL return in under 60 seconds in Redshift!
A new offering is Amazon Spectrum. This tech is super cool. Load up all your data into S3, in standard CSV format. Then without even loading it into Redshift, you can query the S3 data directly. This is super useful. Firstly because S3 is 1/10th the price. But also because it allows you to stage your data before loading into Redshift itself. Goodbye Google Big Query! I talked about spectrum here.
What relational database options are there on Amazon?
Amazon supports a number of options through it’s Relational Database Service or RDS. This is managed databases, which means less work on your DBAs shoulders. It also may make upgrades slower and harder with more downtime, but you get what you pay for.
There are a lot of platforms available. As you might guess MySQL & Postgres are there. Great! Even better you can use MariaDB if that’s your favorite. You can also go with Aurora which is Amazon’s own home-brew drop in replacement for MySQL that promises greater durability and some speedups.
If you’re a glutton for punishment, you can even get Oracle & SQL Server working on RDS. Very nice!
Let’s be honest, Amazon wants to make this really easy. The quicker & simpler it is to get your data there, that more you’ll buy!
Amazon’s Database Migration Service or DMS allows you to configure your old database as a data source, then choose a Amazon db solution as destination, then just turn on the spigot and pump your data in!
ETL is extract transform and load, data warehouse terminology for slicing and dicing data before you load it into your warehouse. Many of todays warehouses are being built with the data lake model, because databases like Redshift have gotten so damn fast. That model means you stage all your source data as-is in your warehouse, then build views & summary tables as needed to speed up queries & reports. Even better you might look a tool like xplenty.