As I was building infrastructure code, I stumbled quite a few times. You hit a wall and you have to work through those confusing and frustrating moments.
Join 38,000 others and follow Sean Hull on twitter @hullsean.
Here are a few of the lessons I learned in the process of building code for AWS. It’s not easy but when you get there you can enjoy the vistas. They’re pretty amazing.
Don’t pass credentials
As you build your applications, there are moments where components need to use AWS in some way. Your webserver needs to use S3 or your ELK box needs to use CloudWatch. Maybe you want to do an RDS backup, or list EC2 instances.
However it’s not safe to pass your access_key and secret_access_key around. Those should be for your desktop only. So how best to handle this in the cloud?
IAM roles to the rescue. These are collections of privileges. The cool thing is they can be assigned at the INSTANCE LEVEL. Meaning your whole server has permissions to use said resources.
Do this by first creating a role with the privileges you want. Create a json policy document which outlines the specific rules as you see fit. Then create an instance profile for that role.
When you create your ec2 instance in Terraform, you’ll specify that instance profile. Either by ARN or if Terraform created it, by resource ID.
Related: How to avoid insane AWS bills
Keep passwords out of code
Even though we know it should not happen, sometimes it does. We need to be vigilant to stay on top of this problem. There are projects like Pivotal’s credential scan. This can be used to check your source files for passwords.
What about something like RDS? You’re going to need to specify a password in your Terraform code right? Wrong! You can define a variable with no default as follows:
variable "my_rds_pass" {
description = "password for rds database"
}
When Terraform comes upon this variable in your code, but sees there is no “default” value, it will prompt you when you do “$ terraform apply”
Related: How best to do discovery in cloud and devops engagements?
Versioning your code
When you first start building terraform code, chances are you create a directory, and some tf files, then do your “$ terraform apply”. When you watch that infra build for the first time, it’s exciting!
After you add more components, your code gets more complex. Hopefully you’ve created a git repo to house your code. You can check & commit the files, so you have them in a safe place. But of course there’s more to the equation than this.
How do you handle multiple environments, dev, stage & production all using the same code?
That’s where modules come in. Now at the beginning you may well have a module that looks like this:
module "all-proj" {
source = "../"
myvar = "true"
myregion = "us-east-1"
myami = "ami-64300001"
}
Etc and so on. That’s the first step in the right direction, however if you change your source code, all of your environments will now be using that code. They will get it as soon as you do “$ terraform apply” for each. That’s fine, but it doesn’t scale well.
Ultimately you want to manage your code like other software projects. So as you make changes, you’ll want to tag it.
So go ahead and checkin your latest changes:
# push your latest changes $ git push origin master # now tag it $ git tag -a v0.1 -m "my latest coolest infra" # now push the tags $ git push origin v0.1
Great now you want to modify your module slightly. As follows:
module "all-proj" {
source = "git::https://[email protected]/hullsean/myproj-infra.git?ref=v0.1"
myvar = "true"
myregion = "us-east-1"
myami = "ami-64300001"
}
Cool! Now each dev, stage and prod can reference a different version. So you are free to work on the infra without interrupting stage or prod. When you’re ready to promote that code, checkin, tag and update stage.
You could go a step further to be more agile, and have a post-commit hook that triggers the stage terraform apply. This though requires you to build solid infra tests. Checkout testinfra and terratest.
Related: Are you getting good at Terraform or wrestling with a bear?
Managing RDS backups
Amazon’s RDS service is a bit weird. I wrote in the past asking Is upgrading RDS like a shit-storm that will not end?. Yes I’ve had my grievances.
My recent discovery is even more serious! Terraform wants to build infra. And it wants to be able to later destroy that infra. In the case of databases, obviously the previous state is one you want to keep. You want that to be perpetual, beyond the infra build. Obvious, no?
Apparently not to the folks at Amazon. When you destroy an RDS instance it will destroy all the old backups you created. I have no idea why anyone would want this. Certainly not as a default behavior. What’s worse you can’t copy those backups elsewhere. Why not? They’re probably sitting in S3 anyway!
While you can take a final backup when you destroy an RDS instance, that’s wondeful and I recommend it. However that’s not enough. I highly suggest you take matters into your own hands. Build a script that calls pg_dump yourself, and copy those .sql or .dump files to S3 for safe keeping.
Related: Is zero downtime even possible on RDS?
When to use force_destroy on S3 buckets
As with RDS, when you create S3 buckets with your infra, you want to be able to cleanup later. But the trouble is that once you create a bucket, you’ll likely fill it with objects and files.
What then happens is when you go to do “$ terraform destroy” it will fail with an error. This makes sense as a default behavior. We don’t want data disappearing without our knowledge.
However you do want to be able to cleanup. So what to do? Two things.
Firstly, create a process, perhaps a lambda job or other bucket replication to regularly sync your s3 bucket to your permanent bucket archive location. Run that every fifteen minutes or as often as you need.
Then add a force_destroy line to your s3 bucket resource. Here’s an example s3 bucket for storing load balancer logs:
data "aws_elb_service_account" "main" {}
resource "aws_s3_bucket" "lb_logs" {
count = "${var.create-logs-bucket ? 1 : 0}"
force_destroy = "${var.force-destroy-logs-bucket}"
bucket = "${var.lb-logs-bucket}"
acl = "private"
policy = POLICY
{
"Id": "Policy",
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"s3:PutObject"
],
"Effect": "Allow",
"Resource": "arn:aws:s3:::${var.lb-logs-bucket}/*",
"Principal": {
"AWS": [
"${data.aws_elb_service_account.main.arn}"
]
}
}
]
}
POLICY
tags {
Environment = "${var.environment_name}"
}
}
NOTE: There should be “< <" above and to the left of POLICY. HTML was not having this, and I couldn't resolve it quickly. Oh well.
Related: Why generalists are better at scaling the web