Categories
All Cloud Computing CTO/CIO Devops

When should I use Ansible versus packer or Terraform?

via GIPHY

I was having a conversation with a colleague recently. We wee discussing devops, and the topic of Ansible came up as I was advocating it as a great too to get things done.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

Here’s what he had to say…

— quote —
I’ve tried using ansible a few times and this is what I found with it.

It is great for what it does. It’s wonderful to be able to spin up a new app or web server automatically. However what I have found for my needs is …

It is easier to build a piece of furniture than it is to explain all the steps required for someone else to build it. Or in order to replicated the steps automatically.

With cloud servers, it’s enough, for me that I’ve built it once. When I need to spin up another, I simply clone the working copy.

— unquote —

My thoughts below.

1. When is Terraform good

Terraform is a coss-platform infrastructure building tool. If you need an IAM user or S3 bucket, Terraform can create it. Need an ec2 instance of a particular type, deployed with an autoscaling group TF is a great tool for that.

With Terraform you can capture in code, everything about your application stack, so that you can standup a complete copy in another region, that’s powerful!

Read: How can 1% of something equal nothing?

2. When is packer right?

Packer is another useful tool that Devops can use to automate. Like AWS own EC2 Image Builder, it allows you to create the images that you boot your instances off of. Think of them as docker images for the server itself.

For example there are lots of dependencies your application requires, and you’ll install with your package manager. And there are services you want to start. You *could* use an ansible playbook to get these going, but better to build a new image that contains all the software you need on the box.

Packer easily sits into your CI pipeline, so you can have new software deployed and ready anytime.

The principal difference is that a new AMI requires you to spinup a new server. You can’t take action on a running server with this tool.

Related: Is Fred Wilson right about dealing in an honest, direct and transparent way?

3. When does Ansible make sense?

In particular here’s what my response was about Ansible itself.

— quote —

Absolutely. It’s an interesting balance to strike.

Because of course packer or EC2 image builder are very powerful and fit neatly into a CI pipeline. That said there are things ansible is nicely suited for too.

For example I want to distribute public keys onto specific servers. I have a yml file with the keys. I have a new developer starting, I have him or her git checkout branch, edit keys.yml, commit, push changes, then make a pull request. When the new keys.yml file gets merged, an ansible playbook kicks off to distribute the new set of keys to the relavant servers.

— unquote —

If you want to take actions on running servers, like deploying keys or other ongoing tweaks, that is where Ansible really shines.

Related: What mistakes did you make when starting as a consultant?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Categories
All Database Operations Web Operations Website Basics

Extract Transform & Load – What is it and why is it important?

So-called ETL relates to moving data from external sources into and out of relational databases or data warehouses.

Extract

Source systems may store data in an infinite variety of formats.  Extracting involves getting that data into common files for moving to the destination system.  CSV file also known as comma separated values is named because each of the records is stored as one line in the file, and fields are separated by commas, and often surrounded by quotes as well.  In MySQL INTO OUTFILE syntax can perform this function.  If you have a lot of tables to work with, you can script the process using the data dictionary as a lookup for table names, and create a .mysql script to then run with the mysql shell.  In Oracle you would use the spool command in SQL*Plus the command line shell.  Spool sends subsequent output from the screen also to a file.

Transform

This step involves modifying the extracted data in preparation for moving it into the target database server.  It may involve sweeping out blank records, or rearranging columns, or breaking files into smaller subsets of data.  You might also map values differently for instance if one column in the source database was gender with values M/F you might transform those to the strings “Male” and “Female” if that is more useful for your target database server.  Or you might transform those to numerical values, for instance Male & Female might be 0/1 in your target database.

Although I myriad of high level GUI tools exist to perform these functions, the Unix operating system includes a plethora of very powerful tools that every experience System Administrator is familiar with.  Those include grep & sed which operate on regular expressions and can perform data transformation at lightening speed.  Then there is sort which can sort data and send the results to stdout or the file of your choosing.  Other tools include wc – word count, cut which can remove columns and so forth.
Load

This final step involves moving the data into the database server, and it’s final target tables.  For instance in MySQL this might be done with the LOAD DATA INFILE syntax, while in Oracle you might use SQL*Loader, which is a very fast flat file dataloader.

Quora discussion by Sean Hull – What is ETL?