Tag Archives: security

Locking down cloud systems from disgruntled engineers

medieval gate fortified aws

I worked at a customer last year, on a short term assignment. A brilliant engineer had built their infrastructure, automated deployments, and managed all the systems. Sadly despite all the sleepless nights, and dedication, they hadn’t managed to build up good report with management.

Join 32,000 others and follow Sean Hull on twitter @hullsean.

I’ve seen this happen so many times, and I do find it a bit sad. Here’s an engineer who’s working his butt off, really wants the company to succeed. Really cares about the systems. But doesn’t connect well with people, often is dismissive, disrespectful or talks down to people like they’re stupid. All burns bridges, and there’s a lot of bad feelings between all parties.

How to manage the exit process. Here’s a battery of recommendations for changing credentials & logins so that systems can’t be accessed anymore.

1. Lock out API access

You can do this by removing the administrator role or any other role their IAM user might have. That way you keep the account around *just in case*. This will also prevent them from doing anything on the console, but you can see if they attempt any logins.

Also: Is AWS too complex for small dev teams?

2. Lock out of servers

They may have the private keys for various serves in your environment. So to lock them out, scan through all the security groups, and make sure their whitelisted IPs are gone.

Are you using a bastion box for access? That’s ideal because then you only have one accesspoint. Eliminate their login and audit access there. Then you’ve covered your bases.

Related: Does Amazon eat it’s own dogfood?

3. Update deployment keys

At one of my customers the outgoing op had setup many moving parts & automated & orchestrated all the deployment processes beautifully. However he also used his personal github key inside jenkins. So when it went to deploy, it used those credentials to get the code from github. Oops.

We ended up creating a company github account, then updating jenkins with those credentials. There were of course other places in the capistrano bits that also needed to be reviewed.

Read: Is aws a patient that needs constant medication?

4. Dashboard logins

Monitoring with NewRelic or Nagios? Perhaps you have a centralized dashboard for your internal apps? Or you’re using Slack?

Also: Is Amazon too big to fail?

5. Non-key based logins

Have some servers outside of AWS in a traditional datacenter? Or even servers in AWS that are using usernames & passwords? Be sure to audit the full list of systems, and change passwords or disable accounts for the outgoing sysop.

Also: When hosting data on Amazon turns bloodsport?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Does FedRAMP formalize what good devops already do?



Amazon’s GovCloud provides a specialized region within Amazon’s global footprint of datacenters. These are hosted within the United States, and provide a subset of the full Amazon cloud functionality.

Join 32,000 others and follow Sean Hull on twitter @hullsean.

However, hosting within GovCloud is not the whole story. Beyond this, you’ll want to implement FedRAMP compliant procedures & policies.

Are these policies new? As a seasoned systems administrator of Unix & Linux networks, you’ll likely find these very familiar best practices. What they do however, is formalize those into a set of procedures for testing compliance. And that’s a good thing.

1. Use a bastion box

A bastion box is a single point of entry for all your SSH activity. Instead of allowing SSH access to any of your servers from *anywhere* on the internet, you limit it to one box. This box is hardened with multi-factor authentication for security, only opens port 22, monitors & logs access, and funnels movement to all your other boxes. Thus you gain a virtual perimeter that you’re already familiar with in more traditional firewall setups.

Also: Ward Cunningham explains the high cost of technical debt (video)

2. Monitor & scan for vulnerabilities

Monitoring, scanning & logging are all key facilities for security management. Regular patch management of each of your servers, is essential to protect from newly discovered vulnerabilities. FedRAMP also requires scanning by tools such as Nessus or Retina.

Also centralizing your authorization, access & error logs allows easy monitoring & alerting of threats & improper access attempts.

Related: Do managers underestimate the cost of operations?

3. Policy of least privilege

The policy of least privilege is an old friend in computing & managing unix systems. It means first to eliminate all privileges (default to none) and then grant only those a user requires to do his or her work.

In Amazon it means not using the root account for provisioning infrastructure, it means a clear separation of dev, test & production environments. It limits who can access production & especially make changes there. It limits who can see sensitive data.

As well, you’ll use Access Control Lists (ACL’s) and security groups to control which servers can reach which other servers, whom on the internet can touch specific servers & ports, and so forth. These are the Amazon Cloud equivalent of perimeter security you may be familiar with in more traditional firewalls.

Read: When hosting data on Amazon turns bloodsport

4. Encrypt your data

If you want to be truly secure, you’ll want to encrypt your data at rest. You can do this by using encrypted filesystems in Linux. That way data is in a digital envelope, even on disk. Only when data is read into memory is it unencrypted. This provides additional insurance, because your EBS snapshots, backups & so forth are all hidden from prying eyes.

Also: Why dropbox didn’t have to fail

5. Conclusion

Amazon’s GovCloud provides access to a subset of their cloud offerings including EC2 their elastic compute cloud virtual servers, EBS the elastic block storage their own storage area network, S3 for file storage, VPC, IAM, RDS, Elasticache & Redshift.

FedRAMP formalizes what good systems administrators do already. Secure systems, deliver reliability & high availability & protect from unauthorized entry.

Also: Is Amazon too big to fail?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Secrets of a happy Amazon hacker – IAM, MFA & locking down your account

aws logo

If you’re still using a password to login to your AWS account it’s time you batten down the hatches. With a little work you can dramatically improve security.

1. install command line tools

First get ahold of the aws comand line tools. They’re python based so you’ll need the package manager “pip” first.

$ curl -O https://bootstrap.pypa.io/get-pip.py
$ pip install awscli

Next configure your access key & secret key. You can edit the file below or use “$ aws configure”

$ cat .aws/credentials
aws_access_key_id = AAAAAAAAAAAAAAAABCD
aws_secret_access_key = ABcdefghijklmnop!mnors323


Also: Is Amazon too big to fail?

2. Create a new user

You don’t want to be using your aws root user for everything. So we’ll create a new user called “seancli”.

$ aws create-user --user-name "seancli"
$ aws iam create-login-profile --user-name "seancli" --password "seanpass"

Related: Did Airbnb have to fail?

3. give admin privileges

We want our new user to be able to administer things. So let’s give him administrator privileges to AWS resources. AdministratorAccess is a collection of permissions & a policy managed by AWS.


$ aws iam create-group –group-name “admin”
$ aws iam attach-group-policy –group-name “admin” –policy-arn “arn:aws:iam::aws:policy/AdministratorAccess”
$ aws iam add-user-to-group –group-name “admin” –user-name “seancli”

Read: When hosting data on Amazon turns bloodsport

4. Enable MFA

Now for the fun bit. Enable multi-factor authentication. This is important for really making your aws account secure. Remember anyone who gets into your account can delete *ALL* your infrastructure, and/or spinup servers which cost a lot of money. So just a password alone is not sufficient.

MFA uses your phone (or a key fob if you like) as the second factor.

A. Install Google Authenticator
B. Login to your aws dashboard
C. Click your name menu then select “Security Credentials”


amazon security credentials







D. Open the Multi-factor section


activate amazon mfa







E. Click “activate MFA” & a QR code with display


virtual mfa device amazon





F. Open your Google Authenticator app & click (+)
G. Select scan barcode
H. Point your smartphone camera at the QR code from step E.

You’ll be asked to enter *two* consecutive six-digit sequences. Once completed, try logging in again.

Also: Are SQL Databases Dead?

5. Test with command line

After you’ve created your new user, you should test it to make sure you can login properly.

Also: 5 Reasons to move data to Amazon Redshift

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Lulzsec, Anonymous and the sorry state of internet security

zalgo text

If you’ve been hiding under a rock for the past few years, you might not have heard of Anonymous, the headline grabbing hacker group that’s famous for attacking citibank, ebay, Sony, the FBI, CIA and the websites of various world governments.

Parmy Olson takes us on a ride, through tales that are riveting, and quite a bit scary for what they reveal about today’s internet, and the false sense of security we all have.

Join 14,000 others and follow Sean Hull on twitter @hullsean.

Kids these days!

By now you’ve probably heard their names T-flow, Topiary, Sabu, & Kayla. And then there was AVunit, pwnsauce, Sup_g, and Havij. Cool characters, sitting at keyboards all over the world hatching menacing attacks, and seeming more organized than they actually were…

Topiary jumped into the role as spokesman for the group. Listening to this live hack only seems amusing in retrospect, now that the group has been brought down…

Read: Why devops talent is in short supply

For all the subcultures you’ve never heard of…

Today’s internet is rife with fascinating subcultures, many I’d never heard of. Parmy’s book on Anonymous takes us to the door of all these places, and gives us a candid peak at what goes on there. Kids these days are up to no good!

The bizarro Encyclopedia Dramatica is a wikipedia of weirdness. And then there’s Googledorks, a hackers delight of exploits (ways to break into systems online), and hacks.

And let’s not forget 4Chan the online community and forum that hatched Anonymous.

You thought Ascii Art was cool, but have you heard of zalgo text? That’s the text garbling software that created this posts image.

If you’re looking to dig a little deeper, browse over to know your meme, a sort of urban dictionary for internet subcultures.

Don’t forget the 47 rules of the internet. I’m still looking for rules two through thirty three. Does this have something to do with this 33?

Read: How to evaluate an independent consultant expert

With only a very thin blanket to secure us…

If you’re not already a touch paranoid with the risks of online banking, social networks and identity theft you will be after reading this tale.

Anonymous troublemakers were able to send SWAT teams to unsuspecting people’s homes, crowd source personal information, social engineer their way to facts about someone and then dox them publishing all that personal information online.

On the more technical side, many sites are vulnerable to SQL Injection a rather technical sounding method to trick websites into dumping the contents of their databases back to a hacker. There’s even an automated tool called sqlmap to help you with the dirty work.

And then there are the very illegal denial of service attack tools like the ominous sounding low orbit ion cannon. Please don’t try this at home!

Definitely the worst of all offenders are the botnets, swarms of infected computers that can be controlled from a central location, to wreak havoc on users and internet firms alike. Thanks Bill!

As a parting word, take a quick look at this instructional video on using backtrack5, a hacking & security testing tool…

Also: Why a killer title can make or break your content efforts

The older roots of hacking circa 80’s and 90’s

I remember back in the 80’s when War Games came out. It was a scary premise. With the cold war between the US and the former Soviet Union in full bloom, it felt very real.

The 90’s brought Clifford Stoll hunting a hacker through his computer systems in The Cuckoo’s Egg.

And then along comes Kevin Mitnick, turning his finger up at US agents, and wreaking his own havoc in his wake.

The anonymous story turns more political when they meet the likes of Julian Assange, but even that isn’t new. Remember the Pentagon Papers?

What’s really knew is how the internet has grown, but how computers have not gotten more secure through that period. It has all grown more brittle, with many websites, and personal computers steered by unsuspecting users.

Read: Why high availability is so very hard to deliver

Surprisingly soft landing

One thing that really surprised me in this tale, was the sentences many Anons received. The way the headlines read, this was real all-out warfare on governments and corporations a like. But reading the judgements, it appears judges had a different perspective.

Although there were certainly compromises of personal information, the group really wasn’t responsible for a huge amount of theft & fraud. Sure they took down some websites, but whom does that really harm. It makes great headlines, but the bigger systems behind the scenes are actually more secure than that.

”IRC is just the crap out of everyone’s minds…” – Topiary on words thought-typed in IRC chats

After flipping through to the end, it seems we’ve taken a ride through the internet underground, but not through the criminal underworld. That is out there surely, but it’s not run by this scattered team of recluse misfits.

Related: Why Airbnb didn’t have to fail

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

5 startup & scalability blogs I never miss – week 2

5 blogs week 2

Join 11,500 others and follow Sean Hull on twitter @hullsean.

Hunter Walk – Startups

If you want to have your finger on the pulse of startup land, there aren’t many better places to start than Hunter Walk’s 99% humble writings. Google finds his top posts on topics like AngelList, Advisors, and reinventing the movie theatre. Good writing, insiders view.

Read: NYC technology startups are hiring

Arnold Waldstein – Marketing

I first found Arnold’s blog using my trusty disqus discovery hack. He had written an interesting piece about new mobile shopping at popup stores like Kate Spade.

Follow him on Disqus, follow the blog, get the newsletter. All good stuff.

Read This: Why hiring is a numbers game

Claire Diaz Ortiz – Social Media

Claire writes a lot about social media, twitter & blogging. She wrote an excellent guide to increasing your pagerank, another on 30 important people to follow on twitter and more. She can even help you find a job.

Check out: Top MySQL DBA Interview questions for candidates, managers & recruiters

Bruce Schneier – Security

Bruce Schneier is one of the original bad boys of computer security. He writes about broad topics, that affect us all everyday from common sense about airport security, to the impacts of cryptography for you and me. Very worth looking at regularly, just to see what he’s paying attention to.

Also: Why operations & MySQL DBA talent is hard to find

Eric Hammond – Amazon Cloud

Eric Hammond has been writing about Amazon Web Services, EC2 & Ubuntu for years now. He maintains and releases some excellent AMIs, those are the machine images for spinning up new servers in Amazon’s cloud.

Even if you’re not big on the command line, you can get a lot of critical insight about the Amazon cloud by keeping up with his blog. Jeff Barr’s AWS blog is also good, but not nearly as critical and boots on the ground as Eric’s.

Also: 8 Questions to ask an AWS expert

Get some in your inbox: Exclusive monthly Scalable Startups. We share tips and special content. Here’s a sample

Oracle to MySQL Migration Considerations

There are a lot of forms of transportation, from walking to bike riding, motorcycles and cars to busses, trains and airplanes.  Each mode of transport will get you from point a to point b, but one may be faster, or more comfortable and another more cost effective.  It’s important to keep in mind when comparing databases like Oracle and MySQL that there are indeed a lot of feature differences, a lot of cultural differences, and a lot of cost differences.  There are also a lot of impassioned people on both sides arguing at the tomfoolery of the other.  Hopefully we can dispel some of the myths and discuss the topic fairly.

** See also: Migrating MySQL to Oracle Guide **

As a long time Oracle DBA turned MySQL expert, I’ve spent time with clients running both database engines and many migrating from one to the other.  I can speak to many of the differences between the two environments.  I’ll cover the following:

  1. Query & Optimizer Limitations
  2. Security Differences
  3. Replication & HA Are Done Differently
  4. Installation & Administration Simplicity
  5. Watch Out – Triggers, Stored Procedures, Materialized Views & Snapshots
  6. Huge Community Support – Open-source Add-ons
  7. Enter The Cloud With MySQL
  8. Backup and Recovery
  9. Miscellaneous Considerations

Check back again as we edit and publish the various sections above.

Managing Security in Amazon Web Services

Security is on everyone’s mind when talking about the cloud.  What are some important considerations?

For the web operations team:

  1. AWS has no perimeter security, should this be an overriding concern?
  2. How do I manage authentication keys?
  3. How do I harden my machine images?

** Original article — Intro to EC2 Cloud Deployments **

Amazon’s security groups can provide strong security if used properly.  Create security groups with specific minimum privileges, and do not expose your sensitive data – ie database to the internet directly, but only to other security groups.  On the positive side, AWS security groups mean there is no single point to mount an attack against as with a traditional enterprises network security.  What’s more there is no opportunity to accidentally erase network rules since they are defined in groups in AWS.

Authentication keys can be managed in a couple of different ways.  One way is to build them into the AMI.  From there any server spinup based on that AMI will be accessible by the owner of those credentials.  Alternatively a more flexible approach would be to pass in the credentials when you spinup the server, allowing you to dynamically control who has access to that server.

Hardening your AMIs in EC2 is much like hardening any Unix or Linux server.  Disable user accounts, ssh password authentication, and unnecessary services.  Consider a tool like AppArmor to fence applications in and keep them out of areas they don’t belong.  This can be an ongoing process that is repeated if the unfortunate happens and you are compromised.

You should also consider:

  • AWS password recovery mechanism is not as secure as a traditional managed hosting provider.  Use a very strong password to lock down your AWS account and monitor it’s usage.
  • Consider encrypted filesystems for your database mount point.  Pass in decryption key at server spinup time.
  • Consider storing particularly sensitive data outside of the cloud and expose through SSL API call.
  • Consider encrypting your backups.  S3 security is not proven.

For CTOs and Operations Managers:

  1. Where is my data physically located?
  2. Should I rely entirely on one provider?
  3. What if my cloud provider does not sufficiently protect the network?

Although you do not know where your data is physically located in S3 and EC2, you have the choice of whether or not to encrypt your data and/or the entire filesystem.  You also control access to the server.  So from a technical standpoint it may not matter whether you control where the server is physically.  Of course laws, standards and compliance rules may dictate otherwise.

You also don’t want to put all your eggs in one basket.  There are all sorts of things that can happen to a provider, from going out of business, to lawsuits that directly or indirectly affect you to even political pressure as in the wikileaks case.  A cloud provider may well choose the easier road and pull the plug rather than deal with any complicated legal entanglements.  For all these reasons you should be keeping regular backups of your data either on in-house servers, or alternatively at a second provider.

As a further insurance option, consider host intrusion detection software.  This will give you additional peace of mind against the potential of your cloud provider not sufficiently protecting their own network.

Additionally consider that:

  • A simple password recovery mechanism in AWS is all that sits between you and a hacker to your infrastructure.  Choose a very secure password, and monitor it’s usage.
  • EC2 servers are not nearly as reliable as traditional physical servers.  Test your deployment scripts, and your disaster recovery scenarios again and again.
  • Responding to a compromise will be much easier in the cloud.  Spinup the replacement server, and keep the EBS volume around for later analysis.

As with any new paradigm there is an element of the unknown and unproven which we are understandably concerned about.  Cloud hosted servers and computing can be just as secure if not more secure than traditional managed servers, or servers you can physically touch in-house.

Review: Cloud Application Architectures

George Reese’s book doesn’t have the catchiest title, but the book is superb.  One thing to keep in mind, it is not a nuts and bolts or howto type of book.  Although there is a quick intro to EC2 APIs etc, you’re better off looking at the AWS docs, or Jeff Barr’s book on the subject.  Reese’s book is really all about answering difficult questions involving cloud deployments. Continue reading Review: Cloud Application Architectures

Tracking the Wily Proxy Hackers

Recently the server that hosts our business was hacked. This interrupted the service of twelve different websites we host, as well as our corporate mail. Needless to say it caused us plenty of headaches, sleepless nights, and frustrating hours. In retrospect, however it has instilled a greater appreciation for computer security, a greater awareness, and further, a stronger perseverence to keep the systems locked down.
Watching the news these days, and sites like Security Focus can be disheartening to say the least. SPAM is at an all-time high, windows viruses, trojans, and malware are wreaking havoc to corporate intranets, and the internet at large, and the situation only seems to get worse. Running a server on the internet nowadays is like opening shop in New York City back in the days of street crime and daily trouble.
Unfortunately some of us in the Unix and Macintosh world have grown a bit too confident. With all of the vulnerabilities being found in various versions of Windows, IIS and Internet Explorer, folks on the other side of the fence figure they have less to worry about. We may have less to worry about, but that certainly doesn’t mean nothing. So here is the story of what happened to us, and what we did about it.
We upgraded our systems in December of 2004, and figuring Mandrake 9.2 was more stable than 10.x we installed that. We spent the time recovering all of our websites from backups, rsyncing things accross the internet. Each website has it’s own document root as well as specific configuration lines in the Apache httpd.conf file. In addition the mail server had to be configured, as well as DNS changes. Lastly once the system was up and running, we mirrored everything on root for redundancy and protection of loss of a single drive. All told we spent about 30+ hours but we were back up and running soon enough. A lot of the bulk of that time was spent moving data accross the internet, and was unattended.
Around the end of January we started seeing some spikes in hits on some of our sites, but didn’t think much about it. A few weeks went by, but generally the systems were behaving normally, but starting to be a bit slow. By mid-February we were starting to have problems. The network we are hosting on was having trouble with bandwidth, browsing, and experiencing outages of their own. We also showed up on the Composite Blocking List and the Spamhaus List.
When that happened it opened our eyes, if only a bit. We knew something was happening which was originating from that network. So we did two things. First we tested our Postfix mail server for Open Mail Relay. We had experienced this a year earlier with a qmail misconfiguration, and since it is quite common, thought this might be the problem. However, we were setup correctly, and that was not the issue. Next, we scanned all of the windows and Macintosh machines on that network for viruses, trojans, and so on. We found a couple of things, and fixed them. We then removed ourselves from the CBL + spamhaus lists.
Once again our mail was flowing out, but a day later, the problem struck again. Being the Unix folks we our, we starting pointing fingers at the Windows machines. Sometimes Norton, MacAfee et al. don’t catch all the viruses. We suspected those pesky windows machines to be the culprit. Many of the malware programs that Windows users unwittingly install on their machines relay spam so that spammers can send email out anonymously. So your windows machine is coopted as a spam host, sending out thousands of messages a minute.
To get around the problem in the short term, we contacted some associates of ours, to relay mail through them. This is different than an open mail relay, since you are specifically requesting permission to send mail through another agent. So we could once again send mail, and our problem was temporarily solved. However, our server got slower, and so did our websites. It got to the point where the network hosting our server couldn’t send outbound traffic, or visit websites. Quite a problem.
The admin managing that network contacted Verizon, the broadband provider, and discussed the problem with their tech department. They suggested unplugging machines on the network one-by-one, until the traffic spike subsided. He proceeded to do just that, and what do you know but when our server was unplugged, the bandwidth usage dropped to ZERO. The support rep suspected we were streaming audio or video files, which of course we were not, so the only obvious conclusion was spam.
What to do, well first hide your head between your tail, and admit that your unix server has been hacked is a start. Next we rebuilt the server with Mandrake 10.1. There were some vulnerabilities in SSH that we were using, as well as Apache, and PHP, so upgrading to the latest Mandrake distro version upgraded all these packages in one go. We broke our mirrored drives, and installed Mandrake on one of them, and the did a disk to disk copy of all the data from /home to the new drive. Once that was complete we started up again, and things were looking good.
Back on the internet, things started slowing down again, so we started monitoring our Apache logs. We saw some strange activity in there, so blocked HTTP at the router, and found the performance problems, and bandwidth problems eliminated. So we knew there was something wrong with Apache. We searched for bugs, but didn’t find anything too heinous. Upon closer examination of the logs, however, we found strange redirects to port 25 on other machines. How was that happening?
Apache has a facility for acting as a proxy. That is it can get webpages, and in fact make other requests of remote machines, and proxy those requests back to an originating source. Imagine standing on a mountain top. You can see to the other side of the mountain, and are reading smoke signals from a village there. You then send those same smoke signals to the next village over. They can read your smoke signals, but don’t know the identity of the sender, only that you’re sending a message to them. You can understand the message, but can’t determine the sender. Proxying with internet based servers works much the same way. In fact the Open Mail Relay we discussed above is exactly that, which is why it’s so important that it be closed.
So we looked over these logs and found strangely that Apache was doing the same thing! In fact Apache was an open mail relay, and open proxy in general. This mod_proxy module came preinstalled with our apache, and though we did not configure it, it was working none the less. So we researched the issue, and found it was not considered a bug. It was in fact part of the software that when configured correctly can come in quite handy. Of course we didn’t need it, so we spent some time disabling through configuration changes in the httpd.conf. Despite these changes, we were still seeing some traffic, so we decided to play rough. We recompiled apache from scratch with the module completely disabled. Further attempts to configure httpd.conf using that module failed, proving to us that it was indeed no longer present in the software.
We disabled the block at our router, and watched things for a couple of days. We were still seeing funny traffic. Paranoid at this point, we blocked at the router, to analyze the logs some more. We could not figure out how this might still be happening, and checked the PHP forums for bugs related to this. Finding none, and not wanting to just start recompiling modules at random, we looked at the logs again.
We found that our server, when making a failed request, was redirecting the user to our homepage. So the proxy requests were failing, but redirecting the user to our homepage. Checking the stats confirmed this. We received 5000 hits that day, a 1000% above normal. Realizing these scans and attempts to proxy were failing, we began to relax. Knowing we were probably on some spammers top-10 hacked sites in North America list, we also figured that their automated systems would remove us from such a list once our server stopped server proxy requests. And that’s exactly what we found. After a couple days the hits dropped off to 2500, and then back below 1000 before weeks end.