When a client takes a swing at you

MUHAMMAD ALI ROCKS GEORGE FOREMAN ON THE JAW

Join 16,000 others and follow Sean Hull on twitter @hullsean.

1. A changing of the guard

Back in the dot-com era, circa 1999 I worked for a startup in some transition. Upon meeting the team, I met the new CTO Harvey, who joined just a month before. Also on the team was the IT director Bill, who had been with the firm for five years.

After spending time in initial meetings & discovery, I put together an outline and my plan to migrate them to Oracle. The project kicked off shortly thereafter.

Also: Why Oracle Won’t Kill MySQL

2. Team lead sucker punches you

I spent the first week onsite so I could work closely with the team, specifically at Bill’s request. We worked almost side-by-side for a few days, and as I worked through some of the challenges of their application, and how it might interact with Oracle. At that time I was still working on some test boxes, as the new Oracle server was not yet setup.

First thing Monday while working remote I email Bill and CC Harvey to ask how things are going setting up the new server to house Oracle. A fairly harmless email, after what seemed like a successful previous week.

The response from Bill the director of IT was sharp and quick. He emailed back:

“The server is already setup, and I’ve installed Oracle on it. I have much of the data moved over. I’m not sure what you’ve been working on or how you will be able to help us on this project. Please advise.”

This came as a big surprise, as we had been working so closely together. We had also exchanged various emails to get details & configuration steps as well. It also seemed strange that he’d go ahead and complete the work that he had asked me to work on.

Related: Are SQL Databases Dead?

3. Proceed with caution

I quickly reached out to him, discussed status over IM and next steps. I also suggested that I come into the office again, to help with communication.

The following day I returned to the office, and met with him privately. I gently asked about his concerns, and if he had reviewed my task list and consulting agreement. It seemed that some of the terms & details had been overlooked. What’s more he and the CTO weren’t seeing eye-to-eye.

I then explained in a nice way, and to express that I had no plans to step on any toes, but that

“I’m glad to work with you Bill, in any way you see best, and on whatever tasks you decide I can help with.”

This seemed to put him at ease, and we moved forward.

Read this: AirBNB Didn’t Have to Fail

4. Green Shoots

As the engagement progressed it came to light that Harvey had hired me against Bill’s wishes. So Bill’s move seemed more motivated by feeling threatened than anything else.

Over the years I’ve learned time and again not to jump to conclusions. Especially at the start of a consulting assignment, there are likely a complex mix of personalities, and human dynamics that come into play. Sometimes when someone lashes out, it isn’t even directed at you per se, but because of a difficult transition period.

Patience, understanding and renewed efforts to communicate often win the day.

Check this: Why Are Devops & DBAs in Short Supply?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Connect to MySQL in the Amazon Public Cloud

MySQL on Amazon Cloud AWS

Troubleshooting MySQL on Amazon can be a real test of patience. There are quite a few different things to watch out for in terms of connectivity & networking. Sometimes a checklist can help.

Join 16,000 others and follow Sean Hull on twitter @hullsean.

Here’s my exhaustive list of things that can block you.

1. Be sure to create users & grants

Chances are you did something like this to create your user:


mysql> CREATE USER ‘sean’@‘localhost’ IDENTIFIED BY ‘password’;
mysql> GRANT ALL PRIVILEGES ON sean_schema.* TO ‘sean’@‘localhost' WITH GRANT OPTION;

But that won’t help you when connecting from a remote Amazon box. So what to do? Here’s an example:


mysql> CREATE USER ‘sean’@’10.10.%’ IDENTIFIED BY ‘password’;
mysql> GRANT ALL PRIVILEGES ON sean_schema.* TO ‘sean’@‘%’ WITH GRANT OPTION;

You may need to make your source IP wildcard *more* aggressive. For example consider ’10.%’. You *may* even with with ‘%’ which allows *all* source IPs. This may sound dangerous, but if you use a tight security group (see item #3 below), you can still be safe.

Related: Why Oracle Won’t Kill MySQL

2. Make sure iptables is not a problem

IPTables is a Linux service that acts like a private firewall for each server. Some AMIs will have it enabled by default. If you’re having trouble like I did, this can definitely trip you up. That’s because your connection will fail silently without telling you, hey the OS won’t let me into that port!

If you are a networking pro you’ve probably already fiddled with iptables. Feel free to add specific rules, and keep it turned on. However I’d recommend just disabling it completely, and using your Amazon security groups to protect your ports.


$ /etc/init.d/iptables stop
$ chkconfig --list iptables
iptables 0:off 1:off 2:on 3:on 4:on 5:on 6:off
$ chkconfig --del iptables
$ chkconfig --list iptables
service iptables supports chkconfig, but is not referenced in any runlevel (run 'chkconfig --add iptables')

Also: Are SQL Databases Dying Out?

3. Test & verify amazon security group settings

Security groups in Amazon can be tricky. I recommend the following:

o create a security group webserver_group
- allow port 80 from 0.0.0.0/0
- allow port 443 from 0.0.0.0/0
- allow port 22 from

o create a security group db_group
- allow port 22 from
- allow 3306 from

What’s happening here? We can’t specify a fixed set of IP addresses because they can change in Amazon. So essentially what we’ve done is say *any* requests from servers in our Amazon package, which are in the webserver_group security group, can connect to port 3306. Pretty cool right?

This means we’re pretty locked down. No internet connections to 3306, so we can be a little looser (see item #1 above) about our grants and source IPs.

What about if you want to use your GUI tools to hit your Amazon hosted MySQL boxes? Say you like to use the Oracle Workbench, Navicat or Toad to connect to MySQL. One way you could do this is configure your db_group to allow 3306 from your office subnet. Then anyone VPN’d into your office will be able to use the tools they like.

Another option is to use Amazon VPC for your servers. You’ll setup an Amazon Virtual Private Gateway, which is a direct VPN connection between Amazon’s datacenter and your datacenter. This can be a messy process, and you’ll want to contact your network admin to help. Once it’s setup, amazon boxes appear to sit on your office or datacenter network. Cool stuff!


$ mysql -h xxx.xxx.xxx.xxx -u admin -p
Enter password:
ERROR 2003 (HY000): Can't connect to MySQL server on 'xxx.xxx.xxx.xxx'

Read this: Why are MySQL experts in such short supply?

4. MySQL network settings

If MySQL is bound to the wrong IP address you can have real problems. First be sure skip_networking is OFF. If it is ON change it in /etc/my.cnf & restart MySQL.


mysql> show variables like 'skip_net%';
+-----------------+-------+
| Variable_name | Value |
+-----------------+-------+
| skip_networking | OFF |
+-----------------+-------+
1 row in set (0.00 sec)

The other MySQL setting that can be problematic is bind-address. First check what it is set to:


$ cat /etc/my.cnf | grep bind
bind-address=127.0.0.1

This isn’t going to allow remote connections. In amazon however, your IP address may change upon reboot. So there is a special setting to allow binding to any IP:


bind-address=0.0.0.0

Related: Bulletproofing MySQL Replication with Checksums

5. installing mysql client & telnet for troubleshooting

You have two options for troubleshooting on the webserver side. If you’re simply trying to check by mysql command line, you may get blocked up if the network settings & security groups aren’t configured right. So use telnet first.


$ yum install -y telnet

$ telnet 10.10.10.1 3306
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
4
5.1.71??gu9Y6B'/y9Oay`QV Connection closed by foreign host.
$

If you don't get a responce, it's not an issue with users or grants, but rather that the port isn't opened. Check iptables, check bind-address and check security groups.

Check this: Top MySQL DBA Interview Questions

6. SE Linux related issues

SE Linux will do a lot of good, if managed properly. However if you're not aware of it's existence, it can be very very frustrating. Symptoms can be as abstract as allergies, a cold or flu. It can monitor files, and prevent MySQL from being able to write where it needs to,

Read this: Migrating MySQL to Oracle

7. RPM & later centos yum repo install conflicts

I had real problems doing a custom install for a customer. They didn't want to use a repository for various settings, but preferred downloading RPMs. There were a few other customizations which were tripping things up.

Based on all the connectivity issues I was having, I backed out of the RPM based install, and then ran through a stock yum install. After doing that, I started seeing these weird errors in the mysqld.log

120328 21:32:40 [ERROR] Can't start server: Bind on TCP/IP port: Address already in use
120328 21:32:40 [ERROR] Do you already have another mysqld server running on port: 3306 ?
120328 21:32:40 [ERROR] Aborting
If I run "netstat -nat | grep 3306" in my terminal, I get the following:
tcp4 0 0 *.3306 . LISTEN

I spent hours spinning my wheels and not able to figure out what was happening here. At first it seemed a leftover pid file was the culprit. In the end it appeared the *old* /etc/init.d/mysql script was still in place, and the new yum packages wouldn't work with that.

I ended up just scrapping the whole box, and starting from scratch. Sometimes you have to do that. After a clean build, all was fine.

Related: RDS or MySQL 10 Use Cases

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don't work with recruiters

5 Things Frans Johansson says about innovation

medici affect johansson

You may not have heard of Medici before, but you’ve probably heard of the renaissance. The medici family hosted the round tables, the meetups, the social gatherings & mixers. They brought diverse artisans engineers & thinkers together, and the world hasn’t been the same since!

b>Join 16,000 others and follow Sean Hull on twitter @hullsean.

In the Medici Effect, Frans dissects what this famous family did. His case studies include the likes of Richard Branson, Deepak Chopra, Charles Darwin, Thomas Edison, Orit Gadiesh, Marcus Samuelsson, George Soros & our own favorite Linus Torvalds,

What he discovered really surprised me.

1. Swim at the intersection

Hanging out with folks in your field is great. Whether you’re a physician, financial analyst, Ruby programmer, or artist. But it won’t expose us to enough new ideas. To get that, you need to hang out with those in other disciplines. Learn a language, take dance classes, try your hand at a new sport, or attend meetups of wedding planners or DJs. Whatever it takes to get out of your comfort zone is what will put you at the intersection.

Also: Why a killer title can make or break your content efforts

2. You need quantity to get quality

This was a very surprising finding of their research. One might think that greats like Albert Einstein were geniuses from the start. But it turns out one consistent factor between all these folks is the quantity of their attempts. They came up with many many ideas, and chased as many as they could. Of course they are only remembered for their successes, but this hides the underlying mathematics. It’s a numbers game in almost all of these cases.

Read: Are SQL Databases Dead?

3. Peel all the potatoes and cook them together

Peel one potato and cook it. Then peel another and cook it. Doesn’t sound like a recipe for efficiently preparing dinner does it? Turns out it’s also not great for innovating. Peel & prepare many ideas at once, and try to execute them in parallel if you can. That’s what these greats have done.

Related: Why generalists are better at scaling the web

4. Be ok with more failures

This is a tricky one. But Johansson puts in perspective with this key quote:

”Inaction is far worse than failure.”

Viewed that way, our caution about diving into a new idea seems more limiting. True it costs money, time & resources to pursue new ideas, ventures & startups. So be sure to reserve resources. That’s right spend that money & time carefully lest you run out before hitting on the big one.

He also says to be suspicious of low failure rates. In yourself or those you’re evaluating. This probably indicates you’re not risking enough, or trying new things constantly.

Read this: Why Oracle Won’t Kill MySQL

5. break out of your network

Your network is powerful to pursue your career, or following existing well traveled paths. But they can be an obstacle when forging new paths, which is what innovation is all about.

So break away from your networks. One way you can do this is by building a new one. But be sure to surround yourself with diverse cultures, upbringing, backgrounds & expertise.

Also: RDS or MySQL 10 Use Cases

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

How to deploy on Amazon EC2 with Vagrant

vagrant logo

Join 16,000 others and follow Sean Hull on twitter @hullsean.

Why do I want Vagrant?

Vagrant is a really powerful tool for managing virtual machines. If you’re a developer it can make it push-button simple to setup a dev box on your laptop. It manages the images, and uses configuration files to describe specifics of your machines.

In the amazon environment, you can deploy machines just as easily as on your desktop. That’s pretty exciting for those of us already familiar with Vagrant. With that I’ve provided a simple 7 step howto for doing just that!

Also: Are SQL Databases Dead?

1. Use the Mac OS X installer

Fetch your download file here:

Vagrant Installer Downloads

Run the installer. It should do the right thing!

Also: Why Oracle Won’t Kill MySQL

2. Install the vagrant-aws plugin


$ vagrant plugin install vagrant-aws

Also: Bulletproofing MySQL Replication with Checksums

3. Fetch a vagrant box image

Box images vary depending on your “provider” which is vagrant-speak for the environment you’re running in. For aws, they’re some simple json files that tell Vagrant how to work in that environment.

The creator of the plugin has provided a dummy box. Let’s fetch it:


$ vagrant box add dummy https://github.com/mitchellh/vagrant-aws/raw/master/dummy.box

This command is straight out of the readme. What does it do? Take a look:


$ cd /var/root/.vagrant.d/boxes/dummy/aws

$ cat metadata.json
{
"provider": "aws"
}

There’s also the info.json file which looks like this:


$ cat info.json
{"url":"https://github.com/mitchellh/vagrant-aws/raw/master/dummy.box","downloaded_at":"2014-01-14 17:42:33 UTC"}

There’s not a whole lot going on here. If you’re deploying VirtualBox VMs with Vagrant, you’d see a VMware4 disk image. But with Amazon, it stores it’s own AMIs on S3, so Vagrant simply fetches them and runs them for you.

Related: Intro to EC2 Cloud Deployments

4. Configure Vagrantfile

Create a directory to hold your vagrant metadata. This would be the name of your machine:


$ cd /var/root
$ mkdir testaws
$ cd testaws
$ vagrant init

Edit the file as follows:


Vagrant.configure("2") do |config|
# config.vm.box = "sean"

config.vm.provider :aws do |aws, override|
aws.access_key_id = "AAAAIIIIYYYY4444AAAA”
aws.secret_access_key = "c344441LooLLU322223526IabcdeQL12E34At3mm”
aws.keypair_name = "iheavy"

aws.ami = "ami-7747d01e"

override.ssh.username = "ubuntu"
override.ssh.private_key_path = "/var/root/iheavy_aws/pk-XHHHHHMMMAABPEDEFGHOAOJH1QBH5324.pem"
end
end

If you’re familiar with the Amazon command line tools, you’ve probably setup environment variables. Otherwise these may not be familiar to you, so lets go through them:

Your access_key_id and secret_access_key are two pieces of information Amazon uses to identify your instances and bill you. Those are unique to your environment so keep them close to the vest. Here’s how you create them or find them on your aws dashboard.

The keypair_name is your personal SSH key. You may have one on your laptop which you use to access other servers. If so you can upload to the amazon environment. If not you can also use the dashboard to create your own. Whenever you spinup a server, you can instruct amazon to drop that key on the box in the right place. Then you’ll have secure command line access to the box, without password. Great for automation!

Next is your AMI. This is an important choice, as it determines the OS of the machine you’ll spinup, and many other characteristics. You can go with a Amazon Linux AMI but I quite like the Alestic ones from Eric Hammond. Trusted & reliable.

Looking for an ubuntu AMI? Try this ami locator tool.

Check this: 8 Best Practices for Deplying MySQL on AWS

5. Startup the box

Starting an instance once you’ve configured your Vagrantfile is pretty straightforward.


$ vagrant up —-provider=aws

Related: How to autoscale MySQL on Amazon EC2

6. Verify in the Amazon dashboard

Jump over to your amazon dashboard with this link. If you’re logged in already, that will take you to your EC2 instances. You should see a new one, based on the parameters in your Vagrantfile.

Read: Why devops talent is in short supply

7. Login to your Amazon instance

Last but not least, you’ll want to login. Note I’m explicitly specifying my SSH key here. Your path may vary…


$ ssh -i ./iheavy.pem ubuntu@ec2-50-220-50-40.compute-1.amazonaws.com

Also: 5 more things deadly to scalability

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Why cloud computing is the spotify-cation of hosting

dvd collection

Join 16,000 others and follow Sean Hull on twitter @hullsean.

1. Music collections of old

Way way back in the 70’s I remember riding around in a VW beetle. Maybe I’d be driving
with my dad or my uncle. Everybody seemed to own a VW! What everybody also had was a huge collection of 8-track taps in a big box. You’d dig through the box and find what you wanted to play, then pop in the tape. It was exciting because before 8-tracks you only had records, and you couldn’t play those in the car!

But even record collections were new in the 60’s. Before that, most music was consumed live or on the radio.

Also: Why a killer title can make or break your content efforts

2. When books left the library

A similar trend followed for books and reading. Although newspapers have been sold by subscription for a lot longer, books were mostly consumed in libraries. But the consumer itch to build collections eventually built Barnes & Noble into a powerhouse brick and mortar store.

Internet disruption of that business model came too. Enter Amazon’s Kindle. Although you theoretically *buy* digital books, if you read the fine print you’ll see you actually rent them in perpetuity. In fact there have been cases where Amazon has reached into devices and removed previously purchased media.

Related: Why AirBNB didn’t have to fail

3. Managing collections (even stolen ones) is hard work

When you download music or movies, either from iTunes or god forbid grabbing it off of Bittorrent networks, you need to put it somewhere. You’ll store it on your laptop harddrive or if your collection is large enough, on some shared storage system at home. And you’ll also probably never back it up.

The thing is harddrives themselves have a life of about two to four years. As an operations guy I manage data everyday. Backups are a big part of that process, so when the media fails, you won’t lose the collection of movies & music you built lovingly over so many years.

Sadly most people learn the hard way. And when you learn this lesson you probably think, where did all that time go? What did I even *have* in my collection?

Also: Are SQL Databases Dead?

4. Why music & movie theft was just a blip on the historical radar

I’m also a bit of a Doctor Who fan. Since it’s a rather obscure British TV show (or was) I spent some time buying many of the old episodes on DVD. Or I *did* rather, until Netflix starting offering the whole classic collection on subscription. They did this with Star Trek too. Now I have no reason to fish through my shelves for a DVD. Why would I?

As users become more accustomed to the subscription model, they’re less likely to want to build a whole collection of media. This goes well for books, music & videos. Who would bother downloading off of Bittorrents, managing your home collection, and all that trouble when you can just subscribe. Easy. No mess!

Read: Why Oracle Won’t Kill MySQL

5. Subscriptions, subscriptions everywhere!

Whether you managed a datacenter of physical servers in-house, or bought servers managed by a hosting company before the subscription model you had to worry about moving parts. You had to worry about failing harddrives, memory & all the rest.

Then along comes Amazon Web Services and it’s EC2 servers bringing the subscription model to hosting too. This raises the bar on the biggest failing component harddrives, but putting all data on EBS, their virtual storage network. All of this raises the bar for a lot of organizations and reduces the drudgery.

What spotify is doing with music, Netflix is doing for movies & tv shows, and kindle is doing for books. That same trend has brought great disruption to the internet & server hosing. Startups and consumers win big in this game.

Can you think of any businesses where a subscription model might work? They may be ripe for disruption by a new startup.

Check out: Why your startup is failing at Devops

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

What I learned from Ryan Holiday

trust me ryan holiday

Join 16,000 others and follow Sean Hull on twitter @hullsean.

Picked up Ryan Holiday’s Trust Me I’m Lying, Confessions of a Media Manipulator recently. Boy is he good at what he does. This book reads like a howto on free PR & marketing. It also of course serves to protect those who might want to be a bit more informed.

1. Yellow journalism is back

A brief dig into the history of journalism uncovers a festering mess. Before newspapers like NY Times were sold by subscription, most were sold on the street. That meant the echo of a screaming headline had to sell the paper. Sound familiar?

Apparently these yellow papers always had screaming headlines, lots of pictures, “anonymous” sources plus frauds & faked interviews. Anyone seen this before on the interwebs?

Also: When there are conflicts of interest in consulting

2. The medium sets the bar

The nature of the medium sets all the standards, journalistic integrity be damned! Since time is money, the impulse to check facts is damped or in many cases completely absent.

Blogging demands newness. Nothing new, then it must be invented. Take the presidential election campaigns & nobody candidates as prime example.

Also: When you have to take the fall

3. The economics of blogging is horrible

Search engines and readers alike reward newness with their attention. This puts a constant pressure on bloggers to publish even if it’s crap.

”In a pay-per-pageview model, every post is a conflict of interest.” -Ryan Holiday

He quotes Henry Blodget’s formula that writers need to generate 3x their salary in pageview generating ads to break even. Apparently that comes to 1.8m pageviews per month. Wow!

All this drives up the frequency of posts. It also pushes the average length of a post online down to 335 words. Not exactly a medium for thorough analysis.

Read this: Are SQL Databases Dead?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Are SQL Databases Dead?

mesa verde city

I like the image of this city of Mesa Verde. It’s fascinating to see how ancient cities were built, especially as an inhabitant of one of the worlds largest cities today, New York.

I’m a long time relational database guy. I worked at scores of dot-coms in the 90′s as an old-guard Oracle DBA, and pivoted to MySQL into the new century. Would a guy like me who’s seen 20 years of relational database dominance really believe they could be dying?

There’s a lot to be excited about in this new realm of db, and some interesting bigger trends that are pushing things in a new way.

Join 15,100 others and follow Sean Hull on twitter @hullsean.

1. Growing use of ORMs

ORM probably sounds like some strange fossil archeologists just dug up in the ancient city of Mesa Verde. But they’re important. You may know them by their real-life names, Hibernate, Active Record, SQL Alchemy and Cake. There are many others. Object Relational Modelers provide a middleware between developers and the SQL of your chosen relational database. They abstract away the nitty gritty, and encapsulate it into a library.

In a way they’re like code generators. Mark Winand talks about them in SQL Performance Explained warning of the “eager fetching” problem. This is DBA speak for specifying all columns (SELECT *) or fetching all rows, when you don’t need them all. It’s inefficient in terms of asking the database to read & cache all that data, but also to send it across the network and then discard it on the webserver side. Like a lazy housekeeper the clutter & dust will grow to overwhelm you.

Martin Fowler is the author of the great book NoSQL Distilled. He tries to walk the fence in his post ORM Hate, trying to balance developers love of ORMs, and the obvious need for scalability. Ted Neward calls ORMs the Vietnam of Computer Science. Mattias Geniar points out that BAD ORMs are infinitely worse than bad SQL and another on High Scalability by Drewsky ”>The Case Against ORM Frameworks.

If you agree the ORM conversation is still a huge mess, you’ll be excited to know that NoSQL sidesteps it completely. They’re built out of the box to interface more like data structures, than reading rows and columns. So you eliminate the scalability problems they introduce when you go NoSQL. That makes developers happy, and pleases DBAs and techops too. Win!

Read: Why Oracle won’t kill MySQL

2. Widening field of options

NoSQL databases are not simply key value stores, though some like Memcache and Riak do fit that mold.

Mongodb offers configurable consistency & durability & the advantages of document storage, no need for an ORM here. You also have a mix of indexing options, that go a little deeper than other NoSQL solutions. A sort of middle ground solution that offers the best of both worlds.

Cassandra, a powerful db that is clustered out of the box. All nodes are writeable, and there are various ways to handle conflict resolution to suit your needs. Cassandra can grow big, and naturally takes advantage of cloud nodes. It also has a nice feature to naturally age out data, based on settings you control. No more monumental archiving jobs.

Hbase is the database part of Hadoop, based on Google’s seminal Bigtable paper.

Redis is another option with growing popularity. It’s a key-value store, but allowing more complex data in it’s buckets, such as hashes, lists, sets and sorted sets. Developers should be salivating at this one.

Also: 5 Great Things about Markus Winand’s Book SQL Performance Explained

3. Lowering bar

The old world of relational databases treat data as sacrosanct. DBAs are tasked with protecting it’s integrity & consistency. They manage backups to protect against disaster. In this world, every bit of data written is as sacred as any other, whether it’s your bank account balance, or a comment added to a facebook discussion.

But modern non-relational databases introduce the idea of eventually consistent. DBAs and architects would say we are relaxing our durability requirements. What they mean is data can get slightly out of sync and we’re ok with that. We’ll build our web applications to plan for that, or even in the case of Riak expose the levers of durability directly to the developers, allowing them to make some changes instant, while others more lax and lazy.

Check this: Why high availability is so very hard to deliver

4. Cloud demands

Virtualized environments like Amazon EC2, give easy access to legions of servers. Availability zones & regions only widen the deployment options. So deploying a single writeable master, the way traditional relational databases work best, is not natural.

Databases like Cassandra, Mongo & Redis are clustered right out of the box. They grew up in this virtual datacenter environment and feel comfortable there.

Related: Why I wrote the book on Oracle & Open Source

5. Only DBAs understand them

Devs may whine at this statement, and to be fair it’s a generalization. The popularity of ORMs speaks volumes here. Anything to eliminate the dreaded SQL writing. Meanwhile DBAs bemoan the use of ORMs for they represent everything they’re trying to fix.

SQL is hard enough, but the ugly truth is each database vendor has their own implementation, their own optimizations, their own optimal tweaks. Even between database versions, SQL code may not perform consistently.

Identifying slow SQL and tweaking it remains one of the primary tasks of performance tuning, for this reason. It hasn’t changed much in my two decades on the job.

Also: Why bemoaning AWS performance sounds like Linux detractors circa 1999

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Scalable Startups is back – happy holidays!

CC license via barrettward

CC license via barrettward

Join 15,100 others and follow Sean Hull on twitter @hullsean.

After some intermittent outages over the past weeks, we managed to get the site rebuilt on a brand new server. Yes we’re still hosted at a traditional datacenter, 1and1.com to be exact. They were very helpful through the recovery process.

Here are five lessons learned.

1. Keep various types of backups

While the site was down, I was scrambling to consider what I had lost and what would be the fastest way to get the site back online. First there is the content, images and database backup. But wordpress also has an export function, so an additional XML backup of all site content proved invaluable. Further you have the software, the content management system of wordpress itself. Yes you can download a new copy, but what about the plugins, and header modifications to support Google Analytics?

Read: Why Oracle won’t kill MySQL

2. Be patient with the support staff

Eventually you’ll need to get on a call with the support techs. They may try to help, but speak of worse case scenarious, and ask if you have a backup, like you’re really going to need it! Remain calm, and ask for clarification.

In my case, the mirrored root drives looked unsalvageable. Turns out that although the md volumes – Linux’s software RAID – could not be rebuilt, the good drive contained most of the data I needed to restore. In that case a filesystem check fixed the journals and brought back a copy of data.

Related: Why Affordable Care Act desperately needs Techops

3. Keep detailed notes of all your components

When you’re scrambling, and frustrated, notes save you. There are way more moving parts than you can keep all in your head, so why not keep good notes too?

For my site, there is wordpress, the version, MySQL and it’s version, config files for Apache & MySQL, dump files of individual schemas, existing installed plugins, themes, mods, google analytics configuration, passwords for the server, and wordpress itself. And don’t forget the logins to your hosting dashboard, PINs and secret identifiers. Write it all down!

Also: 5 Great Things about Markus Winand’s Book SQL Performance Explained

4. Monitor and test more

As it turns out I wasn’t hacked this time around, but was having a disk failure. This was manifesting in a strange way. Software RAID needs to be monitored, as does the syslog & mysql & apache logs.

Also, always be testing. I found that although I was using pingdom to notify me of outages, it would not *repeat* that notification. Since I had an extended outage of many days off and on, I had no idea of this. Pingdom had sent me one notification which I missed, and none to follow.

Check this: Why high availability is so very hard to deliver

5. Don’t forget the holidays!

Yes an outage is serious, but keep a sense of perspective. Have a splash page sitting at the ready, while you’re fiddling with all the components.

Although your digital billboard is down, your audience and customers are probably more patient than you realize. If you provide valued content, they’ll be back!

Read: How to do an architecture assessment for a merger

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

How to do technical assessment for a merger acquisition

tsmitty11 big eats small fish

Join 15,000 others and follow Sean Hull on twitter @hullsean.

My newsletter goes out the first few days of every month. I invite everyone I’ve ever worked with, so many of my present and former colleagues & clients receive it. I often receive a few emails in the days following, with requests for help, advice & expertise.

Recently I was referred on a merger acquisition project. The project involved evaluating the technology of the target company, to understand how it would fit in with existing infrastructure, and what challenges they might face.

1. What’s your tech stack?

The scope of the project included evaluating existing application code. We looking closely at:

o What programming languages & versions were in use?
o What volume of code was there, when was it built, and how was it maintained?
o Was the code well commented & organized?
o Are those languages or technologies popular in the marketplace today?
o Are the foundation technologies still supported, either commercially or by open source communities?

Related: When you’re hired to solve a people problem

2. What’s your team stack?

One of the points that came up during the discovery phase was subject matter expertise. We found that the acquiring company’s DNA was around Windows, SQL Server & IIS. Their applications had been developed mostly on C#, so their team had skills around that stack.

The target company, on the other hand was Oracle on Unix. Here the team & expertise had a different heritage.

These two stacks have very different people & cultures behind them. Those would likely introduce unique challenges were you to merge those two firms, as the former would not have skills and expertise to manage & maintain the latter. Trimming teams down, or consolidating hardware & components would likely prove challenging.

Also: A CTO must never do this

3. Where are your pain points

We also evaluated current problem areas in the target application.

o Where were team members struggling most?
o What type of performance problems existed?
o Was there data mismatch or redundancy?
o Were business units struggling in some area to report on the data they needed?

Read this: When you have to take the fall

4. Style of software development

In software development, the traditional building model is called waterfall. It involves specing out requirements, spending a long period writing code, and then releasing it all at once to be tested & deployed. Typically during development period, there isn’t a working version of the system, as it’s undergoing change.

The risk with waterfall is that what comes out the other end won’t be what feature specification teams envisioned. Worse still is when the resulting product is full of bugs, or has major performance problems. The ACA used this method to develop healthcare.gov website, resulting in a whole host of problems on launch. It’s why I’ve advocated for real techops at Healthcare.gov.

These days, the modern and arguably superior model is called agile software development. This involves writing code in much much smaller chunks, and for each releasing a small test to verify that it performs it’s function. These unit tests allow for software to be continuously deployed, perhaps multiple times per day.

Check this: How we do a performance review

5. Legacy or Open Source

A last point of evaluation is the use of legacy software components such as Oracle, versus the more nimble and open source components many internet firms use, such as Linux, Apache, MySQL & PHP or Python. These later components make the stack of many modern web facing applications, are supported by the internet community, and provide flexibility & configurability in the cloud.

Fred Wilson has advocated for Open Source in various postings on his A VC blog. Although these technologies offer great opportunity & strength, your team may not have experience bushwacking through the DIY world of open source and that would be a major consideration for a merger.

DNA and culture of the acquiring company, and the target company have a huge impact on whether those technologies will all play well together as the firm grows up.

Also: A sample executive summary we did for Acme Startup, Inc.

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

5 great things about Markus Winand’s book SQL Performance Explained

markus winand sql performance explained book

Join 12,100 others and follow Sean Hull on twitter @hullsean.

1. Covers databases broadly

You may not have noticed, but there’s a whole spectrum of relational databases on offer. Of course in the database world, most get infatuated with one, and that becomes their bread & butter before long. Their life, their passion, their devotion.

That’s fine as far as it goes, but Winand really stands out, offering a spectrum of ideas and optimization techniques for different platforms. If you’re an Oracle-only or MySQL-only dba you’ll gain a lot from this book but even more importantly if you work in professional services, and need to communicate with DBAs brought up on one of these platforms, it becomes like a rosetta stone for SQL query tuning.

Read: Why devops talent is in short supply

2. Shows you how to fish

I find database books and methods fall into two sort of broad categories. There’s the call Oracle support method, where you’ll be handed one very specific set of steps, commands, and a path to solve each specific problem. It’s more about memorization, it’s like they actually hand you the fish.

Then there is the investigative method, where you learn how to use a magnifying glass to look at fingerprints, and check for DNA samples, and interrogate suspects. You know learn the tools of the trade.

That’s what Markus brings you, in all it’s delicious glory.

Read: Why high availability is so very hard to deliver

3. It’s concise

Another gripe I have of technical books is that the publishing model is, this is a textbook and since the price tag is large, let’s make the physical book large! Of course no one wants to carry those books around. I even recently bought a kindle to solve this problem.

SQL Performance Explained is more a paperback book form factor, and that means you can tote it around with you easily, and keep it with you at work. Read it on the train, commuting to work.

200 pages packed cover to cover with all sorts of good chapters, including a primer on indexes & types, scalability & performance, joins, clustering, Top-N queries, DML, and more.

Read this: Why a four letter word divides dev and ops

4. It’s technical but accessible

If you’re a real rock bottom beginner, you might want to dig a bit more on your SQL syntax, and some of the basics. You could also keep a 101 book side-by-side, while you’re reading this book.

For the intermediate & advanced DBAs out there, this book will sit comfortably in your paws as you flip the pages and learn something new. For instance just today I learned that Postgres can index NULLs while MySQL, Oracle and SQL*Server cannot. Learn something new everyday.

Related: MySQL interview guide for managers and candidates alike

5. Gives you answers you can use today

After twenty years of consulting, I’ve seen a few patterns emerge. Besides the spectrum of team & communication challenges, firms hitting the performance wall often have issues with their relational databases.

Yes those databases are sometimes on the wrong hardware, or their are other obscure problems with setup or configuration. But the bulk of issues center on badly written SQL.

SQL is a much reviled language and often misunderstood. And it doesn’t seem like developers have gotten that much better at it over the years. It would explain the rise of NoSQL databases, as they often speak REST or xml, no need for pesky sequel.

One parting note. For all the devs and architects out there, who want to sing the virtues of ORMs, this book hits that squarely in the nose. By showing how differently each relational database implements SQL, performs work, and optimizes, Winand also illustrates the naivete behind trying to write database independent application code.

If you’re a developer and don’t know how to profile a query or run explain plan, don’t walk, run to your closest Amazon.com store and get this book!

Also: 5 more things deadly to scalability

Criticisms

If I were to offer two slight criticisms, it would be these. First, the index is a bit wonky. When I look under “P” for example, there’s no Postgres, while one quarter of the book is obviously devoted to that platform. Further, looking up NULL which are covered in depth, in various places in the book, only has one entry in the index, p54 on Oracle. So the index could be a bit more robust to be useful.

The other criticism is more perhaps my bias. On page 96, when he discusses ORMs I thought he was rather… shall we say gentle. Although he clearly states that “eager fetching” is problematic, I don’t think he goes far enough to condemn it. In my experience ORMs are always trouble.

Then again why am I complaining, their use keeps me forever employed.

Want a copy? Markus Winand’s book site has all the goods!.

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters