Category Archives: All

Wrestling with bears or how I tamed Tungsten replicator

tungsten replicator

I just dove into Tungsten replicator very recently as I need to replicate from Amazon RDS to Redshift. I’d heard a lot of great things about Tungsten, but had yet to really dig my heels in.

Join 28,000 others and follow Sean Hull on twitter @hullsean.

I fetched the binary and began to dig through the docs. Within a day I felt like I was sinking in quicksand. Why was this thing so darn complicated? To my mind unix software is a config file simple.cfg, a logfile simple.log, a daemon simpled. Open some ports & voila you’re cooking.

Unfortunately for beginners Tungsten took a very different approach. Although they support .ini files to config, they seem to encourage these huge commands to “configure” which means generate a config file in Tungsten speak, and install which literally means install the software for you into /opt/continuent.

After various posts to the forums, and a lot of head scratching, I discovered the cookbook. At first I thought this referenced puppet or chef type cookbooks. But it’s something different. It allowed me to setup a very basic tungsten just to see things working.

Tungsten supports all sorts of “topologies”. For this example I am just doing master-slave. There are two nodes and each has mysql running on it. Node1 serves as the master, and runs the tungsten replicator (master node service). And node2 serves as the slave and runs the tungsten applier (slave node service).

Good luck and hope this helps others speedup the learning curve!

1. Download tarball (on master)

Note the forums indicated they may be moving off of google code. So this url may change.


$ cd /tmp
$ wget http://downloads.tungsten-replicator.org/download.php?file=tungsten-replicator-oss-4.0.0-18.tar.gz

Also: Why Airbnb didn’t have to fail

2. Expand the tarball (on master)

Use your vast unix skills to expand the tarball!


$ mv download.php\?file\=tungsten-replicator-oss-4.0.0-18.tar.gz tungsten.tgz
$ tar xvzf tungsten.tgz
$ mv tungsten-replicator-4.0.0-18 stage

Also: Is the difference between dev & ops a four-letter word?

3. Install MySQL (each box)

Hopefully you’ve done this before. Pretty straightforward:


$ apt-get install mysql-client mysql-server

Also: Are SQL Databases Dead?

4. Edit cookbook files (on master)

Inside /tmp/stage/cookbook you’ll need to make a few simple edits:

edit COMMON_NODES.sh

Edit NODE1 and NODE2 and comment out other lines.


export NODE1=ip-172-31-0-117
export NODE2=ip-172-31-1-188

edit NODES_MASTER_SLAVE.sh


export MASTERS=($NODE1)
export SLAVES=($NODE2)

USER_VALUES.sh


export TUNGSTEN_BASE=/opt/continuent
export MY_CNF=/etc/mysql/my.cnf
export DATABASE_USER=sync
export DATABASE_SUPER_USER=root
export DATABASE_PASSWORD=

Also: 5 Things toxic to scalability

5. Create install directory (each box)

The /tmp/stage directory you created above is just a holding ground for the tarball. Continuent in it’s infinite wisdom wants to install itself. So, create a directory for it:

As root:


$ mkdir /opt/continuent
$ chown tungsten /opt/continuent

Then as tungsten:


$ touch /opt/continuent/testfile.txt

Also: How to deploy on Amazon EC2 with Vagrant?

6. Configure aws security groups

AWS security group permissions are key to getting any of this tungsten stuff working. And there are a lot of moving parts here. Ping is required for various tests, as is MySQL’s port. But you’ll also need to enable Tungsten History Log port 2112 and the replicator ports.

o ping ICMP from your group
o enable inbound 3306 from your group
o enable THL – inbound 2112 from your group
o enable RMI – inbound 10000, 10001 from your group

Also: Howto interview an AWS expert for managers, recruiters & engineers alike

7. Test mysql client from the other (each box)


$ mysql -h ip-172-31-1-188 -u root

$ mysql -h ip-172-31-0-117 -u root

If these hang, verify 3306 in your aws security groups. If you get a mysql error, be sure the “host” is appropriate. Check with:


mysql> select user, host, password from mysql.user where user = 'root';

For more info see Connect to MySQL in the Amazon public cloud.

Also: 5 Reasons to move to amazon redshift

8. Setup ssh auto-login (each box)

You need to be able to login to each box as a user “tungsten” without a password.


$ adduser tungsten
$ ssh-keygen
$ scp .ssh/id_rsa.pub tungsten@ip-172-31-1-188:/home/tungsten/.ssh/authorized_keys

Be sure the authorized_keys file is 600:


$ chmod 600 .ssh/authorized_keys

Also: Did MySQL & Mongo have a beautiful baby called Aurora

9. Install ruby (each box)

You need to have ruby on both boxes.


$ apt-get install ruby

Also: Top interview questions for a MySQL expert – recruiters, managers & candidates

10. Create sync user (each box)

Both of your mysql instances will need a user that tungsten connects as. I called it “sync”.


mysql> create user 'sync'@'%' identified by 'secret';
mysql> grant all privileges on *.* to 'sync'@'%';
mysql> flush privileges;

Note, you may want to use a blank password at this stage, to eliminate that as a potential problem to debug. Also consider the security implications of ‘%’ and consider a subnet wildcard such as ’10.0.%’.

Also: Myth of five nines why high availability is overrated

11. Enable binary log (each box)

Enable the mysql binary log with log_bin parameter. Also ensure that /var/lib/mysql is readable by the tungsten user, either by changing /var/lib to 655 or adding tungsten to the mysql group. Test this as well using less or cat.

Also: What is high availability and why is it important?

12. Update MySQL startup settings (each box)

Fire up your favorite editor and update /etc/mysql/my.cnf settings (both servers). The following are the main ones:


server_id = 1 (server_id=2 for slave)
sync_binlog = 1
max_allowed_packet = 52m
open_files_limit = 65535
innodb_log_file_size=50m
#bind_address localhost (comment it to disable)

Note that changing innodb_log_file_size is tricky. You’ll need to stop mysql, rename old ib_logfile0 to ib_logfile0.old and ib_logfile1 to ib_logfile1.old. Then change the param in my.cnf and start mysql. Otherwise you’ll get errors on startup.

Also: Is the difference between dev & ops a four-letter word?

13. Run the installer (on master)

This step is fairly straightforward. If there are problems in this step, you’ll see ERROR lines in the output. Sift through them and resolve one by one.


$ ./cookbook/install_master_slave

Also: RDS or MySQL – Ten use cases

14. Check tungsten status (each box)

The “trepctl” utility allows you to check the current status. It does a lot more, but for now that’s enough. If you want to make it easier add “/opt/continuent/tungsten/tungsten-replicator/bin” to your path.

Also notice the line “state”. It should be ONLINE. If it says “GOING-ONLINE:SYNCHRONIZING” that likely means you didn’t open up the tungsten ports. You’ll need both RMI ports 10000 and 10001 as well as THL ports 2112. We all know how finicky AWS security groups can be. I’ll leave it to your own exercise to confirm those are open.

ON MASTER


root@ip-172-31-0-117:/opt/continuent/tungsten/tungsten-replicator/bin# ./trepctl status
Processing status command...
NAME VALUE
---- -----
appliedLastEventId : mysqld-bin.000005:0000000000000321;235
appliedLastSeqno : 5
appliedLatency : 34824.086
autoRecoveryEnabled : false
autoRecoveryTotal : 0
channels : 1
clusterName : cookbook
currentEventId : mysqld-bin.000005:0000000000000321
currentTimeMillis : 1432840323683
dataServerHost : ip-172-31-0-117
extensions :
host : ip-172-31-0-117
latestEpochNumber : 0
masterConnectUri : thl://localhost:/
masterListenUri : thl://ip-172-31-0-117:2112/
maximumStoredSeqNo : 5
minimumStoredSeqNo : 0
offlineRequests : NONE
pendingError : NONE
pendingErrorCode : NONE
pendingErrorEventId : NONE
pendingErrorSeqno : -1
pendingExceptionMessage: NONE
pipelineSource : /var/lib/mysql
relativeLatency : 44450.683
resourcePrecedence : 99
rmiPort : 10000
role : master
seqnoType : java.lang.Long
serviceName : cookbook
serviceType : local
simpleServiceName : cookbook
siteName : default
sourceId : ip-172-31-0-117
state : ONLINE
timeInStateSeconds : 82860.917
timezone : GMT
transitioningTo :
uptimeSeconds : 82861.492
useSSLConnection : false
version : Tungsten Replicator 4.0.0 build 18
Finished status command...
root@ip-172-31-0-117:/opt/continuent/tungsten/tungsten-replicator/bin#

ON SLAVE


tungsten@ip-172-31-1-188:/opt/continuent/tungsten/tungsten-replicator/bin$ ./trepctl status
Processing status command...
NAME VALUE
---- -----
appliedLastEventId : mysqld-bin.000005:0000000000000321;235
appliedLastSeqno : 5
appliedLatency : 42569.62
autoRecoveryEnabled : false
autoRecoveryTotal : 0
channels : 1
clusterName : cookbook
currentEventId : NONE
currentTimeMillis : 1432840348936
dataServerHost : ip-172-31-1-188
extensions :
host : ip-172-31-1-188
latestEpochNumber : 0
masterConnectUri : thl://ip-172-31-0-117:2112/
masterListenUri : thl://ip-172-31-1-188:2112/
maximumStoredSeqNo : 5
minimumStoredSeqNo : 0
offlineRequests : NONE
pendingError : NONE
pendingErrorCode : NONE
pendingErrorEventId : NONE
pendingErrorSeqno : -1
pendingExceptionMessage: NONE
pipelineSource : thl://ip-172-31-0-117:2112/
relativeLatency : 44475.936
resourcePrecedence : 99
rmiPort : 10000
role : slave
seqnoType : java.lang.Long
serviceName : cookbook
serviceType : local
simpleServiceName : cookbook
siteName : default
sourceId : ip-172-31-1-188
state : ONLINE
timeInStateSeconds : 1906.6
timezone : GMT
transitioningTo :
uptimeSeconds : 82882.334
useSSLConnection : false
version : Tungsten Replicator 4.0.0 build 18
Finished status command...
tungsten@ip-172-31-1-188:/opt/continuent/tungsten/tungsten-replicator/bin$

Also: Is zero downtime even possible with Amazon RDS?

15. Perform simple test

Create a table on the master mysql node.


mysql master> create database test;
mysql master> create table test.sean (c1 varchar(64));
mysql master> insert into test.sean values ('hi there');
mysql master> insert into test.sean values ('this should show up in tungsten thl file');
mysql master> insert into test.sean values ('new data on tungsten02 THL??');

Verify that the table is on the slave mysql node.


mysql slave> show databases;
mysql slave> select * from test.sean;
+------------------------------------------+
| c1 |
+------------------------------------------+
| hi there |
| this should show up in tungsten thl file |
| new data on tungsten02 THL?? |
+------------------------------------------+
3 rows in set (0.00 sec)

Success!

Also: Why are MySQL & other database experts so hard to find?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Are we fast approaching cloud-mageddon ?

storm coming

One look at StackShare’s trending technologies, and you’ll discover the exploding growth of languages, webservers, load balancers, databases, caching servers, automation & monitoring tools, continuous integration suites & a broad spectrum of Software as a service solutions.

The choices today boggles the mind. Choice is good, but too much choice can mean trouble too.

Join 30,000 others and follow Sean Hull on twitter @hullsean.

1. What am I actually using?

Erich Schubert wrote a superb piece about the sad state of the sysadmin in the age of containers. Here’s what caught my eye:

Stack is the new term for “I have no idea what I’m actually using”.

That definitely rings true for me. The customers I’m seeing these days have such complicated stacks, that nobody really knows what’s installed. That’s dangerous!

Also: Do today’s startups assemble at their own risk?

2. Embrace failure more broadly

Recently I wrote a blog post asking Is AWS the patient that needs constant medication?. It got a lot of traction, and here’s why I think that happened.

AWS uses very commodity, cheapo components. The assumption is, with an infinitely redundant datacenter, component failure is ok. It’s ordinary & everyday.

Unfortunately most startups, even ones that employ some Ansible & devops, still don’t have Netflix grade automation.

Those regularly everyday failures are still getting detected by old-school manual monitoring. And that’s a recipe for trouble

Also: 5 Things toxic to scalability

3. What are complex systems?

In this excellent deck, James Urquhart talks about emergent behavior in complex systems. It’s worth a quick read.

***

Read: How I find entrepreneurial focus

4. What to do? Do you like boring?

Dan McKinley formerly principal engineer at Etsy & now with Stripe wrote a brilliant essay arguing for boring technology.

This comes as a shock to many in the startup world. It sort of smacks in the face of open source, or does it?

I worked in the enterprise space as an Oracle DBA for a decade starting in the mid-nineties. Among DBAs there was always a chuckle when a new version of Oracle came out. No one wanted to touch it with a ten foot pole. Sure we’d install it on test boxes, start learning the new features and so forth. But deploy it? No way, wait a good 2 or even 3 years before upgrading.

Meanwhile management was eager for the latest software. Don’t we want the newest? The Oracle sales guys would be selling the virtues of all sorts of features that nobody needed right away anyway.

Choosing boring components takes discipline to fight sexy new technologies & bleeding edge versions. But staid & stodgy wins you everyday in operations uptime.

Related: Is automation killing old-school operations?

5. Use tried & tested components

Do you find your application or stack contains java, ruby, python & PHP? Choose one.

One webserver like nginx, one caching server like memcache or redis, one search server like solr or elasticsearch, one database like MySQL or postgres. Standardize all your components on one image, so you can use that for all your servers, regardless of which you use.

Fewer components will mean fewer interdependencies, less maintenance, & less chaos.

Also: What’s the luckiest thing that’s happened in your career?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

What happens when entrepreneurs treat data as a product?

Sneachta Pix

Sneachta Pix

I’ve been reading DJ Patil’s thoughts on building data products. As the chief data scientist of the united states, he knows a thing or two.

Join 30,000 others and follow Sean Hull on twitter @hullsean.

I also attended a recent Look & Tell event, where Lincoln Ritter talked about Data Democracy at Animoto. He expressed many of Patil’s lessons.

I took away a few key lessons from these that seem to be repeating refrains…

1. UX of data

UX design involves looking at how customers actually use a product in the real world. What parts of the product work for them, how they flow through that product and so on.

That same design sense can be applied to data. At high level that means exposing data in a measured, meaningful & authoritative way. Not all the tables & all the data points but rather key ones that help the business make decisions. Then layering on top discovery tools like Looker to allow the biz-ops to make more informed decisions.

Also: 5 Reasons to move data to Amazon Redshift

2. Be iterative

Clean data, presented to business operations in a meaningful way, allows them to explore the data, and find useful trends. What’s more with good discovery tools, biz-ops is empowered to do their own reporting.

All this reduces the need to go to engineering for each report. It reduces friction and facilitates faster iteration. That’s agile!

Related: Is automation killing old-school operations?

3. Be authoritative

Handing the keys to the data kingdom over to business means more eyes on the prize. That may well surface data inconsistencies. Each such case can reduce trust on your data.

Being authoritative means building checks into your data feeds, and identifying where data is amiss. Then fixing it at the source.

Read: Are SQL Databases dead?

4. Spot checks & balances

Spot checks on data are like unit tests on code. They keep you honest. Those rules for how your business works, and what your data should look like, can be captured in code, then applied as tests against source data.

Also: Is Apple betting against big data?

5. Monitoring for data outages

As data is treated as a product, it should be monitored just like other production systems. A data inconsistency or failed spot check then becomes an “outage”. By taking these very seriously, and fire fighting just as you do other production systems, you can build trust in that data, as those fires become less frequent.

Also: Why Airbnb didn’t have to fail

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

How I find entrepreneurial focus

Brian K

Relentless focus. This is surely a key to entrepreneurial success. But how to find it? And how to maintain that focus through the ups & downs?

Join 28,000 others and follow Sean Hull on twitter @hullsean.

I’ve found a simple system with a few rules has brought me success. I’ve used it for two decades of as entrepreneurial, and still do.

Here’s how it works…

1. keep a list of small tasks

This is the number one thing I do daily. I start out early in the morning, over coffee. Anything that “needs to get done” goes on the todo list, but I also separate out the things I’ll do today.

Todo list items are not big ones, like “save the world” or “get new job”. They are small nuggets of work that take 15 to 60 minutes. If they’re larger, they need to be broken down.

A typical day covers five major tasks. You will get sidetracked. You’ll need to answer calls & emails that aren’t captured on the list.

All this sounds simple, but it actually requires a discipline. Both to keep tasks actionable & small. You’ll learn your work pace with practice. But there’s one more thing to remember.

Also: Are generalists better at scaling the web?

2. trust the list

One thing I find myself doing is pushing just because something is on the list.

I don’t do list based on feelings or moods

This requires a lot of habit building, but it becomes valuable. By doing this over time, you begin to trust the list. You know things on it will get done. So you can safely “add it to the list” and forget about it for the moment.

This lets your mind relax and bcomes a real godsend when you have a mountain of work to do.

Just work list because it’s there. And trust that things that need to get done simply go on the list.

Related: Why airbnb didn’t have to fail

3. done with list, done for day

On days where things get hairy, and you work more, you’ll have to slog through to get everything on the list done. But sticking to it will build a habit that’s valuable.

At the same time some days will be easier. Avoid the temptation to add more work to fill the day.

When you’re done with the list, you’re done for the day

This is a discipline too. Pat yourself on the back, and give yourself a break. You did what you said you would do. Time for a beer. :)

Read: Which tech do startups use most?

4. big projects require faith

Anothe lesson I’ve learned is that really large projects, or ones bringing you into new areas, require a lot of faith. For me, with an engineering background, I don’t have an easy time finding that. I want to measure, and dice up everything from the start.

When I was embarking on a project to buy real estate in Brooklyn, I really learned this lesson. There was so much unknown. How do I work with real estate brokers that have a different style of communicating than engineers & corporate professionals? How do I negotiate? What’s the right price? What about mortgages & their brokers? Architect inspections, land surveys, flood zones, crime maps, loans & assets, legal & closing costs. The list of unknown & nebulous areas of expertise was staggering.

Hang around edges to get lay of land

If I would distill this faith idea down for someone embarking on a new career or diving into a pool of unknown depth, I would say start by hanging around the edges. Pick off pieces that you can, and add them to the todo list.

I went to open houses. I asked questions. I researched online, and I always made sure I understood how agents & players were incentivized. That means believing none of the words people speak, but rather, look behind the curtain and make educated guesses about those realities.

Also: Do todays startups assemble at their own risk?

5. break down & do

It is inevitable that you will experience writers block. Or any other kind of block that manifests as procrastination. Don’t over think it.

Continue to break down to smallest viable unit that you *can* do.

get started on anything to get inertia

With writing I find blocks where I don’t have a solid idea formulated. Maybe you have a topic? So then my todo list for the day is “write five bullet points”. This by itself will take some time, but you know you can write something. By moving past your block, I sometimes find I wanna keep writing and finish the piece. This is the kind of habit you want to form.

Also: Is Fred Wilson right to say speed is a feature?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

5 Things I learned from Fred Wilson & Mark Suster

I was recently flipping through AVC.com and saw this interview by Mark Suster of Upfront Ventures. He talks in depth with veteran in the VC world, Fred Wilson.

Join 30,000 others and follow Sean Hull on twitter @hullsean.

Here’s a more in-depth blog post on Mark’s interview with Fred Wilson from Union Square Ventures.

1. I’m not into debt

Around the 20:25 in the interview, Fred is discussing a period in his career before some of his first big investments, where things were financially challenging. He makes a rather candid comment about personal debt:

“I’m not that kinda person. I don’t like debt. I’m not into debt”

I think this is key. I also think it frames the whole way people approach business & career.

Also: Are generalists better at scaling the web?

2. Brains & hustle is key

Among the most successful entrepreneurs there are certainly many who are very intellectually astute. Meanwhile there are others who are great speakers, who can sell an idea, and persuade, but perhaps not as deep product wise or deeply technical.

The very best though, tend to have both brains & hustle.

Related: 8 questions to ask an AWS expert

3. Best technology doesn’t win markets

Around 11:45 in the interview, Mark & Fred are discussing Novell & Banyan.

“That was when I learned that best technology doesn’t win markets”

t’s interesting because as you hear the story of how Banyan lost out to Novell, it resonates today with companies that often have the best tech, but don’t win in markets. Interesting.

Read: Why Airbnb didn’t have to fail

4. Find answers through blogging

“It’s like Venus Fly Paper. When I write about topics that are relevant, suddenly anybody with a startup solution in that field will approach us. This works brilliantly.”

Indeed, I’ve found blogging to be crucial myself to career building. It helps in a myriad of ways.

Blogging brings visibility, as your blog gains in popularity. That is certainly big. But also it helps you craft & formalize your voice & your vision. Blogging asks you everyday to think about your perspective, and share it in a way that appeals to a broad audience. And analytics give you real feedback that you are saying something of value to people.

Also: Are SQL databases dead?

5. Listen to the younger generation

Around 1:11:15 in the interview, there’s an interesting point where Mark asks Fred if there were any deals that they regret not getting into. Fred responds that AirBNB was such a deal, as it was a quintessential Union Square ventures company.

As it turns out they didn’t invest because they couldn’t imagine using the service. Meanwhile the younger members of their team had a different perspective.

“We’re not gonna reject anything that we wouldn’t do and the younger team would.”

Interesting point. I think of Venmo as another example of this. I personally wouldn’t use the service, meanwhile it is clearly very popular among teen & twenty something demographic.

Also: 5 Things toxic to scalability

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

What’s the luckiest thing that’s happened in your career?

sashi [via Flickr]

I was browsing through Career Dean recently, a site that facilitates professionals to share knowledge & experience with more junior & recent college grads about the work world. It’s a great site. I saw the question What’s the luckiest thing that’s ever happened for your career?

I read in John Adam’s AMA his “million dollar piss” (www.careerdean.com/q/howd-you-get-the-job-twitter), which he sowed the seeds of his success basically during a piss. That’s a 1 in a million kind of story I know. I’d like to hear if anyone else has ever experienced anything remotely lucky in that way? =) something fun to come back and read if anyone answers.

Join 28,000 others and follow Sean Hull on twitter @hullsean.

Here’s how I responded…

I moved to NYC & worked at a tiny startup in the mid-nineties. Got to do Mac stuff, windows & Sun Solaris unix as well. Jumped on an Oracle project where I was a bit underwater. The firm hired a consultant to assist me for a few days. I watched what he did and learned like a sponge. Within a few months I dove into Oracle consulting and never looked back.

I felt this was an amazingly lucky opportunity to for a few reasons.

1. DIY

I’ve been consulting for almost twenty years now. And I get asked all the time how to get into freelance or independent consulting. For me the jumping off point was working for a really small ten person startup.

An environment like this is very different from a large corporation where you do one thing. At a tiny shop, everything is very do-it-yourself. You have to be self-serve & lean. It’s a constant challenge to teach yourself what you don’t already know. It’s a very vibrant environment as you enter your career.

Also: 5 Things toxic to scalability

2. Generalist

I also found that I had the chance to really apply everything I learned in computer science. It’s a hardware problem? It’s a software problem? These kind of silos that you experience at university don’t apply. One day you can be doing windows, mac, or Unix operating system configuration, the next you can be writing code. And on the third day you can be doing dba work.

In today’s terminology, this role was site reliability engineer or SRE, fullstack developer, tech support, evangelist, CTO, DBA, scalability & performance lead and more.

Related: Are generalists better at scaling the web?

3. Cutting edge

Startups to be sure are on the bleeding edge. They’re constrained by budgets, and through sheer will & experimentation, are cutting their teeth on the newest technologies out there.

These days that might be Cassandra & Kafka, Docker, MongoDB, hdfs, Redshift and so on.

Read: Do managers underestimate operational cost?

4. Ok to Fail

In larger enterprises, a lot of politics weigh on decisions, and exotic technologies are risky. When you’re at a startup, and by design you are entering uncharted waters, it’s sort of a given that it is ok to fail. This encourages learning, as there is less risk of failure.

Also: Is the difference between dev & ops a four-letter word?

5. Iterative & Agile

We talk about being agile, and lean at startups. At a very small place like this, you have one or two developers, and you deploy code constantly. It’s agile by default. And that’s a good thing.

Also: Is high availability overrated? The myth of five nines.

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Is AWS the patient that needs constant medication?

storm coming

I was just reading High Scalability about Why Swiftype moved off Amazon EC2 to Softlayer and saw great wins!

We’ve all heard by now how awesome the cloud is. Spinup infrastructure instantly. Just add water! No up front costs! Autoscale to meet seasonal application demands!

But less well known or even understood by most engineering teams are the seasonal weather patterns of the cloud environment itself!

Join 28,000 others and follow Sean Hull on twitter @hullsean.

Sure there are firms like Netflix, who have turned the fickle cloud into one of virtues & reliability. But most of the firms I work with everyday, have moved to Amazon as though it’s regular bare-metal. And encountered some real problems in the process.

1. Everyday hardware outages

Many of the firms I’ve seen hosted on AWS don’t realize the servers fail so often. Amazon actually choosing cheap commodity components as a cost-savings measure. The assumption is, resilience should be built into your infrastructure using devops practices & automation tools like Chef & Puppet.

The sad reality is most firms provision the usual way, through the dashboard, with no safety net.

Also: Is your cloud speeding for a scalability cliff

2. Ongoing network problems

Network latency is a big problem on Amazon. And it will affect you more. One reason is you’re most likely sitting on EBS as your storage. EBS? That’s elastic block storage, it’s Amazon’s NAS solution. Your little cheapo instance has to cross the network to get to storage. That *WILL* affect your performance.

If you’re not already doing so, please start using their most important & easily missed performance feature – provisioned IOPS.

Related: The chaos theory of cloud scalability

3. Hard to be as resilient as netflix

We’ve by now heard of firms such as Netflix building their Chaos Monkey to actively knock out servers, in effort to test their ability to self-healing infrastructure.

From what I’m seeing at startups, most have a bit of devops in place, a bit of automation, such as autoscaling around the webservers. But little in terms of cross-region deployments. What’s more their database tier is protected only by multi-az or just a read-replica or two. These are fine for what they are, but will require real intervention when (not if) the server fails.

I recommend building a browse-only mode for your application, to eliminate downtime in these cases.

Read: 8 questions to ask an aws expert

4. Provisioning isn’t your only problem

But the cloud gives me instant infrastructure. I can spinup servers & configure components through an API! Yes this is a major benefit of the cloud, compared to 1-2 hours in traditional environments like Softlayer or Rackspace. But you can also compare that with an outage every couple of years! Amazon’s hardware may fail a couple times a hear, more if you’re unlucky.

Meanwhile you’re going to deal with season weather problems *INSIDE* your datacenter. Think of these as swarms of customers invading your servers, like a DDOS attack, but self-inflicted.

Amazon is like a weak immune system attacking itself all the time, requiring constant medication to keep the host alive!

Also: 5 Things toxic to scalability

5. RDS is going to bite you

Besides all these other problems, I’m seeing more customers build their applications on the managed database solution MySQL RDS. I’ve found RDS terribly hard to manage. It introduces downtime at every turn, where standard MySQL would incur none.

In my experience Upgrading RDS is like a shit-storm that will not end!

Also: Does open source enable the cloud?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Can entrepreneurs learn from how science beat Cholera?

cholera ghost map johnson

When I picked up Johnson’s book, I knew nothing about Cholera. Sure I’d heard the name, but I didn’t know what a plague it was, during the 19th century.

Johnson’s book is at once a thriller, of the deadly progress of the disease. But in that story, we learn of the squalor inhabitants of victorian england endured, before public works & sanitation. We learn of architecture & city planning, statistics & how epidemiology was born. The evidence is weaved together with map making, the study of pandemics, information design, environmentalism & modern crisis management.

Join 28,000 others and follow Sean Hull on twitter @hullsean.

“It is a great testimony to the connectedness of life on earth that the fates of the largest and the tiniest life should be so closely dependent on each other. In a city like Victorian London, unchallenged by military threats and bursting with new forms of capital and energy, microbes were the primary force reigning in the city’s otherwise runaway growth, precisely because London had offered Vibrio cholerae (not to mention countless other species of bacterium) precisely what it had offered stock-brokers and coffee-house proprietors and sewer-hunters: a whole new way of making a living.”

1. Scientific pollination

John Snow was the investigator who solved the riddle. He didn’t believe that putrid smells carried disease, the miasma theory prevailing of the day.

“Part of what made Snow’s map groundbreaking was the fact that it wedded state-of-the-art information design to a scientifically valid theory of cholera’s transmission. “

Also: Did Airbnb have to fail?

2. Public health by another name

“The first defining act of a modern, centralized public health authority was to poison an entire urban population”

Although they didn’t know it at the time, the dumping of waste water directly into the Thames river was in fact poisoning people & wildlife in the surrounding areas.

In large part the establishment was blinded by it’s belief in miasma, the theory that disease was originated from bad smells & thus traveled through the air.

Related: 5 reasons to move data to amazon redshift

3. A Generalist saves the day

The interesting thing about John Snow was how much of a generalist he really was. Because of this he was able to see thing across disciplines that others of the time were not able to see.

“Snow himself was a kind of one-man coffeehouse: one of the primary reasons he was able to cut through the fog of miasma was his multi-disciplinary approach, as a practicing physician, mapmaker, inventor, chemist, demographer & medical detective”

Read: Are generalists better at scaling the web?

4. Enabling the growth of modern cities

The discovery of the cause of Cholera prompted the city of London to a huge public works project, to build sewers that would flush waste water out to sea. Truly, it was this discovery and it’s solution that has enabled much of the population densities we see in the 21st century. Without modern sanitation, 20 million plus cities would certainly not be possible.

Also: Is automation killing old-school operations?

5. Bird Flu & modern crisis management

In recent years we have seen public health officials around the world push for Thai poultry workers to get their flu shots. Why? Because although avian flu only spreads from animal to humans now, were it to encounter a person who had our run-of-the-mill flu, it could quickly learn to move human to human.

All these disciplines of science, epidemiology and so forth direct decisions we make today. And much of that is informed by what Snow pioneered with the study of cholera.

Also: 5 Things toxic to scalability

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Is Agile right for fixing performance issues?

storm coming

I was sifting through the CTO school email list recently, and the discussion of performance tuning came up. One manager had posted asking how to organize sprints, and break down stories for the process.

Join 29,000 others and follow Sean Hull on twitter @hullsean.

Another CTO chimed in with a response…

“Agile is not right for fixing performance issues.”

I agree with him & here’s why.

1. Agile roadblocks

At a very high level, agile seeks to organize work around sprints of a few weeks, and sets of stories within those sprints. The assumption here is that you have a set of identified issues. With software development, you have features you’re building. With performance tuning, it’s all about investigation.

How long will it take to solve the crime? Very good question!

Also: 5 things toxic to scalability

2. Reproduce problem

Are you seeing general site slowness? Is there a particular feature that loads extremely slowly? Or is there a report that runs forever? Whatever it is, you must first be able to reproduce it. If it’s general site slowness, identify when it is happening.

Related: Are SQL Databases dead?

3. Search for bottlenecks

Once you’ve reproduced your problem, next you want to start digging. Looking at logfiles can help you find errors, such as timeouts. The database has a slow query log, which you’ll definitely want to review. Slow queries can be surfaced by new code deploys, or middleware in front of your database, such as an ORM.

If you find your logfiles aren’t enabled, it’s a good first step to turn them on. Also look at how you’re caching. The browser should be directed to cache, assets should be on CDN, a page cache should protect your application server, and an object cache in front of your database.

Read: Is five nines a myth that just won’t die?

4. Find the root cause

As you dig deeper into your problem, you’ll likely uncover the root of your scalability problem. Likely causes include synchronous, serial or locking processes & requests, object relational modelers, lack of caching or new code that has not been tuned well.

Also: Did Airbnb, reddit , heroku & flipboard have to fail?

5. Optimize

This is what I think of as the fun part. You’ve measured the issues, found the problem. Now it’s time to fix it. This is an exciting moment, to bring real benefit to the business. Eliminating a performance problem can feel like springtime at the end of a long cold winter!

Also: Is zero downtime even possible on RDS?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Are top candidates evaluating your startup?

Editor & writer in friendly dialog

I work for a lot of startups. Many ask me for referrals. I play matchmaker when I can. But as the market continues to heat up, the demand for top talent is reaching a boiling point.

Join 29,000 others and follow Sean Hull on twitter @hullsean.

1. It’s a sellers market

That means folks with technical skills across the spectrum are very indemand. How in demand? Check Angellist, Made In NY or Indeed.com. From SRE’s to full stack developers, devops & automation experts to DBAs. Java, Ruby, Python, PHP, node.js, and of course design skills too.

I was speaking with a recruiter just today, and heard the same refrain…

Top candidates are evaluating us just as we are evaluating you.

That means firms must go the extra mile to stand out, and draw in the best talent.

Also: 5 Things toxic to scalability

2. Open the glassdoor

That’s right, manage your social media presence. Sites like Glass Door provide forums where employees past & present can discuss the day-to-day work environment. This gives prospects a chance to peer behind the curtain.

Other social media can be avenues too, from Facebook to Twitter. Having someone on staff that monitors online reputation can be crucial.

Related: Are SQL Databases dead?

3. Host a tech blog & meetups

A lot of top firms have great tech blogs. Truth be told many are dormant as demands of the day trump these outward facing initiatives. But they also put a face on the technical side of working for a firm. What problems are they solving? How cutting edge is their team?

Meetups are also a limitless forum. Smart minds will be mixing, your company brand will be spreading. Hosting technical discussions brings your firm front & center in multiple ways. It also brings possible new hires to your living room.

Read: Is high availability a myth?

4. Show warmth & transparency

I know everybody loves to grill candidates at interviews. But interviewees should be schooled on politeness & how to give a pleasant interview.

I remember one interview where I faced off with four other engineers at a round table. As the discussion unfolded, each aimed shots in succession, almost rapid fire at me. It was not only intimidating, but frustrating. Needless to say it made me a stronger more resilient interviewer, but it’s not a great way to welcome great talent. Buyer beware!

Also: The chaos theory of cloud scalability

5. Show me the money

I know I know, for engineers it’s not all about the money. Or is it? Truth be told compensation is always something prospects will weigh. Equity is fine, for what it is. But it’s a promise into the future.

More senior talent who have been through a few startups or even dot-com 1.0, may be a bit more dubious of abstract compensation. In the end competitive real dollars will speak volumes.

Also: Is upgrading Amazon RDS like a shit-storm that will not end?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters