Always be publishing

giraffe zebras

Join 28,000 others and follow Sean Hull on twitter @hullsean.

As an advisor to New York area startups & an long time entrepreneur, I’ve found writing & publishing to be extremely valuable use of time.

I follow the motto “Always be publishing” here’s why.

1. Form your voice

According to Fred Wilson, blogging has been one of the seminal decisions contributing to his success.


“It’s like Venus Fly Paper. When I write about topics that are relevant, suddenly anybody with a startup solution in that field will approach us. This works brilliantly.”

Also: 5 Things I learned from Fred Wilson & Mark Suster

2. Get in the conversation

The world online moves quickly and it can move in surprising directions. Hype, hysteria & buzz can direct the conversation as much as facts.

Getting into the conversation allows you to weigh in. This builds your credibility. As it puts you in the line of fire, you stand up & get heard.

Related: Is blogging crucial to career building?

3. Be in the line of fire

In sales there’s a saying, “always be closing”. It means always be in front of your customers, always be on point, always be getting deals done. That’s embodying your role as a salesman.

For builders, consultants, advisors, speakers & entrepreneurs, writing puts you directly in the line of fire. You express your opinions online loud & clear. Sometimes you will find critics picking apart your ideas. Sometimes they may correct you.

This process will help you hone your ideas. Strengthen some & modify & adjust others. All of it is good.

Read: Is building traffic & pagerank possible through active blogging?

4. Share your knowledge

As an advisor, entrepeneur or professional services consultant you sell your knowledge & expertise. Why not share a bit of that with the world at large.

This is one part good samaritan, and one part testimonial of your skill & style.

Also: Is Ryan Holiday about the internet & the death of journalism?

5. Learn by doing

Back in 2001 I wrote a book called Oracle + Open Source.

Along the way, writing chapter after chapter of material, there were times when I had to brush up on material. Or write & rewrite sections. Some of it wasn’t explained well, and other material I didn’t know as well as I needed to.

Today I intersperse howtos with writing on consulting, or industry trends. Inevitably a howto like Wrestling with bears or how I tamed Tungsten Replicator involves a lot of hands-on learning.

All of this is driven by blogging & publishing.

Also: Is the difference between dev & ops a four-letter word?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Why I use Airbnb chat even when texting is easier

airbnb

If you’ve ever traveled & stayed with an Airbnb host, you know that once you book you can easily switch to text messaging. Sometimes this is easier. But as I found out, it’s smarter to stick with a channel that we all can share.

Join 30,000 others and follow Sean Hull on twitter @hullsean.

I had a similar experience with a recent consulting customer. The lesson was much the same.

Choose your communication channels wisely, for you may need them for other reasons later on.

1. Not what I paid for

I’ve been hosting travelers off & on through Airbnb for some time. It’s a fun past time, as you can meet some interesting people, and share a little bit of *your* city. That and there’s a little bit of extra income too, which doesn’t hurt.

One visitor I had wasn’t particularly happy with the setup. I’ve hosted dozens of people before, so I know that the space is popular to most. However this one guy seemed unhappy from the start. He didn’t read the fine print that it was a shared space with separate rooms. He was unhappy with the specific location too. And later he complained about a bicycle I had loaned him.

Also: Is Amazon too big to fail?

2. How Airbnb chat helped

At the end of his visit he asked for some of the fee to be refunded.

As I dug through our Airbnb chat, I copy/pasted our various communications, and in the end this helped clarify & remedy the situation. It also didn’t hurt that Airbnb themselves were there behind the scenes and could review all these messages as well.

Having a third party to arbitrate can make a big difference. Lets hope it doesn’t come to that, but if it does, you want a communication channel they can also see.

Related: 5 Reasons to move data to Amazon Redshift

3. Consulting engagements & corporate emails

Over the years I generally use my own email for projects & engagements. However recently I took on a longer engagement. At the start there was some insistence on using an internal email for communications. I was hesitant, but eventually conceded as it tied in with google calendaring and various internal aliases.

As the months went by, I tried & failed to use both emails for correspondence. It was a habit that was hard to change. What’s more forwarding *all* emails to my own was also difficult. With an ongoing barrage of all@company.com messages numbering in the hundreds, it simply blew up my email account. That wasn’t sustainable either.

Read: Do managers underestimate operational cost?

4. After you leave

You may not be thinking of after your consulting assignment at the start of it. But you should be. You’ll engage in many communications, about a lot of different topics. Some about what is & isn’t in scope. Some about deliverables & timelines.

You’ll also have communications has things unfold, and as they are delivered. All of these are crucial to the engagement, as evidence of what was done when. If after you leave, all those emails are gone (at least that you can reach), it can be problematic.

What’s more once you set a precedent communicating one way, it’s hard to change habits. Best to set the precedent strongly up front.

Also: Are we fast approaching cloud-mageddon

5. Your channel is your paper trail

In todays mobile-heavy world, there are tons of channels we can use to communicate. From Whatsapp to Slack, Hipchat to email & text. They all have their strengths & weaknesses.

But sometimes we need to choose based on future needs. Leaving a paper trail can be important. Having future control over those past communications can bring legal benefit.

And all of these communications can help avoid misunderstandings if they’re available for review later.

Also: Which tech do startups use most?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Is Amazon too big to fail?

aws fault tolerance

Amazon is the huge online retailer everyone knows well. However there is another side of Amazon, namely Amazon Web Services that hosts many of the internets largest websites.

Join 28,000 others and follow Sean Hull on twitter @hullsean.

In the infrastructure & operations world, Amazon is the Citibank, JP Morgan or Goldman Sachs of cloud providers.

1. Outage takes down Yelp & Netflix

As reported on Thousand Eyes among other places, Amazon had a major outage yesterday.

Amazon experienced a problem with how they route data over the network. Routing is the technical term for how the internet moves data around. When routing goes wrong at a provider like Amazon, the websites they host will go down too.

Also: Are we fast approaching cloud-mageddon?

2. Automation can’t save you

Netflix is famous for their great streaming service, and shows like House of Cards.

On the technology side they’re also pretty famous. They deploy legions of Amazon servers to stream movies using Chaos Monkey. This open source suite allows them to remain resilient even if individual servers or components go offline.

Yet a heavy reliance on Amazon itself, meant a wider outage for them was also an outage for Netflix.

Related: What tech do startups use most?

3. Of cloud monopolies

Amazon’s dominance in the cloud hosting space is incredible. There are providers that can beat them in compute power, speed & price. But with their incredible reach of global datacenters & relentless growth they are still the first choice for most internet shops.

What is the downside of such dominance? What happened yesterday illustrates it clearly. When Amazon goes down, so do financial companies like Experian,

Read: Do managers underestimate operational cost?

4. Diversify your data portfolio

In the banking world we can put together legislation, regulating banks. We can enact capital requirements or consider breaking up the largest ones. For investors & consumers you can diversify your portfolio, putting money in different asset classes & institutions. If one fund fails, others will balance it out.

We can do the same with cloud hosting. For larger internet applications, deploying on multiple clouds can be very beneficial. In that case an outage at Amazon, would merely mean your global load balancer kicks in, sending traffic to your plan B servers.

Also: Replicate big data to Amazon Redshift with Tungsten

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Replicate MySQL to Amazon Redshift with Tungsten: The good, the bad & the ugly

tungsten replicator

Heterogenous replication involves moving data from one database platform to another. This is a complicated endevour because datatypes, date & time formats, and a whole lot more tend to differ across platforms. In fact it’s so complex many enterprises simply employ a commercial solution to take away the drudgery.

Join 31,000 others and follow Sean Hull on twitter @hullsean.

Enter Tungsten, which supports these types of deployments, on platforms as Postgresql, Mongodb, Oracle, Redshift, Vertica. With custom built appliers the field is infinite!

With that I’ve set out to get things working with Amazon Redshift. If you’re still struggling with the basics check out Wrestling with bears or how I tamed Tungsten Replicator.

1. Connect to redshift

The first thing you’ll need to do is allow your Tungsten boxes to reach redshift. Seems obvious, but when you’re juggling all these apples & oranges for the first time, it may slip you mind.

Configure your AWS security group to allow tungsten boxes

Get the external IP address of your tungsten box. If it’s in DNS this will work even if ping doesn’t.


$ ping tungsten01.mydomain.net

Add 10.20.30.40/32 to your Redshift security config. I created a special group called Tungsten and added the two tungsten boxes by IP address. That’s because these machines were on a different AWS account. If they’re on the same account, you could allow the entire EC2 group, and be done.

Install psql client

The best way I found to test the connection was psql. Install that:


$ apt-get install postgresql-client

Verify your connection:


$ psql -p 5439 -h 10.20.10.20 --username=root -d dwh

Also: Are SQL Databases dead?

2. Configure S3 access

Tungsten uses S3 heavily to move data into Redshift.

(I outlined this previously in 5 Reasons to move data to Amazon Redshift.

Install s3tools package

Tungsten uses the s3cmd to interface with the Amazon S3 API. Let’s install that:


$ apt-get install s3cmd

Now edit the .s3cfg file of tungsten user. Change


[default]
access_key = AAAAAAA
secret_key = BBBBBBB

Lastly edit the tungsten /opt/continuent/share/s3-config-redshift.json. There are four parameters.


{
"awsS3Path" : "s3://tungstenbucket",
"awsAccessKey" : "AAAAAAA",
"awsSecretKey" : "BBBBBBB",
"cleanUpS3Files" : "false",
}

Related: Is Oracle killing MySQL?

3. Create tables on Redshift

In a heterogenous environment, that is where source and destination databases are different platforms, Tungsten cannot create tables for you.

It will however, give you a helping hand in the process. Enter the ddlscan tool, which scans the CREATE TABLE statements on your source database, and generates them for your target platform.

For each table in source database, there will be a stage table in Redshift:


$ ddlscan jdbc:mysql://localhost:3306/test -user sync -db test -template ddl-mysql-redshift-staging.vm > test_stage.sql

$ cat test_stage.sql
/*
SQL generated on Thu Jun 04 20:06:45 UTC 2015 by ./ddlscan utility of Tungsten

url = jdbc:mysql:thin://tungsten01.mydomain.net:3306/test?jdbcCompliantTruncation=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&allowMultiQueries=true&yearIsDateType=false
user = sync
dbName = test
*/

CREATE SCHEMA test;

DROP TABLE test.stage_xxx_sean;
CREATE TABLE test.stage_xxx_sean
(
tungsten_opcode CHAR(2),
tungsten_seqno INT,
tungsten_row_id INT,
tungsten_commit_timestamp TIMESTAMP,
c1 VARCHAR(256) /* VARCHAR(64) */,
id INT,
PRIMARY KEY (tungsten_opcode, tungsten_seqno, tungsten_row_id)
);

And also a base table in redshift:


$ ddlscan jdbc:mysql://localhost:3306/test -user sync -db test -template ddl-mysql-redshift.vm > test.sql

$ cat test.sql
/*
SQL generated on Thu Jun 04 20:06:51 UTC 2015 by ./ddlscan utility of Tungsten

url = jdbc:mysql:thin://tungsten01.mydomain.net:3306/test?jdbcCompliantTruncation=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&allowMultiQueries=true&yearIsDateType=false
user = sync
dbName = test
*/

CREATE SCHEMA test;

DROP TABLE test.sean;
CREATE TABLE test.sean
(
c1 VARCHAR(256) /* VARCHAR(64) */,
id INT,
PRIMARY KEY (id)
);

Lastly apply those scripts to your redshift database:


$ psql
dwh# \i file_stage.sql
dwh# \i file_table.sql

Read: Are we fast approaching cloud-mageddon?

4. Troubleshoot applier

***
Encountered “Delimiter Not Found” issue

This issue was mysterious and remains so a bit. What I did to fix it:

had an issue with the path, but fixed that:


  "awsS3Path" : "s3://tungstenbucket",

It was causing an interim bucket to be created. But that did not solve things.

Ok. So I hacked this a bit.

Anyone can help me troubleshoot what happened & why?

A. I skipped transactions

I brought the applier back online with this command.


trepctl -service redshift online -skip-seqno 1,1-100

B. I did lots of inserts & deletes on MySQL

I then did about 200 of these:


mysql> insert into test.sean values ('hi there', 20);
mysql> delete from test.sean where id = 20;

C. Now seeing data


dwh=# select * from test.sean;
                 c1                  | id 
-------------------------------------+----
 working......                       | 25
 hello sean i have an exclamation !! | 27
 hello sean i came from mysql        | 26
(3 rows)

I also set cleanupS3Files to false. Now I’m seeing files like this:
test-sean-417.csv
test-sean-418.csv
test-sean-419.csv
test-sean-420.csv

So that indicates all those INSERT followed by DELETES cleaned up things.

Also: How do I find entrepreneurial focus?

5. Test data & table changes

B. Tested INSERT

At first the csv files were getting cleanedup by Tungsten. I added this option to s3-config-redshift.json file:


"cleanUpS3Files" : "false",

Then the files are kept around so we can review them. An insert record shows up in S3 like this:


"I","417","1","2015-06-05 17:44:35.000","tungsten new csv file? ","33",null

C. Tested DELETE

A DELETE record shows up in S3 like this:


"D","419","1","2015-06-05 17:45:48.000",null,"26",null

D. Tested UPDATE

An UPDATE record shows up in S3 like this:


"D","420","1","2015-06-05 17:48:55.000",null,"31",null
"I","420","2","2015-06-05 17:48:55.000","changed message text for redshift+tungsten update","31",null

A. Tested ALTER TABLE

As mentioned previously, this is *NOT* supported. However after doing the ALTER, the applier does *NOT* go offline. Also there are no errors. That’s because Tungsten does not support these and will filter them in a heterogenous environment.

The applier *DOES* go offline, after you try a new INSERT. That’s because it gets a new record for INSERT that doesn’t match.

“trepctl status” shows the following:

pendingExceptionMessage: CSV loading failed: schema=test table=sean CSV file=/tmp/staging/redshift/staging0/test-sean-413.csv message=Wrapped org.postgresql.util.PSQLException: ERROR: Load into table ‘stage_xxx_sean’ failed. Check ‘stl_load_errors’ system table for details. (../../tungsten-replicator//samples/scripts/batch/redshift.js#145)

redshift# alter table test.sean add column c3 integer default null;

redshift# alter table test.stage_xxx_sean add column c3 integer default null;

Then I brought the applier back online:

$ trepctl -service redshift online

Then check the status. It should say ONLINE for state.


$ trepctl status
Processing status command...
NAME VALUE
---- -----
appliedLastEventId : mysqld-bin.000022:0000000000000566;-1
appliedLastSeqno : 424
appliedLatency : 300585.739
autoRecoveryEnabled : false
autoRecoveryTotal : 0
channels : 1
clusterName : redshift
currentEventId : NONE
currentTimeMillis : 1433878195573
dataServerHost : my-dw.aaaa.us-east-1.redshift.amazonaws.com
extensions :
host : my-dw.aaaa.us-east-1.redshift.amazonaws.com
latestEpochNumber : 0
masterConnectUri : thl://tungsten01.mydomain.net:2112/
masterListenUri : null
maximumStoredSeqNo : 424
minimumStoredSeqNo : 0
offlineRequests : NONE
pendingError : NONE
pendingErrorCode : NONE
pendingErrorEventId : NONE
pendingErrorSeqno : -1
pendingExceptionMessage: NONE
pipelineSource : thl://tungsten01.mydomain.net:2112/
relativeLatency : 304511.573
resourcePrecedence : 99
rmiPort : 10000
role : slave
seqnoType : java.lang.Long
serviceName : redshift
serviceType : local
simpleServiceName : redshift
siteName : default
sourceId : my-dw.aaaa.us-east-1.redshift.amazonaws.com
state : ONLINE
timeInStateSeconds : 351940.007
timezone : GMT
transitioningTo :
uptimeSeconds : 600921.759
useSSLConnection : false
version : Tungsten Replicator 4.0.0 build 18
Finished status command...
$

Lastly, let’s see what’s in the table, fire up the postgresql shell and take a look:


dwh=# select * from test.sean;
c1 | id | c3
---------------------------------------------------+----+----
working...... | 25 |
hello sean i have an exclamation !! | 27 |
hello will i break? | 30 |
some more records | 32 |
tungsten new csv file? | 33 |
another tungsten csv file? | 34 |
changed message text for redshift+tungsten update | 31 |
(7 rows)

Also: Was Fred Wilson wrong about Apple?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Wrestling with bears or how I tamed Tungsten replicator

tungsten replicator

I just dove into Tungsten replicator very recently as I need to replicate from Amazon RDS to Redshift. I’d heard a lot of great things about Tungsten, but had yet to really dig my heels in.

Join 28,000 others and follow Sean Hull on twitter @hullsean.

I fetched the binary and began to dig through the docs. Within a day I felt like I was sinking in quicksand. Why was this thing so darn complicated? To my mind unix software is a config file simple.cfg, a logfile simple.log, a daemon simpled. Open some ports & voila you’re cooking.

Unfortunately for beginners Tungsten took a very different approach. Although they support .ini files to config, they seem to encourage these huge commands to “configure” which means generate a config file in Tungsten speak, and install which literally means install the software for you into /opt/continuent.

After various posts to the forums, and a lot of head scratching, I discovered the cookbook. At first I thought this referenced puppet or chef type cookbooks. But it’s something different. It allowed me to setup a very basic tungsten just to see things working.

Tungsten supports all sorts of “topologies”. For this example I am just doing master-slave. There are two nodes and each has mysql running on it. Node1 serves as the master, and runs the tungsten replicator (master node service). And node2 serves as the slave and runs the tungsten applier (slave node service).

Good luck and hope this helps others speedup the learning curve!

1. Download tarball (on master)

Note the forums indicated they may be moving off of google code. So this url may change.


$ cd /tmp
$ wget http://downloads.tungsten-replicator.org/download.php?file=tungsten-replicator-oss-4.0.0-18.tar.gz

Also: Why Airbnb didn’t have to fail

2. Expand the tarball (on master)

Use your vast unix skills to expand the tarball!


$ mv download.php\?file\=tungsten-replicator-oss-4.0.0-18.tar.gz tungsten.tgz
$ tar xvzf tungsten.tgz
$ mv tungsten-replicator-4.0.0-18 stage

Also: Is the difference between dev & ops a four-letter word?

3. Install MySQL (each box)

Hopefully you’ve done this before. Pretty straightforward:


$ apt-get install mysql-client mysql-server

Also: Are SQL Databases Dead?

4. Edit cookbook files (on master)

Inside /tmp/stage/cookbook you’ll need to make a few simple edits:

edit COMMON_NODES.sh

Edit NODE1 and NODE2 and comment out other lines.


export NODE1=ip-172-31-0-117
export NODE2=ip-172-31-1-188

edit NODES_MASTER_SLAVE.sh


export MASTERS=($NODE1)
export SLAVES=($NODE2)

USER_VALUES.sh


export TUNGSTEN_BASE=/opt/continuent
export MY_CNF=/etc/mysql/my.cnf
export DATABASE_USER=sync
export DATABASE_SUPER_USER=root
export DATABASE_PASSWORD=

Also: 5 Things toxic to scalability

5. Create install directory (each box)

The /tmp/stage directory you created above is just a holding ground for the tarball. Continuent in it’s infinite wisdom wants to install itself. So, create a directory for it:

As root:


$ mkdir /opt/continuent
$ chown tungsten /opt/continuent

Then as tungsten:


$ touch /opt/continuent/testfile.txt

Also: How to deploy on Amazon EC2 with Vagrant?

6. Configure aws security groups

AWS security group permissions are key to getting any of this tungsten stuff working. And there are a lot of moving parts here. Ping is required for various tests, as is MySQL’s port. But you’ll also need to enable Tungsten History Log port 2112 and the replicator ports.

o ping ICMP from your group
o enable inbound 3306 from your group
o enable THL – inbound 2112 from your group
o enable RMI – inbound 10000, 10001 from your group

Also: Howto interview an AWS expert for managers, recruiters & engineers alike

7. Test mysql client from the other (each box)


$ mysql -h ip-172-31-1-188 -u root

$ mysql -h ip-172-31-0-117 -u root

If these hang, verify 3306 in your aws security groups. If you get a mysql error, be sure the “host” is appropriate. Check with:


mysql> select user, host, password from mysql.user where user = 'root';

For more info see Connect to MySQL in the Amazon public cloud.

Also: 5 Reasons to move to amazon redshift

8. Setup ssh auto-login (each box)

You need to be able to login to each box as a user “tungsten” without a password.


$ adduser tungsten
$ ssh-keygen
$ scp .ssh/id_rsa.pub tungsten@ip-172-31-1-188:/home/tungsten/.ssh/authorized_keys

Be sure the authorized_keys file is 600:


$ chmod 600 .ssh/authorized_keys

Also: Did MySQL & Mongo have a beautiful baby called Aurora

9. Install ruby (each box)

You need to have ruby on both boxes.


$ apt-get install ruby

Also: Top interview questions for a MySQL expert – recruiters, managers & candidates

10. Create sync user (each box)

Both of your mysql instances will need a user that tungsten connects as. I called it “sync”.


mysql> create user 'sync'@'%' identified by 'secret';
mysql> grant all privileges on *.* to 'sync'@'%';
mysql> flush privileges;

Note, you may want to use a blank password at this stage, to eliminate that as a potential problem to debug. Also consider the security implications of ‘%’ and consider a subnet wildcard such as ’10.0.%’.

Also: Myth of five nines why high availability is overrated

11. Enable binary log (each box)

Enable the mysql binary log with log_bin parameter. Also ensure that /var/lib/mysql is readable by the tungsten user, either by changing /var/lib to 655 or adding tungsten to the mysql group. Test this as well using less or cat.

Also: What is high availability and why is it important?

12. Update MySQL startup settings (each box)

Fire up your favorite editor and update /etc/mysql/my.cnf settings (both servers). The following are the main ones:


server_id = 1 (server_id=2 for slave)
sync_binlog = 1
max_allowed_packet = 52m
open_files_limit = 65535
innodb_log_file_size=50m
#bind_address localhost (comment it to disable)

Note that changing innodb_log_file_size is tricky. You’ll need to stop mysql, rename old ib_logfile0 to ib_logfile0.old and ib_logfile1 to ib_logfile1.old. Then change the param in my.cnf and start mysql. Otherwise you’ll get errors on startup.

Also: Is the difference between dev & ops a four-letter word?

13. Run the installer (on master)

This step is fairly straightforward. If there are problems in this step, you’ll see ERROR lines in the output. Sift through them and resolve one by one.


$ ./cookbook/install_master_slave

Also: RDS or MySQL – Ten use cases

14. Check tungsten status (each box)

The “trepctl” utility allows you to check the current status. It does a lot more, but for now that’s enough. If you want to make it easier add “/opt/continuent/tungsten/tungsten-replicator/bin” to your path.

Also notice the line “state”. It should be ONLINE. If it says “GOING-ONLINE:SYNCHRONIZING” that likely means you didn’t open up the tungsten ports. You’ll need both RMI ports 10000 and 10001 as well as THL ports 2112. We all know how finicky AWS security groups can be. I’ll leave it to your own exercise to confirm those are open.

ON MASTER


root@ip-172-31-0-117:/opt/continuent/tungsten/tungsten-replicator/bin# ./trepctl status
Processing status command...
NAME VALUE
---- -----
appliedLastEventId : mysqld-bin.000005:0000000000000321;235
appliedLastSeqno : 5
appliedLatency : 34824.086
autoRecoveryEnabled : false
autoRecoveryTotal : 0
channels : 1
clusterName : cookbook
currentEventId : mysqld-bin.000005:0000000000000321
currentTimeMillis : 1432840323683
dataServerHost : ip-172-31-0-117
extensions :
host : ip-172-31-0-117
latestEpochNumber : 0
masterConnectUri : thl://localhost:/
masterListenUri : thl://ip-172-31-0-117:2112/
maximumStoredSeqNo : 5
minimumStoredSeqNo : 0
offlineRequests : NONE
pendingError : NONE
pendingErrorCode : NONE
pendingErrorEventId : NONE
pendingErrorSeqno : -1
pendingExceptionMessage: NONE
pipelineSource : /var/lib/mysql
relativeLatency : 44450.683
resourcePrecedence : 99
rmiPort : 10000
role : master
seqnoType : java.lang.Long
serviceName : cookbook
serviceType : local
simpleServiceName : cookbook
siteName : default
sourceId : ip-172-31-0-117
state : ONLINE
timeInStateSeconds : 82860.917
timezone : GMT
transitioningTo :
uptimeSeconds : 82861.492
useSSLConnection : false
version : Tungsten Replicator 4.0.0 build 18
Finished status command...
root@ip-172-31-0-117:/opt/continuent/tungsten/tungsten-replicator/bin#

ON SLAVE


tungsten@ip-172-31-1-188:/opt/continuent/tungsten/tungsten-replicator/bin$ ./trepctl status
Processing status command...
NAME VALUE
---- -----
appliedLastEventId : mysqld-bin.000005:0000000000000321;235
appliedLastSeqno : 5
appliedLatency : 42569.62
autoRecoveryEnabled : false
autoRecoveryTotal : 0
channels : 1
clusterName : cookbook
currentEventId : NONE
currentTimeMillis : 1432840348936
dataServerHost : ip-172-31-1-188
extensions :
host : ip-172-31-1-188
latestEpochNumber : 0
masterConnectUri : thl://ip-172-31-0-117:2112/
masterListenUri : thl://ip-172-31-1-188:2112/
maximumStoredSeqNo : 5
minimumStoredSeqNo : 0
offlineRequests : NONE
pendingError : NONE
pendingErrorCode : NONE
pendingErrorEventId : NONE
pendingErrorSeqno : -1
pendingExceptionMessage: NONE
pipelineSource : thl://ip-172-31-0-117:2112/
relativeLatency : 44475.936
resourcePrecedence : 99
rmiPort : 10000
role : slave
seqnoType : java.lang.Long
serviceName : cookbook
serviceType : local
simpleServiceName : cookbook
siteName : default
sourceId : ip-172-31-1-188
state : ONLINE
timeInStateSeconds : 1906.6
timezone : GMT
transitioningTo :
uptimeSeconds : 82882.334
useSSLConnection : false
version : Tungsten Replicator 4.0.0 build 18
Finished status command...
tungsten@ip-172-31-1-188:/opt/continuent/tungsten/tungsten-replicator/bin$

Also: Is zero downtime even possible with Amazon RDS?

15. Perform simple test

Create a table on the master mysql node.


mysql master> create database test;
mysql master> create table test.sean (c1 varchar(64));
mysql master> insert into test.sean values ('hi there');
mysql master> insert into test.sean values ('this should show up in tungsten thl file');
mysql master> insert into test.sean values ('new data on tungsten02 THL??');

Verify that the table is on the slave mysql node.


mysql slave> show databases;
mysql slave> select * from test.sean;
+------------------------------------------+
| c1 |
+------------------------------------------+
| hi there |
| this should show up in tungsten thl file |
| new data on tungsten02 THL?? |
+------------------------------------------+
3 rows in set (0.00 sec)

Success!

Also: Why are MySQL & other database experts so hard to find?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Are we fast approaching cloud-mageddon ?

storm coming

One look at StackShare’s trending technologies, and you’ll discover the exploding growth of languages, webservers, load balancers, databases, caching servers, automation & monitoring tools, continuous integration suites & a broad spectrum of Software as a service solutions.

The choices today boggles the mind. Choice is good, but too much choice can mean trouble too.

Join 30,000 others and follow Sean Hull on twitter @hullsean.

1. What am I actually using?

Erich Schubert wrote a superb piece about the sad state of the sysadmin in the age of containers. Here’s what caught my eye:

Stack is the new term for “I have no idea what I’m actually using”.

That definitely rings true for me. The customers I’m seeing these days have such complicated stacks, that nobody really knows what’s installed. That’s dangerous!

Also: Do today’s startups assemble at their own risk?

2. Embrace failure more broadly

Recently I wrote a blog post asking Is AWS the patient that needs constant medication?. It got a lot of traction, and here’s why I think that happened.

AWS uses very commodity, cheapo components. The assumption is, with an infinitely redundant datacenter, component failure is ok. It’s ordinary & everyday.

Unfortunately most startups, even ones that employ some Ansible & devops, still don’t have Netflix grade automation.

Those regularly everyday failures are still getting detected by old-school manual monitoring. And that’s a recipe for trouble

Also: 5 Things toxic to scalability

3. What are complex systems?

In this excellent deck, James Urquhart talks about emergent behavior in complex systems. It’s worth a quick read.

***

Read: How I find entrepreneurial focus

4. What to do? Do you like boring?

Dan McKinley formerly principal engineer at Etsy & now with Stripe wrote a brilliant essay arguing for boring technology.

This comes as a shock to many in the startup world. It sort of smacks in the face of open source, or does it?

I worked in the enterprise space as an Oracle DBA for a decade starting in the mid-nineties. Among DBAs there was always a chuckle when a new version of Oracle came out. No one wanted to touch it with a ten foot pole. Sure we’d install it on test boxes, start learning the new features and so forth. But deploy it? No way, wait a good 2 or even 3 years before upgrading.

Meanwhile management was eager for the latest software. Don’t we want the newest? The Oracle sales guys would be selling the virtues of all sorts of features that nobody needed right away anyway.

Choosing boring components takes discipline to fight sexy new technologies & bleeding edge versions. But staid & stodgy wins you everyday in operations uptime.

Related: Is automation killing old-school operations?

5. Use tried & tested components

Do you find your application or stack contains java, ruby, python & PHP? Choose one.

One webserver like nginx, one caching server like memcache or redis, one search server like solr or elasticsearch, one database like MySQL or postgres. Standardize all your components on one image, so you can use that for all your servers, regardless of which you use.

Fewer components will mean fewer interdependencies, less maintenance, & less chaos.

Also: What’s the luckiest thing that’s happened in your career?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

What happens when entrepreneurs treat data as a product?

Sneachta Pix

Sneachta Pix

I’ve been reading DJ Patil’s thoughts on building data products. As the chief data scientist of the united states, he knows a thing or two.

Join 30,000 others and follow Sean Hull on twitter @hullsean.

I also attended a recent Look & Tell event, where Lincoln Ritter talked about Data Democracy at Animoto. He expressed many of Patil’s lessons.

I took away a few key lessons from these that seem to be repeating refrains…

1. UX of data

UX design involves looking at how customers actually use a product in the real world. What parts of the product work for them, how they flow through that product and so on.

That same design sense can be applied to data. At high level that means exposing data in a measured, meaningful & authoritative way. Not all the tables & all the data points but rather key ones that help the business make decisions. Then layering on top discovery tools like Looker to allow the biz-ops to make more informed decisions.

Also: 5 Reasons to move data to Amazon Redshift

2. Be iterative

Clean data, presented to business operations in a meaningful way, allows them to explore the data, and find useful trends. What’s more with good discovery tools, biz-ops is empowered to do their own reporting.

All this reduces the need to go to engineering for each report. It reduces friction and facilitates faster iteration. That’s agile!

Related: Is automation killing old-school operations?

3. Be authoritative

Handing the keys to the data kingdom over to business means more eyes on the prize. That may well surface data inconsistencies. Each such case can reduce trust on your data.

Being authoritative means building checks into your data feeds, and identifying where data is amiss. Then fixing it at the source.

Read: Are SQL Databases dead?

4. Spot checks & balances

Spot checks on data are like unit tests on code. They keep you honest. Those rules for how your business works, and what your data should look like, can be captured in code, then applied as tests against source data.

Also: Is Apple betting against big data?

5. Monitoring for data outages

As data is treated as a product, it should be monitored just like other production systems. A data inconsistency or failed spot check then becomes an “outage”. By taking these very seriously, and fire fighting just as you do other production systems, you can build trust in that data, as those fires become less frequent.

Also: Why Airbnb didn’t have to fail

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

How I find entrepreneurial focus

Brian K

Relentless focus. This is surely a key to entrepreneurial success. But how to find it? And how to maintain that focus through the ups & downs?

Join 28,000 others and follow Sean Hull on twitter @hullsean.

I’ve found a simple system with a few rules has brought me success. I’ve used it for two decades of as entrepreneurial, and still do.

Here’s how it works…

1. keep a list of small tasks

This is the number one thing I do daily. I start out early in the morning, over coffee. Anything that “needs to get done” goes on the todo list, but I also separate out the things I’ll do today.

Todo list items are not big ones, like “save the world” or “get new job”. They are small nuggets of work that take 15 to 60 minutes. If they’re larger, they need to be broken down.

A typical day covers five major tasks. You will get sidetracked. You’ll need to answer calls & emails that aren’t captured on the list.

All this sounds simple, but it actually requires a discipline. Both to keep tasks actionable & small. You’ll learn your work pace with practice. But there’s one more thing to remember.

Also: Are generalists better at scaling the web?

2. trust the list

One thing I find myself doing is pushing just because something is on the list.

I don’t do list based on feelings or moods

This requires a lot of habit building, but it becomes valuable. By doing this over time, you begin to trust the list. You know things on it will get done. So you can safely “add it to the list” and forget about it for the moment.

This lets your mind relax and bcomes a real godsend when you have a mountain of work to do.

Just work list because it’s there. And trust that things that need to get done simply go on the list.

Related: Why airbnb didn’t have to fail

3. done with list, done for day

On days where things get hairy, and you work more, you’ll have to slog through to get everything on the list done. But sticking to it will build a habit that’s valuable.

At the same time some days will be easier. Avoid the temptation to add more work to fill the day.

When you’re done with the list, you’re done for the day

This is a discipline too. Pat yourself on the back, and give yourself a break. You did what you said you would do. Time for a beer. :)

Read: Which tech do startups use most?

4. big projects require faith

Anothe lesson I’ve learned is that really large projects, or ones bringing you into new areas, require a lot of faith. For me, with an engineering background, I don’t have an easy time finding that. I want to measure, and dice up everything from the start.

When I was embarking on a project to buy real estate in Brooklyn, I really learned this lesson. There was so much unknown. How do I work with real estate brokers that have a different style of communicating than engineers & corporate professionals? How do I negotiate? What’s the right price? What about mortgages & their brokers? Architect inspections, land surveys, flood zones, crime maps, loans & assets, legal & closing costs. The list of unknown & nebulous areas of expertise was staggering.

Hang around edges to get lay of land

If I would distill this faith idea down for someone embarking on a new career or diving into a pool of unknown depth, I would say start by hanging around the edges. Pick off pieces that you can, and add them to the todo list.

I went to open houses. I asked questions. I researched online, and I always made sure I understood how agents & players were incentivized. That means believing none of the words people speak, but rather, look behind the curtain and make educated guesses about those realities.

Also: Do todays startups assemble at their own risk?

5. break down & do

It is inevitable that you will experience writers block. Or any other kind of block that manifests as procrastination. Don’t over think it.

Continue to break down to smallest viable unit that you *can* do.

get started on anything to get inertia

With writing I find blocks where I don’t have a solid idea formulated. Maybe you have a topic? So then my todo list for the day is “write five bullet points”. This by itself will take some time, but you know you can write something. By moving past your block, I sometimes find I wanna keep writing and finish the piece. This is the kind of habit you want to form.

Also: Is Fred Wilson right to say speed is a feature?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

5 Things I learned from Fred Wilson & Mark Suster

I was recently flipping through AVC.com and saw this interview by Mark Suster of Upfront Ventures. He talks in depth with veteran in the VC world, Fred Wilson.

Join 30,000 others and follow Sean Hull on twitter @hullsean.

Here’s a more in-depth blog post on Mark’s interview with Fred Wilson from Union Square Ventures.

1. I’m not into debt

Around the 20:25 in the interview, Fred is discussing a period in his career before some of his first big investments, where things were financially challenging. He makes a rather candid comment about personal debt:

“I’m not that kinda person. I don’t like debt. I’m not into debt”

I think this is key. I also think it frames the whole way people approach business & career.

Also: Are generalists better at scaling the web?

2. Brains & hustle is key

Among the most successful entrepreneurs there are certainly many who are very intellectually astute. Meanwhile there are others who are great speakers, who can sell an idea, and persuade, but perhaps not as deep product wise or deeply technical.

The very best though, tend to have both brains & hustle.

Related: 8 questions to ask an AWS expert

3. Best technology doesn’t win markets

Around 11:45 in the interview, Mark & Fred are discussing Novell & Banyan.

“That was when I learned that best technology doesn’t win markets”

t’s interesting because as you hear the story of how Banyan lost out to Novell, it resonates today with companies that often have the best tech, but don’t win in markets. Interesting.

Read: Why Airbnb didn’t have to fail

4. Find answers through blogging

“It’s like Venus Fly Paper. When I write about topics that are relevant, suddenly anybody with a startup solution in that field will approach us. This works brilliantly.”

Indeed, I’ve found blogging to be crucial myself to career building. It helps in a myriad of ways.

Blogging brings visibility, as your blog gains in popularity. That is certainly big. But also it helps you craft & formalize your voice & your vision. Blogging asks you everyday to think about your perspective, and share it in a way that appeals to a broad audience. And analytics give you real feedback that you are saying something of value to people.

Also: Are SQL databases dead?

5. Listen to the younger generation

Around 1:11:15 in the interview, there’s an interesting point where Mark asks Fred if there were any deals that they regret not getting into. Fred responds that AirBNB was such a deal, as it was a quintessential Union Square ventures company.

As it turns out they didn’t invest because they couldn’t imagine using the service. Meanwhile the younger members of their team had a different perspective.

“We’re not gonna reject anything that we wouldn’t do and the younger team would.”

Interesting point. I think of Venmo as another example of this. I personally wouldn’t use the service, meanwhile it is clearly very popular among teen & twenty something demographic.

Also: 5 Things toxic to scalability

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

What’s the luckiest thing that’s happened in your career?

sashi [via Flickr]

I was browsing through Career Dean recently, a site that facilitates professionals to share knowledge & experience with more junior & recent college grads about the work world. It’s a great site. I saw the question What’s the luckiest thing that’s ever happened for your career?

I read in John Adam’s AMA his “million dollar piss” (www.careerdean.com/q/howd-you-get-the-job-twitter), which he sowed the seeds of his success basically during a piss. That’s a 1 in a million kind of story I know. I’d like to hear if anyone else has ever experienced anything remotely lucky in that way? =) something fun to come back and read if anyone answers.

Join 28,000 others and follow Sean Hull on twitter @hullsean.

Here’s how I responded…

I moved to NYC & worked at a tiny startup in the mid-nineties. Got to do Mac stuff, windows & Sun Solaris unix as well. Jumped on an Oracle project where I was a bit underwater. The firm hired a consultant to assist me for a few days. I watched what he did and learned like a sponge. Within a few months I dove into Oracle consulting and never looked back.

I felt this was an amazingly lucky opportunity to for a few reasons.

1. DIY

I’ve been consulting for almost twenty years now. And I get asked all the time how to get into freelance or independent consulting. For me the jumping off point was working for a really small ten person startup.

An environment like this is very different from a large corporation where you do one thing. At a tiny shop, everything is very do-it-yourself. You have to be self-serve & lean. It’s a constant challenge to teach yourself what you don’t already know. It’s a very vibrant environment as you enter your career.

Also: 5 Things toxic to scalability

2. Generalist

I also found that I had the chance to really apply everything I learned in computer science. It’s a hardware problem? It’s a software problem? These kind of silos that you experience at university don’t apply. One day you can be doing windows, mac, or Unix operating system configuration, the next you can be writing code. And on the third day you can be doing dba work.

In today’s terminology, this role was site reliability engineer or SRE, fullstack developer, tech support, evangelist, CTO, DBA, scalability & performance lead and more.

Related: Are generalists better at scaling the web?

3. Cutting edge

Startups to be sure are on the bleeding edge. They’re constrained by budgets, and through sheer will & experimentation, are cutting their teeth on the newest technologies out there.

These days that might be Cassandra & Kafka, Docker, MongoDB, hdfs, Redshift and so on.

Read: Do managers underestimate operational cost?

4. Ok to Fail

In larger enterprises, a lot of politics weigh on decisions, and exotic technologies are risky. When you’re at a startup, and by design you are entering uncharted waters, it’s sort of a given that it is ok to fail. This encourages learning, as there is less risk of failure.

Also: Is the difference between dev & ops a four-letter word?

5. Iterative & Agile

We talk about being agile, and lean at startups. At a very small place like this, you have one or two developers, and you deploy code constantly. It’s agile by default. And that’s a good thing.

Also: Is high availability overrated? The myth of five nines.

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters