Wrestling with bears or how I tamed Tungsten replicator

tungsten replicator

I just dove into Tungsten replicator very recently as I need to replicate from Amazon RDS to Redshift. I’d heard a lot of great things about Tungsten, but had yet to really dig my heels in.

Join 28,000 others and follow Sean Hull on twitter @hullsean.

I fetched the binary and began to dig through the docs. Within a day I felt like I was sinking in quicksand. Why was this thing so darn complicated? To my mind unix software is a config file simple.cfg, a logfile simple.log, a daemon simpled. Open some ports & voila you’re cooking.

Unfortunately for beginners Tungsten took a very different approach. Although they support .ini files to config, they seem to encourage these huge commands to “configure” which means generate a config file in Tungsten speak, and install which literally means install the software for you into /opt/continuent.

After various posts to the forums, and a lot of head scratching, I discovered the cookbook. At first I thought this referenced puppet or chef type cookbooks. But it’s something different. It allowed me to setup a very basic tungsten just to see things working.

Tungsten supports all sorts of “topologies”. For this example I am just doing master-slave. There are two nodes and each has mysql running on it. Node1 serves as the master, and runs the tungsten replicator (master node service). And node2 serves as the slave and runs the tungsten applier (slave node service).

Good luck and hope this helps others speedup the learning curve!

1. Download tarball (on master)

Note the forums indicated they may be moving off of google code. So this url may change.


$ cd /tmp
$ wget http://downloads.tungsten-replicator.org/download.php?file=tungsten-replicator-oss-4.0.0-18.tar.gz

Also: Why Airbnb didn’t have to fail

2. Expand the tarball (on master)

Use your vast unix skills to expand the tarball!


$ mv download.php\?file\=tungsten-replicator-oss-4.0.0-18.tar.gz tungsten.tgz
$ tar xvzf tungsten.tgz
$ mv tungsten-replicator-4.0.0-18 stage

Also: Is the difference between dev & ops a four-letter word?

3. Install MySQL (each box)

Hopefully you’ve done this before. Pretty straightforward:


$ apt-get install mysql-client mysql-server

Also: Are SQL Databases Dead?

4. Edit cookbook files (on master)

Inside /tmp/stage/cookbook you’ll need to make a few simple edits:

edit COMMON_NODES.sh

Edit NODE1 and NODE2 and comment out other lines.


export NODE1=ip-172-31-0-117
export NODE2=ip-172-31-1-188

edit NODES_MASTER_SLAVE.sh


export MASTERS=($NODE1)
export SLAVES=($NODE2)

USER_VALUES.sh


export TUNGSTEN_BASE=/opt/continuent
export MY_CNF=/etc/mysql/my.cnf
export DATABASE_USER=sync
export DATABASE_SUPER_USER=root
export DATABASE_PASSWORD=

Also: 5 Things toxic to scalability

5. Create install directory (each box)

The /tmp/stage directory you created above is just a holding ground for the tarball. Continuent in it’s infinite wisdom wants to install itself. So, create a directory for it:

As root:


$ mkdir /opt/continuent
$ chown tungsten /opt/continuent

Then as tungsten:


$ touch /opt/continuent/testfile.txt

Also: How to deploy on Amazon EC2 with Vagrant?

6. Configure aws security groups

AWS security group permissions are key to getting any of this tungsten stuff working. And there are a lot of moving parts here. Ping is required for various tests, as is MySQL’s port. But you’ll also need to enable Tungsten History Log port 2112 and the replicator ports.

o ping ICMP from your group
o enable inbound 3306 from your group
o enable THL – inbound 2112 from your group
o enable RMI – inbound 10000, 10001 from your group

Also: Howto interview an AWS expert for managers, recruiters & engineers alike

7. Test mysql client from the other (each box)


$ mysql -h ip-172-31-1-188 -u root

$ mysql -h ip-172-31-0-117 -u root

If these hang, verify 3306 in your aws security groups. If you get a mysql error, be sure the “host” is appropriate. Check with:


mysql> select user, host, password from mysql.user where user = 'root';

For more info see Connect to MySQL in the Amazon public cloud.

Also: 5 Reasons to move to amazon redshift

8. Setup ssh auto-login (each box)

You need to be able to login to each box as a user “tungsten” without a password.


$ adduser tungsten
$ ssh-keygen
$ scp .ssh/id_rsa.pub tungsten@ip-172-31-1-188:/home/tungsten/.ssh/authorized_keys

Be sure the authorized_keys file is 600:


$ chmod 600 .ssh/authorized_keys

Also: Did MySQL & Mongo have a beautiful baby called Aurora

9. Install ruby (each box)

You need to have ruby on both boxes.


$ apt-get install ruby

Also: Top interview questions for a MySQL expert – recruiters, managers & candidates

10. Create sync user (each box)

Both of your mysql instances will need a user that tungsten connects as. I called it “sync”.


mysql> create user 'sync'@'%' identified by 'secret';
mysql> grant all privileges on *.* to 'sync'@'%';
mysql> flush privileges;

Note, you may want to use a blank password at this stage, to eliminate that as a potential problem to debug. Also consider the security implications of ‘%’ and consider a subnet wildcard such as ‘10.0.%’.

Also: Myth of five nines why high availability is overrated

11. Enable binary log (each box)

Enable the mysql binary log with log_bin parameter. Also ensure that /var/lib/mysql is readable by the tungsten user, either by changing /var/lib to 655 or adding tungsten to the mysql group. Test this as well using less or cat.

Also: What is high availability and why is it important?

12. Update MySQL startup settings (each box)

Fire up your favorite editor and update /etc/mysql/my.cnf settings (both servers). The following are the main ones:


server_id = 1 (server_id=2 for slave)
sync_binlog = 1
max_allowed_packet = 52m
open_files_limit = 65535
innodb_log_file_size=50m
#bind_address localhost (comment it to disable)

Note that changing innodb_log_file_size is tricky. You’ll need to stop mysql, rename old ib_logfile0 to ib_logfile0.old and ib_logfile1 to ib_logfile1.old. Then change the param in my.cnf and start mysql. Otherwise you’ll get errors on startup.

Also: Is the difference between dev & ops a four-letter word?

13. Run the installer (on master)

This step is fairly straightforward. If there are problems in this step, you’ll see ERROR lines in the output. Sift through them and resolve one by one.


$ ./cookbook/install_master_slave

Also: RDS or MySQL – Ten use cases

14. Check tungsten status (each box)

The “trepctl” utility allows you to check the current status. It does a lot more, but for now that’s enough. If you want to make it easier add “/opt/continuent/tungsten/tungsten-replicator/bin” to your path.

Also notice the line “state”. It should be ONLINE. If it says “GOING-ONLINE:SYNCHRONIZING” that likely means you didn’t open up the tungsten ports. You’ll need both RMI ports 10000 and 10001 as well as THL ports 2112. We all know how finicky AWS security groups can be. I’ll leave it to your own exercise to confirm those are open.

ON MASTER


root@ip-172-31-0-117:/opt/continuent/tungsten/tungsten-replicator/bin# ./trepctl status
Processing status command...
NAME VALUE
---- -----
appliedLastEventId : mysqld-bin.000005:0000000000000321;235
appliedLastSeqno : 5
appliedLatency : 34824.086
autoRecoveryEnabled : false
autoRecoveryTotal : 0
channels : 1
clusterName : cookbook
currentEventId : mysqld-bin.000005:0000000000000321
currentTimeMillis : 1432840323683
dataServerHost : ip-172-31-0-117
extensions :
host : ip-172-31-0-117
latestEpochNumber : 0
masterConnectUri : thl://localhost:/
masterListenUri : thl://ip-172-31-0-117:2112/
maximumStoredSeqNo : 5
minimumStoredSeqNo : 0
offlineRequests : NONE
pendingError : NONE
pendingErrorCode : NONE
pendingErrorEventId : NONE
pendingErrorSeqno : -1
pendingExceptionMessage: NONE
pipelineSource : /var/lib/mysql
relativeLatency : 44450.683
resourcePrecedence : 99
rmiPort : 10000
role : master
seqnoType : java.lang.Long
serviceName : cookbook
serviceType : local
simpleServiceName : cookbook
siteName : default
sourceId : ip-172-31-0-117
state : ONLINE
timeInStateSeconds : 82860.917
timezone : GMT
transitioningTo :
uptimeSeconds : 82861.492
useSSLConnection : false
version : Tungsten Replicator 4.0.0 build 18
Finished status command...
root@ip-172-31-0-117:/opt/continuent/tungsten/tungsten-replicator/bin#

ON SLAVE


tungsten@ip-172-31-1-188:/opt/continuent/tungsten/tungsten-replicator/bin$ ./trepctl status
Processing status command...
NAME VALUE
---- -----
appliedLastEventId : mysqld-bin.000005:0000000000000321;235
appliedLastSeqno : 5
appliedLatency : 42569.62
autoRecoveryEnabled : false
autoRecoveryTotal : 0
channels : 1
clusterName : cookbook
currentEventId : NONE
currentTimeMillis : 1432840348936
dataServerHost : ip-172-31-1-188
extensions :
host : ip-172-31-1-188
latestEpochNumber : 0
masterConnectUri : thl://ip-172-31-0-117:2112/
masterListenUri : thl://ip-172-31-1-188:2112/
maximumStoredSeqNo : 5
minimumStoredSeqNo : 0
offlineRequests : NONE
pendingError : NONE
pendingErrorCode : NONE
pendingErrorEventId : NONE
pendingErrorSeqno : -1
pendingExceptionMessage: NONE
pipelineSource : thl://ip-172-31-0-117:2112/
relativeLatency : 44475.936
resourcePrecedence : 99
rmiPort : 10000
role : slave
seqnoType : java.lang.Long
serviceName : cookbook
serviceType : local
simpleServiceName : cookbook
siteName : default
sourceId : ip-172-31-1-188
state : ONLINE
timeInStateSeconds : 1906.6
timezone : GMT
transitioningTo :
uptimeSeconds : 82882.334
useSSLConnection : false
version : Tungsten Replicator 4.0.0 build 18
Finished status command...
tungsten@ip-172-31-1-188:/opt/continuent/tungsten/tungsten-replicator/bin$

Also: Is zero downtime even possible with Amazon RDS?

15. Perform simple test

Create a table on the master mysql node.


mysql master> create database test;
mysql master> create table test.sean (c1 varchar(64));
mysql master> insert into test.sean values ('hi there');
mysql master> insert into test.sean values ('this should show up in tungsten thl file');
mysql master> insert into test.sean values ('new data on tungsten02 THL??');

Verify that the table is on the slave mysql node.


mysql slave> show databases;
mysql slave> select * from test.sean;
+------------------------------------------+
| c1 |
+------------------------------------------+
| hi there |
| this should show up in tungsten thl file |
| new data on tungsten02 THL?? |
+------------------------------------------+
3 rows in set (0.00 sec)

Success!

Also: Why are MySQL & other database experts so hard to find?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters