INTRODUCTION
————
Let me start by saying that this whole article is rather unorthodox.
That’s why I thought the analogy of hitchhiking was so apt. Also
if you are a hitchhiker, and you happen to be carrying your laptop,
well you can bring along an Oracle 10g Real Application Cluster
to show all your friends. Now there’s a road warrior!
Seriously though, this step-by-step guide does not describe any
supported solution by Oracle. So what good is it? Well a lot
actually. By taking the uncommon route, you often see the
colorful streets, the surprising highways, and undiscovered
nooks. What a great analogy because this is also true in software!
In 2002 I went to Open World and was very excited by the discovery
of the Open Source initiative headed up by Wim Coekerts. One of
their most exciting projects to me was this Oracle 9i RAC running
on Linux with a special Firewire driver patched to allow multiple
systems to mount the same filesystem. That was a tremendous learning
experience, and I wrote up an article, and presented that at the
New York Oracle User Group. Afterward, someone from the audience
came to me and asked if I would present at their user group out
in Edison New Jersey. Both presentations were exciting, and I
think well received.
Of late I’ve happened upon some articles floating around the internet
discussing a single-node AKA laptop RAC setup. How could that work,
I wondered? Perhaps they install virtualization software such as
VMWare to allow a single machine to look like more than one. This
would certainly work in theory. Once I started digging a bit more I
discovered Amit Poddar’s excellent article over at dizwell.com, and
I was intrigued. No virtualization seemed to be required other than
some virtual ethernet interfaces.
I decided to dig my heels in and give it a try. After struggling with
the illustrious and very universally beloved installer for a few months
I finally managed to get all the pieces in place, and get Oracle Real
Application Clusters running with no clustering hardware!!!
Here, my fine friends, are all the gory details of that adventure, which
I hope you’ll enjoy as much as I did writing them down.
Oracle’s model of clustering involves multiple instances (software processes)
talking to a single database (physical datafiles). We’re doing the same
thing here, but both instances will reside on the same machine. This
helps you travel light, save on transportation and learn concepts,
commands, and the architecture. Remember this is not an HA solution,
as there is little redundancy, and with a single disk, you will surely get
abysmal performance.
Let’s get started, what will we need to do? Here’s a quick outline of the
steps involved:
1. setup ip addresses of the virtual servers
2. setup ssh and rsh with autologin configured
3. setup the raw devices Oracle’s ASM software will use
4. install the clusterware softare, and then Oracle’s 10g software
5. setup the listener and an ASM instance
6. create an instance, start it, and register with srvctl
7. create a second instance & undo tablespace, & register it
1. Setup IP Addresses
———————-
Oracle wants to have a few interfaces available to it. To follow our analogy
of a hitchhiker traveling across America, we’ll name our server route66.
So add that name to your /etc/hosts/ file along with the private and vip
names:
192.168.0.19 route66
192.168.0.75 route66-priv
#192.168.0.76 route66-vip
Notice that we’ve commented out route66-vip. We’ll explain more about
this later, but suffice it to say now that the clusterware installer
is very finicky about this.
In order for these two additional names to be reachable, we need
ethernet devices to associate with those IPs. It’s a fairly straightforward
thing to create with ifconfig as follows:
$ /sbin/ifconfig eth0:1 192.168.0.75 netmask 255.255.255.0 broadcast 192.168.0.255
$ /sbin/ifconfig eth0:2 192.168.0.76 netmask 255.255.255.0 broadcast 192.168.0.255
If your IPs, or network is configured differently, adjust the IP or broadcast
address accordingly.
2. setup ssh and rsh with autologin
————————————
Most modern Linux systems do *NOT* come with rsh installed. That’s for
good reason, because it’s completely insecure, and shouldn’t be used at
all. Why Oracle’s installer requires it is beyond me, but you’ll need
it. You can probably disable it once the clusterware is installed.
Head over to http://rpmfind.net and see if you can find a copy for
your distro. You might also have luck using up2date or yumm if you
already have those configured, as they handle dependencies, and always
download the *right* version. With rpm, install this way:
$ rpm Uvh rsh-server-0.17-34.1.i386.rpm
$ rpm Uvh rsh-0.17-34.1.i386.rpm
Next enable autologin by adding names to your /home/oracle/.rhosts file.
After starting rsh, you should be able to login as follows:
$ rsh route66-priv
Once that works, move on the the sshd part. Most likely ssh is already on
your system, so just start it (as root):
$ /etc/rc.d/init.d/sshd start
Next, as the “oracle” user, generate the keys:
$ ssh-keygen -t dsa
Normally you would copy id_dsa.pub to a remote system, but for us we
just want to login to self. So copy as follows:
$ cd .ssh
$ cp id_dsa.pub authorized_keys
$ chmod 644 authorized_keys
Verify that you can login now:
$ ssh route66-priv
3. setup the raw devices
————————-
Most of the time when you think of files on a Unix system, you’re
thinking of files as represented through a filesystem. A filesystem
provides you a way to interact with the underlying disk hardware
through the use of files. A filesystem provides buffering, to
improve I/O performance automatically. However in the case of a
database like Oracle, it already has a sophisticated mechanism for
buffering which is smart in that it knows everything about it’s
files, and how it wants to read and write to them. So for an
application like Oracle unbuffered I/O is ideal. It bypasses a
whole layer of software, making your overall throughput faster!
You achieve this feat of magic using raw devices. We’re going to
hand them over to Oracle’s Automatic Storage Manager in a minute
but first let’s get to work creating the device files for our
RAC setup.
Create three 2G disks. These will be used as general storage space
for our ASM instance:
$ mkdir /asmdisks
$ dd if=/dev/zero of=/asmdisks/disk1 bs=1024k count=2000
$ dd if=/dev/zero of=/asmdisks/disk2 bs=1024k count=2000
$ dd if=/dev/zero of=/asmdisks/disk3 bs=1024k count=2000
Create two more smaller disks, one for the Oracle Cluster Registry,
and another for the voting disk:
$ dd if=/dev/zero of=/asmdisks/disk4 bs=1024k count=100
$ dd if=/dev/zero of=/asmdisks/disk5 bs=1024k count=20
Now we use a loopback device to make Linux treat these FILES as
raw devices.
$ /sbin/losetup /dev/loop1 /asmdisks/disk1
$ raw /dev/raw/raw1 /dev/loop1
$ chown oracle.dba /dev/raw/raw1
You’ll want to run those same three commands on disk2 through disk5 now.
4. Install the Clusterware & Oracle’s 10g Software
————————————————–
Finally we’re done with the Operating System setup, and we can move on
to Oracle. The first step will be to install the clusterware. I’ll
tell you in advance that this was the most difficult step in the entire
RAC on a laptop saga. Oracle’s installer tries to *HELP* you all along
the way, which really means standing in front of you!
First let’s make a couple of symlinks to our OCR and voting disks:
$ ln -sf /dev/raw/raw4 /home/oracle/product/disk_ocr
$ ln -sf /dev/raw/raw5 /home/oracle/product/disk_vot
As with any Oracle install, you’ll need a user, and group already
created, and you’ll want to set the usual environment variables such
as ORACLE_HOME, ORACLE_SID, etc. Remember that previous to this point
you already have ssh and rsh autologin working. If you’re not sure
go back and test again. That will certainly hold you up here, and
give you all sorts of confusing error messages.
If you’re running on an uncertified version of Linux, you may want
to fire up the clusterware installer as follows:
$ ./runInstaller -ignoreSysPrereqs
If your Linux distro is still giving you trouble, you might try
downloading from centos.org where you can find complete ISOs for
RHEL, various versions. You can also safely ignore memory warnings
during startup. If you’re short on memory, it will certainly slow things
down, but we’re hitchhikers right?
You’ll be asked to specify the cluster configuration details. You’ll
want route66-vip to be commented out, so if you haven’t done that and
get an error to the affect of route66-vip already in use go ahead and
edit your /etc/hosts file.
I also got messages saying “route66-priv not reachable”. Check again
that sshd is running, and possibly disable your firewall rules:
$ /etc/rc.d/init.d/iptables stop
Also verify that eth0:1, and eth0:2 are created. Have you rebooted
since you created them? Be sure they’re still there with:
$ /sbin/ifconfig -a
Specify the network interface. This defaults to PRIVATE, just edit
and specify PUBLIC.
The next two steps ask for the OCR disk and voting disk. Be sure to
specify external redundancy. This is your way of telling Oracle that
you’ll take care of mirroring these important disks yourself, as loss
of either of them will get you in deep doodoo. Of course we’re
hitchhikers so we’re not trying to build a system that is never going
to breakdown, but rather we want to get the feeling of the wind blowing
in our hair. Click through to install and you should be in good shape.
At the completion, the installer will ask you to run the root.sh
script. I found this worked fine up until the vipca (virtual ip
configuration assistant). I then ran this one manually. You’ll need
to uncomment route66-vip from your /etc/hosts file as well. Once
all configuration assistants have completed successfully, return to
the installer and click continue, and it will do various other sanity
checks of your cluster configuration.
Since the clusterware install is rather testy, you’ll probably be doing
it a few times before you get it right. Here’s the cleanup if you
have to run through it again:
$ rm rf /etc/oracle
$ rm rf /home/oracle/oraInventory
$ rm rf $CRS_HOME
# these two commands cleanup the contents of these disks
$ /bin/dd if=/dev/zero of=/asmdisks/disk4 bs=1024k count=100
$ /bin/dd if=/dev/zero of=/asmdisks/disk5 bs=1024k count=20
$ rm /etc/rc.d/init.d/init.crs
$ rm /etc/rc.d/init.d/init.crsd
$ rm /etc/rc.d/init.d/init.cssd
$ rm /etc/rc.d/init.d/init.evmd
$ rm /etc/rc.d/rc3.d/S96init.crs
$ rm /etc/rc.d/rc5.d/S96init.crs
Now reboot the server. This will kill any clusterware processes still running.
If you’ve finished the Oracle Universal Installer at this point, and things
seem to be working, check at the command line with the ps command:
$ ps auxw | grep 10.2.0s
root 3728 0.0 0.3 2172 708 ? S May30 0:00 /bin/su -l oracle -c sh -c ‘ulimit -c unlimited; cd /home/oracle/product/10.2.0s/log/bebel/evmd; exec /home/oracle/product/10.2.0s/bin/evmd ‘
root 3736 0.0 5.0 509608 11240 ? S May30 4:51 /home/oracle/product/10.2.0s/bin/crsd.bin reboot
oracle 4047 0.0 2.8 192136 6284 ? S May30 0:00 /home/oracle/product/10.2.0s/bin/evmd.bin
root 4172 0.0 0.3 2164 708 ? S May30 0:00 /bin/su -l oracle -c /bin/sh -c ‘ulimit -c unlimited; cd /home/oracle/product/10.2.0s/log/bebel/cssd; /home/oracle/product/10.2.0s/bin/ocssd || exit $?’
oracle 4173 0.0 0.4 4180 900 ? S May30 0:00 /bin/sh -c ulimit -c unlimited; cd /home/oracle/product/10.2.0s/log/bebel/cssd; /home/oracle/product/10.2.0s/bin/ocssd || exit $?
oracle 4234 0.0 3.9 180108 8808 ? S May30 0:16 /home/oracle/product/10.2.0s/bin/ocssd.bin
oracle 4476 0.0 2.0 24048 4576 ? S May30 0:00 /home/oracle/product/10.2.0s/bin/evmlogger.bin -o /home/oracle/product/10.2.0s/evm/log/evmlogger.info -l /home/oracle/product/10.2.0s/evm/log/evmlogger.log
oracle 6989 0.0 0.1 2676 404 ? S May30 0:00 /home/oracle/product/10.2.0s/opmn/bin/ons -d
oracle 6990 0.0 1.9 93224 4280 ? S May30 0:00 /home/oracle/product/10.2.0s/opmn/bin/ons -d
oracle 7891 0.0 0.2 3676 660 pts/3 S 23:35 0:00 grep 10.2.0s
Also list the nodes you have available to your clusterware software:
$ olsnodes n
route66 1
The Oracle 10g install itself is very trivial, assuming you’ve installed
Oracle before. Use the -ignoreSysPrereqs flag if necessary to start
up the Oracle Universal Installer, and use the software-only option,
as we’ll be creating our RAC database by hand. Also select Enterprise
Edition and things should proceed smoothly. Oracle will recognize that
you have the clusterware installed, and let you know during the
installation.
5. setup the listener and an ASM instance
——————————————-
The listener.ora file is setup as usual, the only difference is you will
include both route66 and route66-vip.
SID_LIST_LISTENER =
(SID_LIST =
(SID_DESC =
(SID_NAME = PLSExtProc)
(ORACLE_HOME = /home/oracle/product/10.2.0)
(PROGRAM = extproc)
)
)
LISTENER =
(DESCRIPTION_LIST =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC))
)
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL=TCP)(HOST=route66)(PORT = 1521)(IP = FIRST))
)
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL=TCP)(HOST=route66-vip)(PORT = 1521)(IP = FIRST))
)
)
)
You may also choose to use the actual IP addresses of these hostnames if
you like. One other difference is that you will use Oracle 10g’s new
srvctl utility to start th listener.
$ srvctl start nodeapps -n route66
Ok, on to the fun stuff. It’s time to configure our ASM instance.
Our instance name will be +ASM1, and we’ll set the ORACLE_SID as usual.
In the init+ASM1.ora file specify:
user_dump_dest=/home/oracle/admin/+ASM1/udump
background_dump_dest=/home/oracle/admin/+ASM1/bdump
core_dump_dest=/home/oracle/admin/+ASM1/cdump
large_pool_size=15m
instance_type=asm
asm_diskstring=’/dev/raw/raw1′, ‘/dev/raw/raw2′,’/dev/raw/raw3’
Then in sqlplus startup the ASM instance:
SQL> startup nomount
Next tell ASM how we want to utilize the space we have by creating disk
groups:
SQL> create diskgroup DBDATA external redundancy disk
‘/dev/raw/raw1’,’/dev/raw/raw2;
SQL> create diskgroup DBRECO external redundancy disk ‘/dev/raw/raw3’;
Now that you have diskgroups, you want to make a note of it in your
init.ora:
SQL> !echo “asm_diskgroups=’DBDATA’,’DBRECO'” >> init+ASM1.ora
Now startup your ASM instance:
SQL> startup force
Exit sqlplus and let srvctl know about the new ASM instance:
$ srvctl add asm -n route66 -i +ASM1 -o /home/oracle/product/10.2.0.1
Now you can shutdown in sqlplus, and startup with srvctl:
SQL> shutdown immediate
$ srvctl start asm -n route66
And lastly use the ps command to check for your new instance.
6. create an instance, start it, and register with srvctl
———————————————————
We’re getting to our clustered database slowly but surely. We’re
just getting over the mountains now, and the open road is ahead of
us.
We’re going to create the first of our two instances and call it
BEATNIK. Edit your initBEATNIK.ora as follows:
db_block_size=8192
db_multiblock_read_count=8
db_name=kerouac
BEATNIK.background_dump_dest=/home/oracle/admin/BEATNIK/bdump
BEATNIK.user_dump_dest=/home/oracle/admin/BEATNIK/udump
BEATNIK.core_dump_dest=/home/oracle/admin/BEATNIK/cdump
BEATNIK.instance_number=1
BEATNIK.instance_name=BEATNIK
BEATNIK.thread=1
BEATNIK.undo_tablespace=beatnikundo
HIPPY.background_dump_dest=/home/oracle/admin/BEATNIK/bdump
HIPPY.user_dump_dest=/home/oracle/admin/BEATNIK/udump
HIPPY.core_dump_dest=/home/oracle/admin/BEATNIK/cdump
HIPPY.instance_number=2
HIPPY.instance_name=HIPPY
HIPPY.thread=2
HIPPY.undo_tablespace=hippyundo
You’ll also have to create the /home/oracle/admin/BEATNIK/* and
/home/oracle/admin/HIPPY/* directories.
Now edit the file crKEROUAC.sql as follows:
CREATE DATABASE KEROUAC”
DATAFILE SIZE 250M EXTENT MANAGEMENT LOCAL
SYSAUX DATAFILE SIZE 125M
DEFAULT TEMPORARY TABLESPACE TEMP TEMPFILE SIZE 20M
UNDO TABLESPACE “beatnikundo” DATAFILE SIZE 200M
CHARACTER SET WE8ISO8859P1
LOGFILE GROUP 1 SIZE 10240K,
GROUP 2 size 10240k,
GROUP 3 size 10240k;
There are other parameters you can specify in this create statement, such as
maxinstances, maxlogmembers, and the sys password. However I’ve tried to
simplify it, to make it easier to review and understand. Check the Oracle
docs for details.
Now startup sqlplus and issue:
SQL> startup nomount pfile=/home/oracle/admin/BEATNIK/pfile/initBEATNIK.ora
SQL> @crKEROUAC.sql
Now get the names of your controlfiles from v$parameter and add them to
the initBEATNIK.ora file.
Now add a couple more parameters to your initBEATNIK.ora file:
*.cluster_database=true
*.cluster_database_instances=5
Use sqlplus to stop and start the db again:
SQL> shutdown immediate
SQL> startup force
Now register our new database:
$ srvctl add database d KEROUAC -o /home/oracle/product/10.2.0.1/
$ srvctl add instance -d KEROUAC -i BEATNIK -n route66
Now one more time, shutdown with sqlplus, and then use srvctl to
start the db. From now on srvctl can stop + start the db.
$ srvctl start instance -d KEROUAC -i BEATNIK
7. create a second instance & undo tablespace, & register it
———————————————————–
Since the database is already created, you don’t have to do that step
again. At this point all you have to do is create another instance.
First fire up sqlplus and create another undo tablespace:
SQL> create undo tablespace hippyundo datafile ‘+DBDATA’ size 100m;
Make a copy of your init.ora for the hippy instance like this:
$ cp initBEATNIK.ora initHIPPY.ora
Then set your ORACLE_SID and use sqlplus to startup:
SQL> startup
Finally register with srvctl:
$ srvctl add instance -i HIPPY -d KEROUAC -n route66
8. create data dictionary
————————–
SQL>@?/rdbms/admin/catalog.sql
SQL>@?/rdbms/admin/catclust.sql
10. Some Things to Understand
————————–
Automatic Storage Management
Global Cache Services
Global Enqueue Services
Clusterware procs crsd, evmd, ocssd,oprocd
RAC processes lms, lmd, lmon, lck0
GV$ data dictionary
11. Further Reading
—————–
Clusterware & RAC Install & Configuration Guide for Linux
Clusterware & RAC Administration Deployment Guide
Oracle Technology Network – http://otn.oracle.com
Please visit http://www.iheavy.com or email me at
Thanks to Amit Poddar & dizwell.com
Apress, Oracle Press books