Category Archives: Technical Article

Part 9: RAC/Linux/Firewire – A Quick 9iRAC Example

A quick 9iRAC example


The names of the two instances I’m using are EAST and WEST, so I’ll use them here to refer to commands you’ll execute at the sqlplus prompt. This test assumes you’re logged into the same schema on both instances. I used ‘sys’ but you can create your own schema if you like.

1. On WEST do:

SQL> create table rac_test (c1 number, c2 varchar2 (64));

2. On EAST do:

SQL> desc rac_test

3. On WEST do:

SQL> insert into rac_test values (1, ‘SEAN’);

SQL> insert into rac_test values (2, ‘MIKE’);

SQL> insert into rac_test values (3, ‘JANE’);

4. On EAST do: (notice no rows are returned)

SQL> select * from rac_test;

5. On WEST do:

SQL> commit;

6. On EAST do: (notice the rows appear now)

SQL> select * from rac_test;

7. On WEST do:

SQL> update rac_test set c2 = ‘SHAWN’ where c1 = 1;

8. On EAST do: (notice the error Oracle returns)

SQL> select * from rac_test where c1 = 1 for update nowait;

select * from rac_test where c1 = 1 for update nowait

*

ERROR at line 1:

ORA-00054: resource busy and acquire with NOWAIT specified

9. Again on EAST do: (notice Oracle waits…)

SQL> update rac_test set c2 = ‘JOE’ where c1 = 1;

10. On WEST do:

SQL> commit;

11. On EAST the transaction completes.

This simple exercise illustrates that two sessions running on different instances, against the same database are behaving just like two sessions on a single instance or machine against a database. This is key. Oracle must maintain transactional consistency. Oracle maintains ACID properties which are Atomicity, Consistency, Isolation, and Duarability. Atomicity means a transaction either executes to completion, or fails. Consistency means that the database operates in discrete transactions, and moves from one consistent state to another. Isolation means that actions of other transactions are invisible until they are completed (commited). Finally Durability means when a transaction has finally completed and commited, it becomes permanent.

Our example above demonstrates that Oracle maintains all these promises, even in a clustered environment. How Oracle does this behind the scenes involves the null, shared, and exclusive locks we described above, and current and past image management. A lot of these details are reserved for a 9iRAC internals article. Take a look below at Madhu Tumma’s article for more on 9iRAC internals.

Part 1 – Introduction

Part 2 – Basic Costs + Hardware Platform Outline

Part 3 – Software Requirements, Versions, etc

Part 4 – Initial Oracle Setup

Part 5 – Firewire + OCFS Setup

Part 6 – Cluster Manager Setup

Part 7 – Cluster Database Setup

Part 8 – Review of Clustered Features + Architecture

Part 9 – A quick 9iRAC example

Part 10 – Summary

Part 10: RAC/Linux/Firewire – Summary

Summary

We covered a lot of ground in this article and it should serve as an introduction to 9iRAC on cheap Linux hardware. There are plenty of other topics to dig into including tuning, backup, SQL*Net setup and Load Balancing, I/O Fencing, NIC Failover and so on.

9iRAC is *NOT* a silver bullet as any good DBA knows. It will protect you from a single instance failure because of memory, kernel panic, or an interconnect failure, but there are still cases where your database could go down, for instance if the cluster manager software fails, or you lose a datafile either from human error, or a storage subsystem problem. Further redundancy can help you, but there are risks associated with an accidentally deleted object, or even and Oracle software bug.

Take a look at some of the documents listed below for further reading.

Other References

Red Hat Linux Advanced Server 2.1 – docs

Oracle Technology Network’s Linux Center

Oracle Cluster Filesystem FAQ

Oracle Cluster File System docs

Internals of Real Application Clusters by Madhu Tumma

Oracle9i Real Application Clusters Concepts

Oracle9i Real Application Clusters Administration

Linux HOWTO Docs – Networking, kernel config, hardware compatability etc


Part 1 – Introduction

Part 2 – Basic Costs + Hardware Platform Outline

Part 3 – Software Requirements, Versions, etc

Part 4 – Initial Oracle Setup

Part 5 – Firewire + OCFS Setup

Part 6 – Cluster Manager Setup

Part 7 – Cluster Database Setup

Part 8 – Review of Clustered Features + Architecture

Part 9 – A quick 9iRAC example

Part 10 – Summary

Oracle 10g Laptop/nodeless RAC Howto

INTRODUCTION

————

Let me start by saying that this whole article is rather unorthodox.

That’s why I thought the analogy of hitchhiking was so apt. Also

if you are a hitchhiker, and you happen to be carrying your laptop,

well you can bring along an Oracle 10g Real Application Cluster

to show all your friends. Now there’s a road warrior!

Seriously though, this step-by-step guide does not describe any

supported solution by Oracle. So what good is it? Well a lot

actually. By taking the uncommon route, you often see the

colorful streets, the surprising highways, and undiscovered

nooks. What a great analogy because this is also true in software!

In 2002 I went to Open World and was very excited by the discovery

of the Open Source initiative headed up by Wim Coekerts. One of

their most exciting projects to me was this Oracle 9i RAC running

on Linux with a special Firewire driver patched to allow multiple

systems to mount the same filesystem. That was a tremendous learning

experience, and I wrote up an article, and presented that at the

New York Oracle User Group. Afterward, someone from the audience

came to me and asked if I would present at their user group out

in Edison New Jersey. Both presentations were exciting, and I

think well received.

Of late I’ve happened upon some articles floating around the internet

discussing a single-node AKA laptop RAC setup. How could that work,

I wondered? Perhaps they install virtualization software such as

VMWare to allow a single machine to look like more than one. This

would certainly work in theory. Once I started digging a bit more I

discovered Amit Poddar’s excellent article over at dizwell.com, and

I was intrigued. No virtualization seemed to be required other than

some virtual ethernet interfaces.

I decided to dig my heels in and give it a try. After struggling with

the illustrious and very universally beloved installer for a few months

I finally managed to get all the pieces in place, and get Oracle Real

Application Clusters running with no clustering hardware!!!

Here, my fine friends, are all the gory details of that adventure, which

I hope you’ll enjoy as much as I did writing them down.

Oracle’s model of clustering involves multiple instances (software processes)

talking to a single database (physical datafiles). We’re doing the same

thing here, but both instances will reside on the same machine. This

helps you travel light, save on transportation and learn concepts,

commands, and the architecture. Remember this is not an HA solution,

as there is little redundancy, and with a single disk, you will surely get

abysmal performance.

Let’s get started, what will we need to do? Here’s a quick outline of the

steps involved:

1. setup ip addresses of the virtual servers

2. setup ssh and rsh with autologin configured

3. setup the raw devices Oracle’s ASM software will use

4. install the clusterware softare, and then Oracle’s 10g software

5. setup the listener and an ASM instance

6. create an instance, start it, and register with srvctl

7. create a second instance & undo tablespace, & register it

1. Setup IP Addresses

———————-

Oracle wants to have a few interfaces available to it. To follow our analogy

of a hitchhiker traveling across America, we’ll name our server route66.

So add that name to your /etc/hosts/ file along with the private and vip

names:

192.168.0.19 route66

192.168.0.75 route66-priv

#192.168.0.76 route66-vip

Notice that we’ve commented out route66-vip. We’ll explain more about

this later, but suffice it to say now that the clusterware installer

is very finicky about this.

In order for these two additional names to be reachable, we need

ethernet devices to associate with those IPs. It’s a fairly straightforward

thing to create with ifconfig as follows:

$ /sbin/ifconfig eth0:1 192.168.0.75 netmask 255.255.255.0 broadcast 192.168.0.255

$ /sbin/ifconfig eth0:2 192.168.0.76 netmask 255.255.255.0 broadcast 192.168.0.255

If your IPs, or network is configured differently, adjust the IP or broadcast

address accordingly.

2. setup ssh and rsh with autologin

————————————

Most modern Linux systems do *NOT* come with rsh installed. That’s for

good reason, because it’s completely insecure, and shouldn’t be used at

all. Why Oracle’s installer requires it is beyond me, but you’ll need

it. You can probably disable it once the clusterware is installed.

Head over to http://rpmfind.net and see if you can find a copy for

your distro. You might also have luck using up2date or yumm if you

already have those configured, as they handle dependencies, and always

download the *right* version. With rpm, install this way:

$ rpm Uvh rsh-server-0.17-34.1.i386.rpm

$ rpm Uvh rsh-0.17-34.1.i386.rpm

Next enable autologin by adding names to your /home/oracle/.rhosts file.

After starting rsh, you should be able to login as follows:

$ rsh route66-priv

Once that works, move on the the sshd part. Most likely ssh is already on

your system, so just start it (as root):

$ /etc/rc.d/init.d/sshd start

Next, as the “oracle” user, generate the keys:

$ ssh-keygen -t dsa

Normally you would copy id_dsa.pub to a remote system, but for us we

just want to login to self. So copy as follows:

$ cd .ssh

$ cp id_dsa.pub authorized_keys

$ chmod 644 authorized_keys

Verify that you can login now:

$ ssh route66-priv

3. setup the raw devices

————————-

Most of the time when you think of files on a Unix system, you’re

thinking of files as represented through a filesystem. A filesystem

provides you a way to interact with the underlying disk hardware

through the use of files. A filesystem provides buffering, to

improve I/O performance automatically. However in the case of a

database like Oracle, it already has a sophisticated mechanism for

buffering which is smart in that it knows everything about it’s

files, and how it wants to read and write to them. So for an

application like Oracle unbuffered I/O is ideal. It bypasses a

whole layer of software, making your overall throughput faster!

You achieve this feat of magic using raw devices. We’re going to

hand them over to Oracle’s Automatic Storage Manager in a minute

but first let’s get to work creating the device files for our

RAC setup.

Create three 2G disks. These will be used as general storage space

for our ASM instance:

$ mkdir /asmdisks

$ dd if=/dev/zero of=/asmdisks/disk1 bs=1024k count=2000

$ dd if=/dev/zero of=/asmdisks/disk2 bs=1024k count=2000

$ dd if=/dev/zero of=/asmdisks/disk3 bs=1024k count=2000

Create two more smaller disks, one for the Oracle Cluster Registry,

and another for the voting disk:

$ dd if=/dev/zero of=/asmdisks/disk4 bs=1024k count=100

$ dd if=/dev/zero of=/asmdisks/disk5 bs=1024k count=20

Now we use a loopback device to make Linux treat these FILES as

raw devices.

$ /sbin/losetup /dev/loop1 /asmdisks/disk1

$ raw /dev/raw/raw1 /dev/loop1

$ chown oracle.dba /dev/raw/raw1

You’ll want to run those same three commands on disk2 through disk5 now.

4. Install the Clusterware & Oracle’s 10g Software

————————————————–

Finally we’re done with the Operating System setup, and we can move on

to Oracle. The first step will be to install the clusterware. I’ll

tell you in advance that this was the most difficult step in the entire

RAC on a laptop saga. Oracle’s installer tries to *HELP* you all along

the way, which really means standing in front of you!

First let’s make a couple of symlinks to our OCR and voting disks:

$ ln -sf /dev/raw/raw4 /home/oracle/product/disk_ocr

$ ln -sf /dev/raw/raw5 /home/oracle/product/disk_vot

As with any Oracle install, you’ll need a user, and group already

created, and you’ll want to set the usual environment variables such

as ORACLE_HOME, ORACLE_SID, etc. Remember that previous to this point

you already have ssh and rsh autologin working. If you’re not sure

go back and test again. That will certainly hold you up here, and

give you all sorts of confusing error messages.

If you’re running on an uncertified version of Linux, you may want

to fire up the clusterware installer as follows:

$ ./runInstaller -ignoreSysPrereqs

If your Linux distro is still giving you trouble, you might try

downloading from centos.org where you can find complete ISOs for

RHEL, various versions. You can also safely ignore memory warnings

during startup. If you’re short on memory, it will certainly slow things

down, but we’re hitchhikers right?

You’ll be asked to specify the cluster configuration details. You’ll

want route66-vip to be commented out, so if you haven’t done that and

get an error to the affect of route66-vip already in use go ahead and

edit your /etc/hosts file.

I also got messages saying “route66-priv not reachable”. Check again

that sshd is running, and possibly disable your firewall rules:

$ /etc/rc.d/init.d/iptables stop

Also verify that eth0:1, and eth0:2 are created. Have you rebooted

since you created them? Be sure they’re still there with:

$ /sbin/ifconfig -a

Specify the network interface. This defaults to PRIVATE, just edit

and specify PUBLIC.

The next two steps ask for the OCR disk and voting disk. Be sure to

specify external redundancy. This is your way of telling Oracle that

you’ll take care of mirroring these important disks yourself, as loss

of either of them will get you in deep doodoo. Of course we’re

hitchhikers so we’re not trying to build a system that is never going

to breakdown, but rather we want to get the feeling of the wind blowing

in our hair. Click through to install and you should be in good shape.

At the completion, the installer will ask you to run the root.sh

script. I found this worked fine up until the vipca (virtual ip

configuration assistant). I then ran this one manually. You’ll need

to uncomment route66-vip from your /etc/hosts file as well. Once

all configuration assistants have completed successfully, return to

the installer and click continue, and it will do various other sanity

checks of your cluster configuration.

Since the clusterware install is rather testy, you’ll probably be doing

it a few times before you get it right. Here’s the cleanup if you

have to run through it again:

$ rm rf /etc/oracle

$ rm rf /home/oracle/oraInventory

$ rm rf $CRS_HOME

# these two commands cleanup the contents of these disks

$ /bin/dd if=/dev/zero of=/asmdisks/disk4 bs=1024k count=100

$ /bin/dd if=/dev/zero of=/asmdisks/disk5 bs=1024k count=20

$ rm /etc/rc.d/init.d/init.crs

$ rm /etc/rc.d/init.d/init.crsd

$ rm /etc/rc.d/init.d/init.cssd

$ rm /etc/rc.d/init.d/init.evmd

$ rm /etc/rc.d/rc3.d/S96init.crs

$ rm /etc/rc.d/rc5.d/S96init.crs

Now reboot the server. This will kill any clusterware processes still running.

If you’ve finished the Oracle Universal Installer at this point, and things

seem to be working, check at the command line with the ps command:

$ ps auxw | grep 10.2.0s

root 3728 0.0 0.3 2172 708 ? S May30 0:00 /bin/su -l oracle -c sh -c ‘ulimit -c unlimited; cd /home/oracle/product/10.2.0s/log/bebel/evmd; exec /home/oracle/product/10.2.0s/bin/evmd ‘

root 3736 0.0 5.0 509608 11240 ? S May30 4:51 /home/oracle/product/10.2.0s/bin/crsd.bin reboot

oracle 4047 0.0 2.8 192136 6284 ? S May30 0:00 /home/oracle/product/10.2.0s/bin/evmd.bin

root 4172 0.0 0.3 2164 708 ? S May30 0:00 /bin/su -l oracle -c /bin/sh -c ‘ulimit -c unlimited; cd /home/oracle/product/10.2.0s/log/bebel/cssd; /home/oracle/product/10.2.0s/bin/ocssd || exit $?’

oracle 4173 0.0 0.4 4180 900 ? S May30 0:00 /bin/sh -c ulimit -c unlimited; cd /home/oracle/product/10.2.0s/log/bebel/cssd; /home/oracle/product/10.2.0s/bin/ocssd || exit $?

oracle 4234 0.0 3.9 180108 8808 ? S May30 0:16 /home/oracle/product/10.2.0s/bin/ocssd.bin

oracle 4476 0.0 2.0 24048 4576 ? S May30 0:00 /home/oracle/product/10.2.0s/bin/evmlogger.bin -o /home/oracle/product/10.2.0s/evm/log/evmlogger.info -l /home/oracle/product/10.2.0s/evm/log/evmlogger.log

oracle 6989 0.0 0.1 2676 404 ? S May30 0:00 /home/oracle/product/10.2.0s/opmn/bin/ons -d

oracle 6990 0.0 1.9 93224 4280 ? S May30 0:00 /home/oracle/product/10.2.0s/opmn/bin/ons -d

oracle 7891 0.0 0.2 3676 660 pts/3 S 23:35 0:00 grep 10.2.0s

Also list the nodes you have available to your clusterware software:

$ olsnodes n

route66 1

The Oracle 10g install itself is very trivial, assuming you’ve installed

Oracle before. Use the -ignoreSysPrereqs flag if necessary to start

up the Oracle Universal Installer, and use the software-only option,

as we’ll be creating our RAC database by hand. Also select Enterprise

Edition and things should proceed smoothly. Oracle will recognize that

you have the clusterware installed, and let you know during the

installation.

5. setup the listener and an ASM instance

——————————————-

The listener.ora file is setup as usual, the only difference is you will

include both route66 and route66-vip.

SID_LIST_LISTENER =

(SID_LIST =

(SID_DESC =

(SID_NAME = PLSExtProc)

(ORACLE_HOME = /home/oracle/product/10.2.0)

(PROGRAM = extproc)

)

)

LISTENER =

(DESCRIPTION_LIST =

(DESCRIPTION =

(ADDRESS_LIST =

(ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC))

)

(ADDRESS_LIST =

(ADDRESS = (PROTOCOL=TCP)(HOST=route66)(PORT = 1521)(IP = FIRST))

)

(ADDRESS_LIST =

(ADDRESS = (PROTOCOL=TCP)(HOST=route66-vip)(PORT = 1521)(IP = FIRST))

)

)

)

You may also choose to use the actual IP addresses of these hostnames if

you like. One other difference is that you will use Oracle 10g’s new

srvctl utility to start th listener.

$ srvctl start nodeapps -n route66

Ok, on to the fun stuff. It’s time to configure our ASM instance.

Our instance name will be +ASM1, and we’ll set the ORACLE_SID as usual.

In the init+ASM1.ora file specify:

user_dump_dest=/home/oracle/admin/+ASM1/udump

background_dump_dest=/home/oracle/admin/+ASM1/bdump

core_dump_dest=/home/oracle/admin/+ASM1/cdump

large_pool_size=15m

instance_type=asm

asm_diskstring=’/dev/raw/raw1′, ‘/dev/raw/raw2′,’/dev/raw/raw3′

Then in sqlplus startup the ASM instance:

SQL> startup nomount

Next tell ASM how we want to utilize the space we have by creating disk

groups:

SQL> create diskgroup DBDATA external redundancy disk

‘/dev/raw/raw1′,’/dev/raw/raw2;

SQL> create diskgroup DBRECO external redundancy disk ‘/dev/raw/raw3′;

Now that you have diskgroups, you want to make a note of it in your

init.ora:

SQL> !echo “asm_diskgroups=’DBDATA’,'DBRECO’” >> init+ASM1.ora

Now startup your ASM instance:

SQL> startup force

Exit sqlplus and let srvctl know about the new ASM instance:

$ srvctl add asm -n route66 -i +ASM1 -o /home/oracle/product/10.2.0.1

Now you can shutdown in sqlplus, and startup with srvctl:

SQL> shutdown immediate

$ srvctl start asm -n route66

And lastly use the ps command to check for your new instance.

6. create an instance, start it, and register with srvctl

———————————————————

We’re getting to our clustered database slowly but surely. We’re

just getting over the mountains now, and the open road is ahead of

us.

We’re going to create the first of our two instances and call it

BEATNIK. Edit your initBEATNIK.ora as follows:

db_block_size=8192

db_multiblock_read_count=8

db_name=kerouac

BEATNIK.background_dump_dest=/home/oracle/admin/BEATNIK/bdump

BEATNIK.user_dump_dest=/home/oracle/admin/BEATNIK/udump

BEATNIK.core_dump_dest=/home/oracle/admin/BEATNIK/cdump

BEATNIK.instance_number=1

BEATNIK.instance_name=BEATNIK

BEATNIK.thread=1

BEATNIK.undo_tablespace=beatnikundo

HIPPY.background_dump_dest=/home/oracle/admin/BEATNIK/bdump

HIPPY.user_dump_dest=/home/oracle/admin/BEATNIK/udump

HIPPY.core_dump_dest=/home/oracle/admin/BEATNIK/cdump

HIPPY.instance_number=2

HIPPY.instance_name=HIPPY

HIPPY.thread=2

HIPPY.undo_tablespace=hippyundo

You’ll also have to create the /home/oracle/admin/BEATNIK/* and

/home/oracle/admin/HIPPY/* directories.

Now edit the file crKEROUAC.sql as follows:

CREATE DATABASE KEROUAC”

DATAFILE SIZE 250M EXTENT MANAGEMENT LOCAL

SYSAUX DATAFILE SIZE 125M

DEFAULT TEMPORARY TABLESPACE TEMP TEMPFILE SIZE 20M

UNDO TABLESPACE “beatnikundo” DATAFILE SIZE 200M

CHARACTER SET WE8ISO8859P1

LOGFILE GROUP 1 SIZE 10240K,

GROUP 2 size 10240k,

GROUP 3 size 10240k;

There are other parameters you can specify in this create statement, such as

maxinstances, maxlogmembers, and the sys password. However I’ve tried to

simplify it, to make it easier to review and understand. Check the Oracle

docs for details.

Now startup sqlplus and issue:

SQL> startup nomount pfile=/home/oracle/admin/BEATNIK/pfile/initBEATNIK.ora

SQL> @crKEROUAC.sql

Now get the names of your controlfiles from v$parameter and add them to

the initBEATNIK.ora file.

Now add a couple more parameters to your initBEATNIK.ora file:

*.cluster_database=true

*.cluster_database_instances=5

Use sqlplus to stop and start the db again:

SQL> shutdown immediate

SQL> startup force

Now register our new database:

$ srvctl add database d KEROUAC -o /home/oracle/product/10.2.0.1/

$ srvctl add instance -d KEROUAC -i BEATNIK -n route66

Now one more time, shutdown with sqlplus, and then use srvctl to

start the db. From now on srvctl can stop + start the db.

$ srvctl start instance -d KEROUAC -i BEATNIK

7. create a second instance & undo tablespace, & register it

———————————————————–

Since the database is already created, you don’t have to do that step

again. At this point all you have to do is create another instance.

First fire up sqlplus and create another undo tablespace:

SQL> create undo tablespace hippyundo datafile ‘+DBDATA’ size 100m;

Make a copy of your init.ora for the hippy instance like this:

$ cp initBEATNIK.ora initHIPPY.ora

Then set your ORACLE_SID and use sqlplus to startup:

SQL> startup

Finally register with srvctl:

$ srvctl add instance -i HIPPY -d KEROUAC -n route66

8. create data dictionary

————————–

SQL>@?/rdbms/admin/catalog.sql

SQL>@?/rdbms/admin/catclust.sql

10. Some Things to Understand

————————–

Automatic Storage Management

Global Cache Services

Global Enqueue Services

Clusterware procs crsd, evmd, ocssd,oprocd

RAC processes lms, lmd, lmon, lck0

GV$ data dictionary

11. Further Reading

—————–

Clusterware & RAC Install & Configuration Guide for Linux

Clusterware & RAC Administration Deployment Guide

Oracle Technology Network – http://otn.oracle.com

Please visit http://www.iheavy.com or email me at

Thanks to Amit Poddar & dizwell.com

Apress, Oracle Press books

Part 3: Rac/Linux/Firewire – Software Requirements, Versions, etc.

Software Requirements, versions, etc


Advance notice for beginners out there. I’ve installed Linux probably 20-30 times over the last 10 years, and it’s gotten a lot easier, however, you’ll need to have some decent skill at kernel installations. I’ll outline what I did that finally worked, but I’ll also say that I tried getting this setup to work on a couple of other boxes before I ended up using the emachines ones. I think the kernel which the Linux Firewire team has compiled and patched to their liking is pretty finicky, hardware-wise.

RedHat 8.0

I can’t go into the details of installing Linux in this short article, and there are plenty of resources on the web to help you. You’ll need to get that up and running first, with the 1394 card in the box.

Oracle 9.2.0.1

Oracle is a beast to install, and challenging for the beginner. Luckily there are a number of resources on the web.

patch 9.2.0.2

This patch is a *REQUIRED* patch. The guys on the Linux Firewire project assure me that the Oracle Cluster Manager has a lot of bugs and won’t work straight out of the box from the 9.2.0.1 release. Here’s a little twist. Although this is an *OPEN SOURCE* project, and the source is all free, and you can download Oracle and play with it to your heart’s content, to download a patch, you need access to Oracle’s Metalink, which requires you to have a software license. Not much I can say here folks. Use your imagination on getting ahold of this patch. Maybe if you place nice, someone on an email list will post it temporarily for you or something. It’s 235M. :-)

Login to Metalink (some like to call it metastink)

Click on the “patches” link along the left.

Search for patch 2632931

Also, take a look at this note:

Doc ID: Note:217811.1

Subject: ALERT: 9.2.0.2 Patchset for Linux Does Not

Apply Correctly in the RAC Environment

Type: ALERT

Status: PUBLISHED

Content Type: TEXT/PLAIN

Creation Date: 07-NOV-2002

Last Revision Date: 08-JAN-2003

Part 1 – Introduction

Part 2 – Basic Costs + Hardware Platform Outline

Part 3 – Software Requirements, Versions, etc

Part 4 – Initial Oracle Setup

Part 5 – Firewire + OCFS Setup

Part 6 – Cluster Manager Setup

Part 7 – Cluster Database Setup

Part 8 – Review of Clustered Features + Architecture

Part 9 – A quick 9iRAC example

Part 10 – Summary

Your Database – A Long-Haul Truck Or A Sports Car?

Introduction

Databases, even those running high-powered software like Oracle, can be incredibly touchy, demanding, and even fragile in their own way. For that reason we must take care to optimize and tune them based on characteristic usage.

Data Warehouse vs OLTP

In broad terms, database applications are divided into two large classes, Data Warehouse, and OLTP (online transaction processing – what a mouthful!). For the purposes of this dicussion, let’s call them a heavy lifting truck, and sports car. Now both have powerful engines, but they’re used for very different purposes.

Our Data Warehouse is characterized by large transactions, huge joins, all of which work to produce very large usually one-off reports. The reports may be run a handful of times. These databases do mostly read-only activity, occaisionally performing large dataloads to add to the archive of data.

On the other hand OLTP databases are characterized by thousands or tens of thousands of very small transactions. Web sites for instance, exhibit this characteristic. Each transaction is doing something quite small, but in aggregate, thousands of users put quite a heavy, and repeated load on the database, and they all expect instantaneous response!

We provide these two very different types of databases by laying out the database for its characteristic usage, and then tuning relevant parameters appropriately. We may allocate more memory to sorting, and less to the db cache for a Data Warehouse, whereas a large db cache might help us a lot with an OLTP application. We may enable parallel query, or partition large tables in our Data Warehouse application.

Choose One Or The Other

If you have a database serving a web-based application, and you are trying to do large ad-hoc reports against it, you will run into trouble. All that memory you’ve setup to cache small web transactions, will get wiped out with the first large report you run. What’s more the heavy disk I/O you perform reading huge tables, and then sorting and aggregating large datasets will put a huge load on your database which you setup specifically for your web application.

Conclusion

There are lots and lots of parameters and features in Oracle best suited to one or the other type of application, to the point where your database really will look like a sports car or a long-haul truck when you’re done tuning it. For that reason it is really essential that these types of applications be divided up into separate instances of Oracle preferrably on separate servers.

Part 4: RAC/Linux/Firewire – Initial Oracle Setup

Initial Oracle Setup


Follow these instructions to get Oracle up and running on your new Linux box.

1. Setup the oracle account, and environment as follows.

Create an oracle user on your linux box:

$ adduser oracle

2. Login to the oracle account and edit .oraenv9i as follows (assumes bash):

# oracle environment variables

export ORACLE_BASE=”/home/oracle”

export ORACLE_HOME=”/home/oracle/product/9.2.0″

# EAST on machines #2

export ORACLE_SID=”WEST”

export PATH=$PATH:$ORACLE_HOME/bin

export LD_LIBRARY_PATH=”$ORACLE_HOME/lib”

export LD_ASSUME_KERNEL=2.2.5

# US7ASCII is the default, but WE8ISO8859P1 support more languages

export NLS_LANG=AMERICAN_AMERICA.WE8ISO8859P1

export ORA_NLS33=$ORACLE_HOME/ocommon/nls/admin/data

export ORACLE_TERM=xterm

export ORACLE_OWNER=oracle

export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib:/usr/local/lib

export TNS_ADMIN=$ORACLE_HOME/network/admin

# run the RH compatability stuff

. /usr/i386-glibc21-linux/bin/i386-glibc21-linux-env.sh

# setup Java

export JAVA_HOME=/usr/local/java

export CLASSPATH=$ORACLE_HOME/jdbc/lib/classes12.zip:$ORACLE_HOME/JRE:$ORACLE_HOM

E/jlib:$ORACLE_HOME/rdbms/jlib:$ORACLE_HOME/network/jlib:.

Add this line to your .bash_profile:

. /home/oracle/.oraenv9i

Next install the glibc backward compatability libs per

compat-

egcs-6.2-1.1.2.14.i386.rpm

compat-

glibc-6.2-2.1.3.2.i386.rpm

compat-libs-6.

2-3.i386.rpm

3. Next install JDK.

4. Edit the hosts file if necessary to include two machines on your local network

by name. I used “utopia” and “zenith”.

5. Next run the Oracle installer as follows (assuming you opened the packages in

/tmp). If you’re not on the X-window console, connect with SSH and you should ha

ve x tunneling by default, and can display remotely:

$ ./tmp/Disk1/install/linux/runInstaller

First install the cluster manager. You’ll be prompted for the local and remote h

ostname, a quorum disk, as well as some other things. Doesn’t matter what you en

ter right now, as we’re going to go back and edit those files and do things by ha

nd anyway.

6. Next go through the 9.2.0.1 software install. Here are some other notes with

various weblinks. Be sure to select “Enterprise Edition” and also “Software Only

Install”.

I got started with this Oracle 8 install doc which details which compatability li

braries you’ll need, how to setup the Oracle account, environment variables, and

Java.

link

>

I ran into some trouble with LD_ASSUME_KERNEL, and compat libs:

You’ll encounter problems with the context makefile, and get this error “Error in

invoking target install of makefile /opt/oracle/product/9.2.0/ctx/lib/ ins_ctx.m

k”. Sad but true, this is *NORMAL* behavior. I suspect making the installer run

flawlessly isn’t at the top of Oracle’s priority list.

link

I also encountered problems with Oracle Net Configuration Assistant “Can’t find $

ORACLE_HOME/jre/1.1.8/bin/../bin/i586/green_threads/jre” and I added this symlink

:

$ ln -s $ORACLE_HOME/jre/1.1.8/bin/i686
$ORACLE_HOME/jre/1.1.8/bin/i586

I got a simlar error for libjava.so:

$ ln -s $ORACLE_HOME/jre/1.1.8/lib/i686
$ORACLE_HOME/jre/1.1.8/lib/i586

Though you’ll probably want to create your own database from scratch later, it’s

sometimes instructive to let the Oracle Database Configuration Assistant create a

starter one for you, and look at what options they use. As with the rest, you’l

l run into an error. This time it is “ORA-27123 unable to attach to shared memor

y segment: Oracle Database Configuration Assistant”. Fix it by doing the followi

ng as root on your linux box:

$ cat /proc/sys/kernel/shmmax

33554432

$ echo `expr 1024 * 1024 * 1024` > /proc/sys/kernel/shmmax

$ cat /proc/sys/kernel/shmmax

1073741824

Obviously you’ll have to go through this process (living hell?) on both boxes you

‘ll be using in your cluster.

Next rerun the installer and specify the location of the itty bitty 235M patch, a

nd install that.

7. Enable RAC in Oracle9i

The Real Application Cluster feature is *NOT* enabled by default. Here’s how you

enable it:

As the oracle user:

$ cd $ORACLE_HOME/rdbms/lib

$ make -f ins_rdbms.mk rac_on

$ make -f ins_rdbms.mk ioracle

As root set permissions on rac_on

$ chown oracle /etc/rac_on

$ chgrp dba /etc/rac_on

Part 1 – Introduction

Part 2 – Basic Costs + Hardware Platform Outline

Part 3 – Software Requirements, Versions, etc

Part 4 – Initial Oracle Setup

Part 5 – Firewire + OCFS Setup

Part 6 – Cluster Manager Setup

Part 7 – Cluster Database Setup

Part 8 – Review of Clustered Features + Architecture

Part 9 – A quick 9iRAC example

Part 10 – Summary

Part 5: RAC/Linux/Firewire – Firewire + OCFS Setup

Firewire + OCFS Setup


In this installment, we’ll discuss how to get the Firewire drive shared between your two Linux boxes.

8. Test Firewire drive

At this point you can test the firewire drive if you like, with the standard Linux driver. You won’t be able to share the drive between the two nodes yet, however.

As root do the following:

$ modprobe ohci1394

$ modprobe ieee1394

$ modprobe sbp2

$ modprobe scsi_mod

Grab a copy of rescan-scsi-bus.sh from here and run it:
http://www.fifi.org/cgi-bin/man2html/usr/share/man/man8/rescan-scsi-bus.sh.8.gz

run rescan scsi bus

Now partition it with fdisk:

$ fdisk /dev/sda

Now try making an ext2 filesystem with mke2fs

$ mke2fs /dev/sda1

Now mount it

$ mount -t ext2 /dev/sda1 /mnt/test

Now unmount it

$ umount /mnt/test

9. Linux Kernel Setup w/Firewire patch

The Linux kernel is a complex beast, and compiling it can often be a challenge. Though I like rolling my own, I downloaded the patched firewire source distro off of OTN, and try as I might, I could not get those compiled kernels to work. If anyone *DOES* get it to work, please send me their “.config” from the kernel source directory. Also I’ve tried to encourage the Oracle/Linux Firewire team to build a patch-only distro which can be applied against a standard Linux source tree. No luck yet.

Assuming you’re not going to roll your own, just download linux-2.4.20rc2-orafw-up.tar.gz from here:

http://otn.oracle.com/tech/linux/open_source.html

Move to the “/” or root directory, and untar the file:

$ tar xvzf linux-2.4.20rc2-orafw-up.tar.gz

Edit your /etc/lilo.conf or /etc/grub.conf file to include the new kernel. Do *NOT* make it the default kernel, it may not boot.

Reboot. If you come up again, you’re in luck, the kernel works for your machine. Next you want to edit your /etc/modules.conf to include these lines:

# options for oracle firewire patched kernel

options sbp2 sbp2_exclusive_login=0

post-install sbp2 insmod sd_mod

post-remove sbp2 rmmod sd_mod

As root, load the modules like this:

$ modprobe ieee1394

$ modprobe ohci1394

$ modprobe ide-scsi

$ modprobe sbp2

$ modprobe scsi_mod

If you’re having trouble seeing the device, grab a copy of rescan-scsi-bus.sh from here:
http://www.fifi.org/cgi-bin/man2html/usr/share/man/man8/rescan-scsi-bus.sh.8.gz

If you want to partition, now is a good time. Use fdisk as root like this:

$ fdisk /dev/sda

If you have other SCSI devices, it may be /dev/sdb or dev/sdc and so on.

10. Go through steps 1-8 on node 2

11. Cluster Filesystem setup (OCFS)

If you wanna play around, use mke2fs on the one of the partitions you created with fdisk, and then mount the partiton on machine a. Then mount the partition again on machine b. Create a file on one of the two boxes. The other machine *WON’T* reflect it. This is equivalent to unplugging a disk which is mounted, such as a USB device, or some such. You can and probably *HAVE* corrupted the filesystem. That’s ok, because we don’t have anything important on the disk yet. Ok, unmount on both machines. If you have trouble, you may need to reboot.

Having gone through the above example, you know why OCFS is so important. Ok, now the fun part. Install OCFS. There are good docs to be found in the linux_ocfs.pdf file here:

http://download.oracle.com/otn/linux/code/ocfs/linux_ocfs.pdf

Without RedHat Advanced Server, the RPMs are *NOT* going to work. Just grab a copy of ocfs-1.0-up.o and put it in
/lib/modules/2.4.20-rc2-orafw/kernel/fs.

Use ocfstool to create the /etc/ocfs.conf file. The pdf doc listed above is pretty good at explaining this.

Load the ocfs kernel module with load_ocfs. If everything goes right it will tell you like this:

$ cd /lib/modules/2.4.20-rc2-orafw/kernel/fs

$ load_ocfs

/sbin/insmod ocfs node_name=zenith
ip_address=192.168.0.9 ip_port=7000 cs=1865 guid=72C2AF5CA29FA17CB9CB000AE6312F24

Using /lib/modules/2.4.20-rc2-orafw/kernel/fs/ocfs.o

Next make the filesystem. ocfstool can do this too.

$ mkfs.ocfs -F -b 128 -L /ocfs -m /ocfs -u 1001 -g 1001 -p 0775 /dev/sda1

And finally mount the filesystem!

$ mount -t ocfs /dev/sda1 /ocfs

$ df -k

Filesystem 1K-blocks Used Available Use% Mounted on

/dev/hda2 55439548 20835260 31788096 40% /

/dev/hda1 101089 18534 77336 20% /boot

none 112384 0 112384 0% /dev/shm

/dev/cdrom 122670 122670 0 100% /mnt/cdrom

/dev/sda1 60049024 30080 60018944 1% /ocfs

12. Perform step 10 on node 2.

13. Test ocfs

Here we quickly verify that a file created on one instance is viewable on another.

On node1 do:

$ cd /ocfs

$ touch mytestfile

On node2 do:

$ cd /ocfs

$ ls

mytestfile

$

You’ll see to your astonishment that the file is now visible on node 2!

Part 1 – Introduction

Part 2 – Basic Costs + Hardware Platform Outline

Part 3 – Software Requirements, Versions, etc

Part 4 – Initial Oracle Setup

Part 5 – Firewire + OCFS Setup

Part 6 – Cluster Manager Setup

Part 7 – Cluster Database Setup

Part 8 – Review of Clustered Features + Architecture

Part 9 – A quick 9iRAC example

Part 10 – Summary