Open Insights 30 – Crowdsourcing

OPEN INSIGHTS Newsletter
Issue 30 – Crowdsourcing
April 1, 2007

by Sean Hull

Founder and Senior Consultant
Heavyweight Internet Group

Welcome back to our Open Insights newsletter. Our readership is now north of 3000 subscribers and growing everyday. Thanks to everyone for your support and for forwarding us on to friends and colleagues!

Reading from your blackberry or other handheld device? We’ve made some formatting changes which we hope improve the appearance on mobile devices. Let us know if you have any suggestions or comments.


In This Issue:

1. Feature:Crowdsourcing

2. Upcoming Speaking Engagements

3. New Articles
4. Audio Interviews
5. Current Reading
6. Lightweight Humor

7. Miscellaneous

8. Past Issues
9. Technical Articles
10. About Heavyweight Internet Group


1. Feature: Crowdsourcing

Crowdsourcing, there’s that word again, source. Open-source, out-source, source-code. The idea behind crowdsourcing is for a company to capitalize on a vast network of internet workers where ordinarily they might rely on a small group of dedicated individuals. The internet makes this possible, as it is decentralized, and pervasive. Crowdsourcing is different than open-sourced projects, because everyone doesn’t benefit from the results of that work.

Wikipedia – Crowdsourcing

Crowdsourcing can allow innovation, and encourage solutions to problems previously difficult to solve. Take Amazon’s Mechanical Turk, where internet legions are encouraged to perform tasks much better solved by people than computers. In return, “turk workers” are reimbursed small amounts depending on the task. Bezos, Amazon’s founder, is also investing in ChaCha which is dubbed “human assisted” search.

So blogs, and sites like wikipedia then, would not really be crowdsourcing per se, because everyone gets to enjoy and benefit from the end product. A recent NYTimes article mentions two main objections. First, like wikipedia, harnessing the knowledge of crowds means you’re only as good as the smartest member. The other objection leveled against crowdsourcing likens the idea to a virtual internet based sweatshop. Though certainly companies like Amazon hold the strings with services like mturk, it seems to share little of the more nefarious qualities of a real sweatshop. Then again, how long until turk workers start putting together a unions?


Another very flatening (to use Tom Friedman’s term) aspect of crowdsourcing is in it’s ability to turn a once scarce resource into a commodity. Some would argue this is happening with stock photography, and even progamming with sites like Rent-a-coder.


For more info, see this Wired News article.

2. Upcoming Speaking Engagements


April 15th, Collaborate 2007 – Las Vegas Nevada


in bed with Oracle – Lifting The Covers On Database Creation

Database creation, because of better GUI tools, has become a more & more overlooked area of Oracle. We pull back the covers, revealing what Oracle is doing at each stage. Why do we have startup nomount, mount, restrict, and open? What OS resources is Oracle using at each step? How do we issue CREATE DATABASE? What is the simplest init.ora file? How many file descriptors does Oracle use and why? From conception to birth, our microscope will reveal the secrets.

3. New Articles

Oracle 10g RAC versus DataGuard for High Availability

4. Audio Interviews

This month we have the opportunity to talk with Norman Yamada CTO of Millburn Corporation.

Norman shares with us his experiences providing world-class computing solutions, and the pros and cons of doing it with open source.

5. Current Reading

Tangram Puzzles – Chris Crawford

The mind is a muscle, and like any other, it needs to be exercised. If thinking outside the box has got you hitting a wall, the Tangram puzzle, also known as the seven-board, will surely help you. These puzzles are a constant challenge, pushing you to think in spatial, unconventional and creative ways.


Number Freaking by Gary Rimmer

Asking funny and surprising questions about numbers, this book puts a new perspective on statistics, and numerical thinking.


Firing Back by Jeffrey Sonnenfeld and Andrew Ward

Those who succeed have fall down often enough. But what separates them is that time after time, they get back up to fight again. This book is full of real-life stories of leaders of industry who crashed and burned, and then turned around and rebuilt their careers.


6. Lightweight Humor

Visit Gaping Void’s funny entry:

Fake Walmart Blog.

7. Miscellaneous


Improve Body Language:


Presentation Zen – Lessig Method


Dick Hardt’s famous Identity 2.0 presentation

8. Past Issues

Issue 28: High Availability

Issue 27: Fragile Foundations

Issue 26: Logistical Fitness

Issue 25: Which Red Button
Issue 24: Consulting Conflicts of Interest
Issue 23: Devil In The Details
Issue 22: Beware of Software Fashion
Issue 21: Open Season, Open Sesame?
Issue 20: Better Web Better Business
Archive: Past Issues

9. Technical Articles

Oracle DBA Interview: click here
Tools for the Intrepid DBA: click here
Oracle9i + RAC on Linux/Firewire: click here
Migrating MySQL to Oracle: click here
MySQL Disaster Recovery: click here

10. About Heavyweight Internet Group

In a nutshell, Oracle. Everything related to and surrounding the database technology we specialize in, but specifically setup, admin and tuning of Oracle technology. I have 10 years experience with Oracle, wrote a book on the technology, and write and lecture frequently. I’m founder and senior consultant of the company. In capacities where your company might hire Deloitte, AIG, or Oracle Consulting we can bring the same level of service and experience, at about half the price. Simple equation.

Looking for top-flight a DBA? Visit us on the web at iheavy.com.

Open Insights 29 – Mainroads or Sidestreets

OPEN INSIGHTS Newsletter
Issue 29 – Mainroads or Sidestreets
March 2, 2007

by Sean Hull

Founder and Senior Consultant
Heavyweight Internet Group

Welcome back to our Open Insights newsletter. Our readership is now north of 3000 subscribers and growing everyday. Thanks to everyone for your support and for forwarding us on to friends and colleagues!

Reading from your blackberry or other handheld device? We’ve made some formatting changes which we hope improve the appearance on mobile devices. Let us know if you have any suggestions or comments.


In This Issue:

1. Feature:Mainroads or Sidestreets

2. Upcoming Speaking Engagements

3. New Articles
4. Audio Interviews
5. Current Reading
6. Lightweight Humor

7. Miscellaneous

8. Past Issues
9. Technical Articles
10. About Heavyweight Internet Group


1. Feature: Mainroads or Sidestreets

Choosing Mainroads or Sidestreets?

Recently I was doing a bit of traveling for vacation. I had heard great things about Costa Rica, with it’s untouched wilderness, and jungle in the middle of tropical Central America. After doing the requisite research, arranging travel & accomodations, I hopped on a plane bound for San Jose. From there a bus took me on the precipitous journey through the jungles, out of the mountains and down to the Caribean coast to a little town known as Puerto Viejo de Limon.
The town is quite small consisting of four or five mainroads which are paved, and then many sidestreets off of those, dirt roads with little shops, soda restaurants, cafes and adventure tour guides. While walking down some of the sidestreets I happened into one of the adventure shops, and struck up a conversation with one of the guides. He started telling me a story about the time he was traipsing through the jungle and tripped, twisting his ankle, and finding himself face to face with a rather large and menacing snake. Then without skipping a beat, he went on to explain that although the tours go through the jungle, they are all quite safe & so on.
I thought about this whole story a moment. My minds gears began to spin. A guide with such real-world experience affords extra confidence that if something does happen, he will know how to handle the situation. There are parallels here with consulting, and experience. I’ve talked in the past about certifications, being quite a bit overrated. A strong foundation is crucial, yes. But truely nothing can substitute for experience gained in the trenches. In searching for consulting resources – in any field really, not just technology related – you want someone who has taken the systems apart and put the proverbial motorcycle back together piece by piece. You want someone who has traveled the sidestreets, and backstreets, knows what to look out for, but also who will steer for the mainstreets to avoid seeking out trouble.
In computing you want someone who enjoys taking the systems apart component by component, hardware and software, and then putting them back together. One who has deliberately broken, hacked, or disassembled, then troubleshooted and put humpty dumpty back together. This is not experience normally gained in the classroom but rather by getting ones hands dirty, figuring out how all those pieces actually work and fit together – not just how they are supposed to operate in principal or on paper.
Keep this in mind the next time you interview a candidate. Ask their opinion & preferences and ask for a war story or two. Don’t overemphasize degrees and certifications or let them blind you to a candidate’s lack of street smarts. Sooner or later your systems will step out of line, and having that real-world experience will come in very handy.
Also checkout: Oracle DBA Interview Questions and our past newsletter, Issue 09 – IT Certifications.

2. Upcoming Speaking Engagements
March 15th, New York Oracle User Group
in bed with Oracle – Lifting The Covers On Database Creation

Database creation, because of better GUI tools, has become a more & more overlooked area of Oracle. We pull back the covers, revealing what Oracle is doing at each stage. Why do we have startup nomount, mount, restrict, and open? What OS resources is Oracle using at each step? How do we issue CREATE DATABASE? What is the simplest init.ora file? How many file descriptors does Oracle use and why? From conception to birth, our microscope will reveal the secrets.
April 15th, Collaborate 2007 – Las Vegas Nevada
in bed with Oracle – Lifting The Covers On Database Creation

Database creation, because of better GUI tools, has become a more & more overlooked area of Oracle. We pull back the covers, revealing what Oracle is doing at each stage. Why do we have startup nomount, mount, restrict, and open? What OS resources is Oracle using at each step? How do we issue CREATE DATABASE? What is the simplest init.ora file? How many file descriptors does Oracle use and why? From conception to birth, our microscope will reveal the secrets.

3. New Articles

Oracle 10g RAC versus DataGuard for High Availability

4. Audio Interviews

This month we have the opportunity to talk with William Hurley aka Whurley, the Chairman of the Open Management Consortium.

In our interview, we discuss open-source, and it’s impact on commercial software and solutions, and wrestle head on with some of the concerns people have on both sides of the fence.

William Hurley is the CTO at Qlusters, where he launched the openQRM project. He has been awarded IBM’s Master Inventor title, multiple awards for innovation at Apple Computer. Prior to joining Qlusters he was CTO and founder at Symbiot. He holds 11 patents for research and development at IBM, Tivoli Systems, and Apple Computer. He was recently elected Chairman of the Open Management Consortium.

5. Current Reading

Set Your Voice Free: How To Get The Singing Or Speaking Voice You Want by Roger Love

Want to improve your voice for a better and stronger presence in public speaking + lecturing? Or perhaps you just want to build up a better and more consistent voice for telephoning clients, and building rapport. Whatever the reason, this book provides exercises and techniques for beginners and advanced alike.
The 33 Strategies of War by Robert Greene

Robert Greene’s books all take an insightful if slightly dark view of human nature to illuminate behavior. His Strategies of War, as his Laws of Power before, put you on a stronger footing in a world that is not always friendly. Ignore his wisdom at your own peril!
The Oracle Hacker’s Handbook: Hacking and Defending Oracle by David Litchfield

The world of hacking, and computer security has finally and abruptly caught up with databases, and Litchfield’s latest book brings into stark contrast the vulnerabilities, back doors, and general risk that your database and consequently your data, may be in. Sobering reading.


6. Lightweight Humor

Visit Gaping Void’s funny entry:

Fake Walmart Blog.

7. Miscellaneous

Improve Body Language:
Presentation Zen – Lessig Method
Dick Hardt’s famous Identity 2.0 presentation

8. Past Issues

Issue 28: High Availability

Issue 27: Fragile Foundations

Issue 26: Logistical Fitness

Issue 25: Which Red Button
Issue 24: Consulting Conflicts of Interest
Issue 23: Devil In The Details
Issue 22: Beware of Software Fashion
Issue 21: Open Season, Open Sesame?
Issue 20: Better Web Better Business
Archive: Past Issues

9. Technical Articles

Oracle DBA Interview: click here
Tools for the Intrepid DBA: click here
Oracle9i + RAC on Linux/Firewire: click here
Migrating MySQL to Oracle: click here
MySQL Disaster Recovery: click here

10. About Heavyweight Internet Group

In a nutshell, Oracle. Everything related to and surrounding the database technology we specialize in, but specifically setup, admin and tuning of Oracle technology. I have 10 years experience with Oracle, wrote a book on the technology, and write and lecture frequently. I’m founder and senior consultant of the company. In capacities where your company might hire Deloitte, AIG, or Oracle Consulting we can bring the same level of service and experience, at about half the price. Simple equation.

Looking for top-flight a DBA? Visit us on the web at iheavy.com.

Open Insights 28 – High Availability

OPEN INSIGHTS Newsletter
Issue 28 – High Availability
February 1, 2007

by Sean Hull

Founder and Senior Consultant
Heavyweight Internet Group

Welcome back to our Open Insights newsletter. Our readership is now north of 3000 subscribers and growing everyday. Thanks to everyone for your support and for forwarding us on to friends and colleagues!

Reading from your blackberry or other handheld device? We’ve made some formatting changes which we hope improve the appearance on mobile devices. Let us know if you have any suggestions or comments.


In This Issue:

1. Feature: High Availability

2. Upcoming Speaking Engagements

3. New Articles
4. Audio Interviews
5. Current Reading
6. Lightweight Humor

7. Miscellaneous

8. Past Issues
9. Technical Articles
10. About Heavyweight Internet Group


1. Feature: High Availability

What is High Availability?


In enterprise applications, that is internet websites, or payroll systems, or other large computing systems we talk about high availability when we need those systems to be available all the time. So High Availability is the catch phrase to discuss that, what does it mean, what can I expect, what are some solutions, and so forth.


One phrase you might hear a lot is Five-9’s. This means systems that are available 99.999% of the time. In a year there are 24×365 hours, so that is about 8 3/4 hours downtime per year. Why not 100%, you might ask? Well we will discuss that too.


Why does it matter?


If your money isn’t available when you go to an ATM, it might be obvious there’s a problem. What about if your favorite search engine, google, isn’t available when you try to reach it? How about online banking? For these day-to-day needs, you can usually tolerate some downtime.


In the corporate environment, however, there is much more at stake. Suppose a large bank does not have access to it’s database systems, and can’t perform transactions. Maybe their customers will go elsewhere. Perhaps some trading systems are down for a few minutes, and can’t conduct transactions, they can lose big time. In each case we have tradeoffs, between the costs of more perfect systems, and the costs of not being able to do business.


How do various risks play a part?


There are all sorts of events that can get in the way of a transaction. Segments of the internet can go down, name services (DNS) can be interrupted, the data center could have a fire, or an earthquake could bring things to a halt. We can even experience a power grid failure. But beyond all these external forces, we can have hardware failure, data corruption, software bugs, and even operator error. Considering all of these factors, the spectrum of risks becomes clearer, as we understand how many interconnected parts come into play.


Take for example the power grid. The entire Northeast lost power for 24 hours. Now that happened once in

roughly 24 years. That means on average one hour per year. That’s not even including other various smaller outages that we’re likely to experience.


In computing systems themselves, we can build redundant databases, and all sorts of parallel or grid solutions, but in so doing we introduce more moving parts, more points of failure, and more software bugs. We even introduce more chance for operator error, and potentially more complex upgrades, leading to more downtime.


Reasonable Goals and Practical Expectations


For those not accustomed to thinking about risk in these terms, a sense of perspective is very very important. Our gut instinct is to go for perfect systems, with 100% uptime, and start selecting solutions based on that. But we may miss a lot of important and relevant facts when we do that. If the bar is already lowered by factors completely outside of our control, we get a better perspective on what we can reasonably expect to have in terms of uptime, and plan accordingly. Much better to fall well within our expected goals, than have the real world come crashing into our forecasts.


Bruce Schneier publishes an excellent newsletter on risk, and security called Cryptogram. Here’s the latest issue.


What are some commercial and open-source solutions that provide it?


Since we work in the Oracle world, as well as the open-source database world, we’ll describe a few solutions available for Oracle and for MySQL.


Oracle Clustering (SE & EE)


Also know as RAC or Real Application Clusters, this is evolved from the former parallel server product. RAC is implemented with shared disk between two or more locally connected computers. That is they’re in the same room, rack, or hosting facility. Based on the discussion above, there are some real-world disasters that you would be vulnerable to here, but what does it help you with? Well it helps you scale your application on commodity hardware, albeit with a few more moving parts, and somewhat more complicated administration. But with rolling upgrades, and smart load balancing, as well as distributed caching scheme called cache fusion, Oracle has taken this technology to the next level, and made it work.


Oracle DataGuard (Enterprise Edition)


For many enterprises, Data Guard is the high availability solution you want and need. It requires Enterprise Edition of the database, so is not cheap, but provides near instantaneous shipping of transaction data to one or more remote databases, keeping them up to the second in sync with the production instance. These standby databases can be very remote, across the country, or around the globe.


Oracle Standby Database (Standard Edition)


The Standby technology is the heart of Oracle’s Data Guard solution, and is available with all versions of the software, including Standard Edition. However the standard edition does not provide the software layering on top to make it seemless, synchronous, and trivial to install. With standard edition you must settle with manual standby database. For many enterprises, the cost savings is worth being behind the production system by perhaps fifteen minutes.


Heavyweight Internet Group has just such a solution, so if you’re looking to implement a standby database, and don’t want to buy Oracle EE, please contact us for details.


Tom Kyte’s article on standby versus RAC and another standby on Oracle SE


MySQL Clustering


In the MySQL realm, you have clustering available to you. Using the NDB storage engine (as opposed to MyISAM or InnoDB), you get clustering built in. Performance on those clustered tables is likely to be a bit slower than other storage engines, but you have the confidence that your data is redundant. This however, does not provide a geographically remote solution. For that you will need to look at MySQL Replication.


MySQL Replication


MySQL Replication can keep various tables in sync, but is best used with a single master. Your databases can be geographically remote, however it is best to modify the application to point all changes towards one master node.


Conclusions


High Availability can be, like everything in computing a loaded term, or the proverbial can of worms. However, having a good sense of real-world risks, as well as what can fail in your computing environment, from hardware to software bugs, to human error, all taken together you will get a better perspective on what is achievable. Only then can you formulate reasonable expectations, and plan for your business needs.

2. Upcoming Speaking Engagements


March 15th, New York Oracle User Group


in bed with Oracle – Lifting The Covers On Database Creation

Database creation, because of better GUI tools, has become a more & more overlooked area of Oracle. We pull back the covers, revealing what Oracle is doing at each stage. Why do we have startup nomount, mount, restrict, and open? What OS resources is Oracle using at each step? How do we issue CREATE DATABASE? What is the simplest init.ora file? How many file descriptors does Oracle use and why? From conception to birth, our microscope will reveal the secrets.


April 15th, Collaborate 2007 – Las Vegas Nevada


in bed with Oracle – Lifting The Covers On Database Creation

Database creation, because of better GUI tools, has become a more & more overlooked area of Oracle. We pull back the covers, revealing what Oracle is doing at each stage. Why do we have startup nomount, mount, restrict, and open? What OS resources is Oracle using at each step? How do we issue CREATE DATABASE? What is the simplest init.ora file? How many file descriptors does Oracle use and why? From conception to birth, our microscope will reveal the secrets.

3. New Articles

Oracle 10g RAC versus DataGuard for High Availability

4. Audio Interviews

This month we have the opportunity to talk with William Hurley aka Whurley, the Chairman of the Open Management Consortium.

In our interview, we discuss open-source, and it’s impact on commercial software and solutions, and wrestle head on with some of the concerns people have on both sides of the fence.

William Hurley is the CTO at Qlusters, where he launched the openQRM project. He has been awarded IBM’s Master Inventor title, multiple awards for innovation at Apple Computer. Prior to joining Qlusters he was CTO and founder at Symbiot. He holds 11 patents for research and development at IBM, Tivoli Systems, and Apple Computer. He was recently elected Chairman of the Open Management Consortium.

5. Current Reading

How to Win Friends and Influence People – Dale Carnegie

Carnegie’s classic book is a great read. It is full of anecdotes, and simple rules to help you understand people better, get along with people better, and allow you to think as though you were in their shoes.


The Meaning of Lost and Mismatched Socks – Perditus Pedale

Dr Pedale takes us along a humorous and enlightening tale, using the metaphor we’re all familiar with, lost socks. A lighthearted read, it nevertheless underlines some of our more quirky human habits, and lets us see them in a new light.


The Wal-Mart Effect: How the World’s Most Powerful Company Really Works – Charles Fishman

Fishman brings the much maligned company to light, discussing them in the context of globalization, and how they impact our economy. As one reviewer notes, “like it or not” they are a mega-corporation that has turned many things, including everyday prices, upside down.


6. Lightweight Humor

Visit Gaping Void’s funny entry:

Fake Walmart Blog.

7. Miscellaneous


Improve Body Language:


Presentation Zen – Lessig Method


Dick Hardt’s famous Identity 2.0 presentation

8. Past Issues

Issue 27: Fragile Foundations

Issue 26: Logistical Fitness

Issue 25: Which Red Button
Issue 24: Consulting Conflicts of Interest
Issue 23: Devil In The Details
Issue 22: Beware of Software Fashion
Issue 21: Open Season, Open Sesame?
Issue 20: Better Web Better Business
Archive: Past Issues

9. Technical Articles

Oracle DBA Interview: click here
Tools for the Intrepid DBA: click here
Oracle9i + RAC on Linux/Firewire: click here
Migrating MySQL to Oracle: click here
MySQL Disaster Recovery: click here

10. About Heavyweight Internet Group

In a nutshell, Oracle. Everything related to and surrounding the database technology we specialize in, but specifically setup, admin and tuning of Oracle technology. I have 10 years experience with Oracle, wrote a book on the technology, and write and lecture frequently. I’m founder and senior consultant of the company. In capacities where your company might hire Deloitte, AIG, or Oracle Consulting we can bring the same level of service and experience, at about half the price. Simple equation.

Looking for top-flight a DBA? Visit us on the web at iheavy.com.

Part 2: RAC/Linux/Firewire – Basic Costs + Hardware Platform Outline

Basic Costs + Hardware Platform Outline

In my test environment, I bought the following equipment. Note that although RedHat Advanced Server seems to be required, I worked with the development team to get it working without that distribution, and included RPMs. If you want to get a copy, get the developer release. I listed that as well, though I didn’t use it.

  • 2x emachines T2460 $650 each link
  • 2x Inland Firewire PCI card $25ea from Fry’s (includes 6pin to 4pin cables) link
  • 1 Pyro 1394 Firewire cabinet $150 (includes 2 + 1 6pin to 6pin cable) ** link
  • 1 Maxtor 7200RPM 60GB ADA/EIDE harddisk $80 link
  • 1 2meter 6pin to 6pin 1394 cable ($10)
  • 1 copy of RedHat AS 2.1 Developer Edition $60 link
  • 1 firewire hub (only for 3+ nodes) $40-$80

You can use just about any EIDE HD which is compatible with the cabinet you get, and these are just the ones I got, so there is some flexibility in cost. Also, I got this stuff from a Fry’s store when I was in California. They have an online store at Fry’s. I would also recommend checking Sparco online as they have pretty good prices, and I’ve had a lot of luck with them here.

** Arup Nanda notes that you must use a firewire enclosure which has a chipset that supports multi-user. I would suggest

checking Tom’s Hardware Guide for details.

Part 1 – Introduction

Part 2 – Basic Costs + Hardware Platform Outline

Part 3 – Software Requirements, Versions, etc

Part 4 – Initial Oracle Setup

Part 5 – Firewire + OCFS Setup

Part 6 – Cluster Manager Setup

Part 7 – Cluster Database Setup

Part 8 – Review of Clustered Features + Architecture

Part 9 – A quick 9iRAC example

Part 10 – Summary

Oracle 9iRAC – Clustering on Linux/Firewire

Introduction

Ever since the announcement of Oracle 9i, Oracle’s Real Application Clustering feature has created quite a stir. For those not familiar, 9iRAC is a complete overhaul of Oracle Parallel Server (OPS) from previous versions of the database into a workable product.

For many DBA’s, however, this technology is completely out of reach. Without an employer who has already committed to OPS and wants to upgrade, or a client who would like to venture into the unknown, there’s no way to get ahold of an environment on which to test it. The lowest entry option for clustering technology has been Fibre Channel. Unfortunately cost is prohibitive.

Enter Oracle’s new Linux Firewire project. To some, this announcement is as exciting as Oracle’s first announcement of a port of their RDBMS to the Linux platform. Through the release of various Open Source software components, such as a modified ieee1394 driver for sharing external Firewire disks, a clustered filesystem (OCFS), as well as a number of other interesting components, this platform is now within our reach at very low cost.

Part 1 – Introduction

Part 2 – Basic Costs + Hardware Platform Outline

Part 3 – Software Requirements, Versions, etc

Part 4 – Initial Oracle Setup

Part 5 – Firewire + OCFS Setup

Part 6 – Cluster Manager Setup

Part 7 – Cluster Database Setup

Part 8 – Review of Clustered Features + Architecture

Part 9 – A quick 9iRAC example

Part 10 – Summary

Tracking the Wily Proxy Hackers

Recently the server that hosts our business was hacked. This interrupted the service of twelve different websites we host, as well as our corporate mail. Needless to say it caused us plenty of headaches, sleepless nights, and frustrating hours. In retrospect, however it has instilled a greater appreciation for computer security, a greater awareness, and further, a stronger perseverence to keep the systems locked down.
Watching the news these days, and sites like Security Focus can be disheartening to say the least. SPAM is at an all-time high, windows viruses, trojans, and malware are wreaking havoc to corporate intranets, and the internet at large, and the situation only seems to get worse. Running a server on the internet nowadays is like opening shop in New York City back in the days of street crime and daily trouble.
Unfortunately some of us in the Unix and Macintosh world have grown a bit too confident. With all of the vulnerabilities being found in various versions of Windows, IIS and Internet Explorer, folks on the other side of the fence figure they have less to worry about. We may have less to worry about, but that certainly doesn’t mean nothing. So here is the story of what happened to us, and what we did about it.
We upgraded our systems in December of 2004, and figuring Mandrake 9.2 was more stable than 10.x we installed that. We spent the time recovering all of our websites from backups, rsyncing things accross the internet. Each website has it’s own document root as well as specific configuration lines in the Apache httpd.conf file. In addition the mail server had to be configured, as well as DNS changes. Lastly once the system was up and running, we mirrored everything on root for redundancy and protection of loss of a single drive. All told we spent about 30+ hours but we were back up and running soon enough. A lot of the bulk of that time was spent moving data accross the internet, and was unattended.
Around the end of January we started seeing some spikes in hits on some of our sites, but didn’t think much about it. A few weeks went by, but generally the systems were behaving normally, but starting to be a bit slow. By mid-February we were starting to have problems. The network we are hosting on was having trouble with bandwidth, browsing, and experiencing outages of their own. We also showed up on the Composite Blocking List and the Spamhaus List.
When that happened it opened our eyes, if only a bit. We knew something was happening which was originating from that network. So we did two things. First we tested our Postfix mail server for Open Mail Relay. We had experienced this a year earlier with a qmail misconfiguration, and since it is quite common, thought this might be the problem. However, we were setup correctly, and that was not the issue. Next, we scanned all of the windows and Macintosh machines on that network for viruses, trojans, and so on. We found a couple of things, and fixed them. We then removed ourselves from the CBL + spamhaus lists.
Once again our mail was flowing out, but a day later, the problem struck again. Being the Unix folks we our, we starting pointing fingers at the Windows machines. Sometimes Norton, MacAfee et al. don’t catch all the viruses. We suspected those pesky windows machines to be the culprit. Many of the malware programs that Windows users unwittingly install on their machines relay spam so that spammers can send email out anonymously. So your windows machine is coopted as a spam host, sending out thousands of messages a minute.
To get around the problem in the short term, we contacted some associates of ours, to relay mail through them. This is different than an open mail relay, since you are specifically requesting permission to send mail through another agent. So we could once again send mail, and our problem was temporarily solved. However, our server got slower, and so did our websites. It got to the point where the network hosting our server couldn’t send outbound traffic, or visit websites. Quite a problem.
The admin managing that network contacted Verizon, the broadband provider, and discussed the problem with their tech department. They suggested unplugging machines on the network one-by-one, until the traffic spike subsided. He proceeded to do just that, and what do you know but when our server was unplugged, the bandwidth usage dropped to ZERO. The support rep suspected we were streaming audio or video files, which of course we were not, so the only obvious conclusion was spam.
What to do, well first hide your head between your tail, and admit that your unix server has been hacked is a start. Next we rebuilt the server with Mandrake 10.1. There were some vulnerabilities in SSH that we were using, as well as Apache, and PHP, so upgrading to the latest Mandrake distro version upgraded all these packages in one go. We broke our mirrored drives, and installed Mandrake on one of them, and the did a disk to disk copy of all the data from /home to the new drive. Once that was complete we started up again, and things were looking good.
Back on the internet, things started slowing down again, so we started monitoring our Apache logs. We saw some strange activity in there, so blocked HTTP at the router, and found the performance problems, and bandwidth problems eliminated. So we knew there was something wrong with Apache. We searched for bugs, but didn’t find anything too heinous. Upon closer examination of the logs, however, we found strange redirects to port 25 on other machines. How was that happening?
Apache has a facility for acting as a proxy. That is it can get webpages, and in fact make other requests of remote machines, and proxy those requests back to an originating source. Imagine standing on a mountain top. You can see to the other side of the mountain, and are reading smoke signals from a village there. You then send those same smoke signals to the next village over. They can read your smoke signals, but don’t know the identity of the sender, only that you’re sending a message to them. You can understand the message, but can’t determine the sender. Proxying with internet based servers works much the same way. In fact the Open Mail Relay we discussed above is exactly that, which is why it’s so important that it be closed.
So we looked over these logs and found strangely that Apache was doing the same thing! In fact Apache was an open mail relay, and open proxy in general. This mod_proxy module came preinstalled with our apache, and though we did not configure it, it was working none the less. So we researched the issue, and found it was not considered a bug. It was in fact part of the software that when configured correctly can come in quite handy. Of course we didn’t need it, so we spent some time disabling through configuration changes in the httpd.conf. Despite these changes, we were still seeing some traffic, so we decided to play rough. We recompiled apache from scratch with the module completely disabled. Further attempts to configure httpd.conf using that module failed, proving to us that it was indeed no longer present in the software.
We disabled the block at our router, and watched things for a couple of days. We were still seeing funny traffic. Paranoid at this point, we blocked at the router, to analyze the logs some more. We could not figure out how this might still be happening, and checked the PHP forums for bugs related to this. Finding none, and not wanting to just start recompiling modules at random, we looked at the logs again.
We found that our server, when making a failed request, was redirecting the user to our homepage. So the proxy requests were failing, but redirecting the user to our homepage. Checking the stats confirmed this. We received 5000 hits that day, a 1000% above normal. Realizing these scans and attempts to proxy were failing, we began to relax. Knowing we were probably on some spammers top-10 hacked sites in North America list, we also figured that their automated systems would remove us from such a list once our server stopped server proxy requests. And that’s exactly what we found. After a couple days the hits dropped off to 2500, and then back below 1000 before weeks end.

Open Insights 27: Fragile Foundations

OPEN INSIGHTS Newsletter
Issue 27 – Fragile Foundations
January 5, 2007

by Sean Hull

Founder and Senior Consultant
Heavyweight Internet Group

Happy New Year and welcome back to our Open Insights newsletter. Our readership is
now north of 3000 subscribers and growing everyday. Thanks to everyone for your
support and for forwarding us on to friends and colleagues!

Reading from your blackberry or other handheld device? We’ve made some formatting
changes which we hope improve the appearance on mobile devices. Let us know
if you have any suggestions or comments.


In This Issue:

1. Feature: Fragile Foundations
2. New Articles
3. Audio Interviews
4. Current Reading
5. Lightweight Humor
6. Past Issues
7. Technical Articles
8. About Heavyweight Internet Group


1. Feature: Fragile Foundations

The technologies we have surrounded ourselves are tremendously powerful and liberating. With many of the newest mobile phones, it’s quite easy to carry our office in our pocket. And with undersea cables wiring the globe, we have almost limitless telephone, and data capabilities turning the world into one very global marketplace. But as we’ll see in a couple of very familiar examples, these technologies remain quite fragile.

The Pacific Cable

If you follow international news, you probably heard about the earthquake in Asia that took out a few of the Pacific undersea cables. These provide internet & telephone connectivity from Taiwan to North America. Although there are four cables, two of them were damaged, leaving them at 40% normal capacity. Very quickly the bottleneck caused havoc in businesses, especially those doing financial transactions. Pen and paper quickly came back in vogue as businesses scrambled to hold things together. Fragile indeed.

A Story Closer to Home

I rang in the New Year with nary a blip on the radar. Clients were happy, and things were
generally quiet. Just three days in, and I drop my mobile phone. Now I’m not normally one to
be careless, but this was not a drop on the concrete, but rather just knocking it from the couch to
the floor. Sounds harmless enough, until I pick it up to see the screen begin to fizzle out, and
then go completely blank. Fear strikes. Panic, what am I going to do without my phone.
After trying to turn it on and off a few times, the first thing I thought of was, I need my contacts
and numbers. Being that my phone is a T-Mobile sidekick, I login to their website to find my contacts all faithfully stored, and available to me. I dig through the menus and find I cannot
export them to a simple datafile. In fact all I can really do is print them, all 18 pages worth!
I choose to print two pages per sheet to simplify things, and then get on the phone with
T-Mobile. After the usual struggle to get a helpful person on the phone, they attempt to transfer
me to the right department, only to drop the call. I can already see this turning into an all-day
affair.

After getting back on the line with T-Mobile, I spend 3/4 of an hour convincing the manager that
I did not do any undue damage to the phone, it effectively just stopped working. He finally accepts my explanation, and provides details on sending it to T-Mobile for replacement. Fortunately
there is a UPS store right here in Union Square, and luckily, they’re open for another hour! I rush over to the store, and the woman smiles at me. We get a lot of those phones in here, battery trouble, screen problems, etc etc. I know just where to send it. Do you have the Department Number?

Luckily I have T-Mobile, which uses SIMM cards. By removing the SIMM and placing it in my old RAZR phone, I’m back online after 20 minutes, albeit without all my contacts. But the experience as many of you can attest, is not rare. In fact I find every few weeks a friend or colleague returns a message left on voicemail or text, saying “Sorry, who is this? I lost my phone, and all my contacts…” or something to that affect. Unfortunately mobile phones still are not built to be interchangeable, grabbing information from the internet, though that technology is evolving.

Some Lessons

The first thing you’re probably likely to think after something like this is behind you is, thank god that’s over. But as quickly as it becomes a faded memory, a systems failure can happen again. What both of these examples illustrate, one small, and one huge, is we should strive first to build robust systems, second to have redundant copies of important information, and lastly to have redundant ways to perform the same business function.

In cases where systems cannot be built as robust as we would like, we need to be doubly vigilant, and build reasonable expectations into our business and our contracts. In addition we should consider these risks when we choose a solution. In some cases we may do well to choose lower tech solutions, out of technologies that have become commodity, those that are easy to replace, and where problems and issues are already well understood through years of past experience. It’s likely that those solutions won’t be as sexy as cutting edge technology, but when your business and reputation are at stake, what’s more important is consistency and reliability.

2. New Articles

Oracle 10g RAC versus DataGuard for High Availability

3. Audio Interviews

This month we have the opportunity to talk with William Hurley aka Whurley, the Chairman of the Open Management Consortium.

In our interview, we discuss open-source, and it’s impact on commercial software and solutions, and wrestle head on with some of the concerns people have on both sides of the fence.

William Hurley is the CTO at Qlusters, where he launched the openQRM project. He has been awarded IBM’s Master Inventor title, multiple awards for innovation at Apple Computer. Prior to joining Qlusters he was CTO and founder at Symbiot. He holds 11 patents for research and development at IBM, Tivoli Systems, and Apple Computer. He was recently elected Chairman of the Open Management Consortium.

4. Current Reading

If You Don’t Have Big Breasts, Put Ribbons on Your Pigtails: And Other Lessons I Learned from My Mom by Barbara Corcoran

Barbara Corcoran is the CEO of The Corcoran Group, the famous real estate company in New York City. By comparing the lessons she learned from her mother growing up, she puts her wisdom within our grasp. In story after story, she retells how she built her business offering up the successes and
failures for us to benefit from.

The Scientist in the Crib
by Andrew N. Meltzoff, and Patricia K. Kuhl

In this very readable book, Meltzoff and Kuhl discuss some of the latest discoveries in language, and human development, offering us insight into who we are, how we learn, and ultimately what makes
us tick.

The 48 Laws of Power by Robert Greene

Greene has distilled down human nature into forty eight simple laws. All of them we may not like to hear, but I think they resonate with how people are, whether that’s the way we’d like them to be. Use the laws to limit the power of others over you, or to build your own empire!


5. Lightweight Humor

The onion strikes again with Americans Celebrate 10 Millionth ‘Bring Yourself To Work Day’

6. Past Issues
Issue 26: Logistical Fitness
Issue 25: Which Red Button
Issue 24: Consulting Conflicts of Interest
Issue 23: Devil In The Details
Issue 22: Beware of Software Fashion
Issue 21: Open Season, Open Sesame?
Issue 20: Better Web Better Business
Archive: Past Issues

7. Technical Articles

Oracle DBA Interview: click here
Tools for the Intrepid DBA: click here
Oracle9i + RAC on Linux/Firewire: click here
Migrating MySQL to Oracle: click here
MySQL Disaster Recovery: click here

8. About Heavyweight Internet Group

In a nutshell, Oracle. Everything related to and surrounding the database technology we specialize in, but specifically setup, admin and tuning of Oracle technology. I have 10 years experience with Oracle, wrote a book on the technology, and write and lecture frequently. I’m founder and senior consultant of the company. In capacities where your company might hire Deloitte, AIG, or Oracle Consulting we can bring the same level of service and experience, at about half the price. Simple equation.

Looking for top-flight a DBA? Visit us on the web at iheavy.com.

Your Database – A Long-Haul Truck Or A Sports Car?

Introduction

Databases, even those running high-powered software like Oracle, can be incredibly touchy, demanding, and even fragile in their own way. For that reason we must take care to optimize and tune them based on characteristic usage.

Data Warehouse vs OLTP

In broad terms, database applications are divided into two large classes, Data Warehouse, and OLTP (online transaction processing – what a mouthful!). For the purposes of this dicussion, let’s call them a heavy lifting truck, and sports car. Now both have powerful engines, but they’re used for very different purposes.

Our Data Warehouse is characterized by large transactions, huge joins, all of which work to produce very large usually one-off reports. The reports may be run a handful of times. These databases do mostly read-only activity, occaisionally performing large dataloads to add to the archive of data.

On the other hand OLTP databases are characterized by thousands or tens of thousands of very small transactions. Web sites for instance, exhibit this characteristic. Each transaction is doing something quite small, but in aggregate, thousands of users put quite a heavy, and repeated load on the database, and they all expect instantaneous response!

We provide these two very different types of databases by laying out the database for its characteristic usage, and then tuning relevant parameters appropriately. We may allocate more memory to sorting, and less to the db cache for a Data Warehouse, whereas a large db cache might help us a lot with an OLTP application. We may enable parallel query, or partition large tables in our Data Warehouse application.

Choose One Or The Other

If you have a database serving a web-based application, and you are trying to do large ad-hoc reports against it, you will run into trouble. All that memory you’ve setup to cache small web transactions, will get wiped out with the first large report you run. What’s more the heavy disk I/O you perform reading huge tables, and then sorting and aggregating large datasets will put a huge load on your database which you setup specifically for your web application.

Conclusion

There are lots and lots of parameters and features in Oracle best suited to one or the other type of application, to the point where your database really will look like a sports car or a long-haul truck when you’re done tuning it. For that reason it is really essential that these types of applications be divided up into separate instances of Oracle preferrably on separate servers.

Oracle 10g Laptop/nodeless RAC Howto

INTRODUCTION

————

Let me start by saying that this whole article is rather unorthodox.

That’s why I thought the analogy of hitchhiking was so apt. Also

if you are a hitchhiker, and you happen to be carrying your laptop,

well you can bring along an Oracle 10g Real Application Cluster

to show all your friends. Now there’s a road warrior!

Seriously though, this step-by-step guide does not describe any

supported solution by Oracle. So what good is it? Well a lot

actually. By taking the uncommon route, you often see the

colorful streets, the surprising highways, and undiscovered

nooks. What a great analogy because this is also true in software!

In 2002 I went to Open World and was very excited by the discovery

of the Open Source initiative headed up by Wim Coekerts. One of

their most exciting projects to me was this Oracle 9i RAC running

on Linux with a special Firewire driver patched to allow multiple

systems to mount the same filesystem. That was a tremendous learning

experience, and I wrote up an article, and presented that at the

New York Oracle User Group. Afterward, someone from the audience

came to me and asked if I would present at their user group out

in Edison New Jersey. Both presentations were exciting, and I

think well received.

Of late I’ve happened upon some articles floating around the internet

discussing a single-node AKA laptop RAC setup. How could that work,

I wondered? Perhaps they install virtualization software such as

VMWare to allow a single machine to look like more than one. This

would certainly work in theory. Once I started digging a bit more I

discovered Amit Poddar’s excellent article over at dizwell.com, and

I was intrigued. No virtualization seemed to be required other than

some virtual ethernet interfaces.

I decided to dig my heels in and give it a try. After struggling with

the illustrious and very universally beloved installer for a few months

I finally managed to get all the pieces in place, and get Oracle Real

Application Clusters running with no clustering hardware!!!

Here, my fine friends, are all the gory details of that adventure, which

I hope you’ll enjoy as much as I did writing them down.

Oracle’s model of clustering involves multiple instances (software processes)

talking to a single database (physical datafiles). We’re doing the same

thing here, but both instances will reside on the same machine. This

helps you travel light, save on transportation and learn concepts,

commands, and the architecture. Remember this is not an HA solution,

as there is little redundancy, and with a single disk, you will surely get

abysmal performance.

Let’s get started, what will we need to do? Here’s a quick outline of the

steps involved:

1. setup ip addresses of the virtual servers

2. setup ssh and rsh with autologin configured

3. setup the raw devices Oracle’s ASM software will use

4. install the clusterware softare, and then Oracle’s 10g software

5. setup the listener and an ASM instance

6. create an instance, start it, and register with srvctl

7. create a second instance & undo tablespace, & register it

1. Setup IP Addresses

———————-

Oracle wants to have a few interfaces available to it. To follow our analogy

of a hitchhiker traveling across America, we’ll name our server route66.

So add that name to your /etc/hosts/ file along with the private and vip

names:

192.168.0.19 route66

192.168.0.75 route66-priv

#192.168.0.76 route66-vip

Notice that we’ve commented out route66-vip. We’ll explain more about

this later, but suffice it to say now that the clusterware installer

is very finicky about this.

In order for these two additional names to be reachable, we need

ethernet devices to associate with those IPs. It’s a fairly straightforward

thing to create with ifconfig as follows:

$ /sbin/ifconfig eth0:1 192.168.0.75 netmask 255.255.255.0 broadcast 192.168.0.255

$ /sbin/ifconfig eth0:2 192.168.0.76 netmask 255.255.255.0 broadcast 192.168.0.255

If your IPs, or network is configured differently, adjust the IP or broadcast

address accordingly.

2. setup ssh and rsh with autologin

————————————

Most modern Linux systems do *NOT* come with rsh installed. That’s for

good reason, because it’s completely insecure, and shouldn’t be used at

all. Why Oracle’s installer requires it is beyond me, but you’ll need

it. You can probably disable it once the clusterware is installed.

Head over to http://rpmfind.net and see if you can find a copy for

your distro. You might also have luck using up2date or yumm if you

already have those configured, as they handle dependencies, and always

download the *right* version. With rpm, install this way:

$ rpm Uvh rsh-server-0.17-34.1.i386.rpm

$ rpm Uvh rsh-0.17-34.1.i386.rpm

Next enable autologin by adding names to your /home/oracle/.rhosts file.

After starting rsh, you should be able to login as follows:

$ rsh route66-priv

Once that works, move on the the sshd part. Most likely ssh is already on

your system, so just start it (as root):

$ /etc/rc.d/init.d/sshd start

Next, as the “oracle” user, generate the keys:

$ ssh-keygen -t dsa

Normally you would copy id_dsa.pub to a remote system, but for us we

just want to login to self. So copy as follows:

$ cd .ssh

$ cp id_dsa.pub authorized_keys

$ chmod 644 authorized_keys

Verify that you can login now:

$ ssh route66-priv

3. setup the raw devices

————————-

Most of the time when you think of files on a Unix system, you’re

thinking of files as represented through a filesystem. A filesystem

provides you a way to interact with the underlying disk hardware

through the use of files. A filesystem provides buffering, to

improve I/O performance automatically. However in the case of a

database like Oracle, it already has a sophisticated mechanism for

buffering which is smart in that it knows everything about it’s

files, and how it wants to read and write to them. So for an

application like Oracle unbuffered I/O is ideal. It bypasses a

whole layer of software, making your overall throughput faster!

You achieve this feat of magic using raw devices. We’re going to

hand them over to Oracle’s Automatic Storage Manager in a minute

but first let’s get to work creating the device files for our

RAC setup.

Create three 2G disks. These will be used as general storage space

for our ASM instance:

$ mkdir /asmdisks

$ dd if=/dev/zero of=/asmdisks/disk1 bs=1024k count=2000

$ dd if=/dev/zero of=/asmdisks/disk2 bs=1024k count=2000

$ dd if=/dev/zero of=/asmdisks/disk3 bs=1024k count=2000

Create two more smaller disks, one for the Oracle Cluster Registry,

and another for the voting disk:

$ dd if=/dev/zero of=/asmdisks/disk4 bs=1024k count=100

$ dd if=/dev/zero of=/asmdisks/disk5 bs=1024k count=20

Now we use a loopback device to make Linux treat these FILES as

raw devices.

$ /sbin/losetup /dev/loop1 /asmdisks/disk1

$ raw /dev/raw/raw1 /dev/loop1

$ chown oracle.dba /dev/raw/raw1

You’ll want to run those same three commands on disk2 through disk5 now.

4. Install the Clusterware & Oracle’s 10g Software

————————————————–

Finally we’re done with the Operating System setup, and we can move on

to Oracle. The first step will be to install the clusterware. I’ll

tell you in advance that this was the most difficult step in the entire

RAC on a laptop saga. Oracle’s installer tries to *HELP* you all along

the way, which really means standing in front of you!

First let’s make a couple of symlinks to our OCR and voting disks:

$ ln -sf /dev/raw/raw4 /home/oracle/product/disk_ocr

$ ln -sf /dev/raw/raw5 /home/oracle/product/disk_vot

As with any Oracle install, you’ll need a user, and group already

created, and you’ll want to set the usual environment variables such

as ORACLE_HOME, ORACLE_SID, etc. Remember that previous to this point

you already have ssh and rsh autologin working. If you’re not sure

go back and test again. That will certainly hold you up here, and

give you all sorts of confusing error messages.

If you’re running on an uncertified version of Linux, you may want

to fire up the clusterware installer as follows:

$ ./runInstaller -ignoreSysPrereqs

If your Linux distro is still giving you trouble, you might try

downloading from centos.org where you can find complete ISOs for

RHEL, various versions. You can also safely ignore memory warnings

during startup. If you’re short on memory, it will certainly slow things

down, but we’re hitchhikers right?

You’ll be asked to specify the cluster configuration details. You’ll

want route66-vip to be commented out, so if you haven’t done that and

get an error to the affect of route66-vip already in use go ahead and

edit your /etc/hosts file.

I also got messages saying “route66-priv not reachable”. Check again

that sshd is running, and possibly disable your firewall rules:

$ /etc/rc.d/init.d/iptables stop

Also verify that eth0:1, and eth0:2 are created. Have you rebooted

since you created them? Be sure they’re still there with:

$ /sbin/ifconfig -a

Specify the network interface. This defaults to PRIVATE, just edit

and specify PUBLIC.

The next two steps ask for the OCR disk and voting disk. Be sure to

specify external redundancy. This is your way of telling Oracle that

you’ll take care of mirroring these important disks yourself, as loss

of either of them will get you in deep doodoo. Of course we’re

hitchhikers so we’re not trying to build a system that is never going

to breakdown, but rather we want to get the feeling of the wind blowing

in our hair. Click through to install and you should be in good shape.

At the completion, the installer will ask you to run the root.sh

script. I found this worked fine up until the vipca (virtual ip

configuration assistant). I then ran this one manually. You’ll need

to uncomment route66-vip from your /etc/hosts file as well. Once

all configuration assistants have completed successfully, return to

the installer and click continue, and it will do various other sanity

checks of your cluster configuration.

Since the clusterware install is rather testy, you’ll probably be doing

it a few times before you get it right. Here’s the cleanup if you

have to run through it again:

$ rm rf /etc/oracle

$ rm rf /home/oracle/oraInventory

$ rm rf $CRS_HOME

# these two commands cleanup the contents of these disks

$ /bin/dd if=/dev/zero of=/asmdisks/disk4 bs=1024k count=100

$ /bin/dd if=/dev/zero of=/asmdisks/disk5 bs=1024k count=20

$ rm /etc/rc.d/init.d/init.crs

$ rm /etc/rc.d/init.d/init.crsd

$ rm /etc/rc.d/init.d/init.cssd

$ rm /etc/rc.d/init.d/init.evmd

$ rm /etc/rc.d/rc3.d/S96init.crs

$ rm /etc/rc.d/rc5.d/S96init.crs

Now reboot the server. This will kill any clusterware processes still running.

If you’ve finished the Oracle Universal Installer at this point, and things

seem to be working, check at the command line with the ps command:

$ ps auxw | grep 10.2.0s

root 3728 0.0 0.3 2172 708 ? S May30 0:00 /bin/su -l oracle -c sh -c ‘ulimit -c unlimited; cd /home/oracle/product/10.2.0s/log/bebel/evmd; exec /home/oracle/product/10.2.0s/bin/evmd ‘

root 3736 0.0 5.0 509608 11240 ? S May30 4:51 /home/oracle/product/10.2.0s/bin/crsd.bin reboot

oracle 4047 0.0 2.8 192136 6284 ? S May30 0:00 /home/oracle/product/10.2.0s/bin/evmd.bin

root 4172 0.0 0.3 2164 708 ? S May30 0:00 /bin/su -l oracle -c /bin/sh -c ‘ulimit -c unlimited; cd /home/oracle/product/10.2.0s/log/bebel/cssd; /home/oracle/product/10.2.0s/bin/ocssd || exit $?’

oracle 4173 0.0 0.4 4180 900 ? S May30 0:00 /bin/sh -c ulimit -c unlimited; cd /home/oracle/product/10.2.0s/log/bebel/cssd; /home/oracle/product/10.2.0s/bin/ocssd || exit $?

oracle 4234 0.0 3.9 180108 8808 ? S May30 0:16 /home/oracle/product/10.2.0s/bin/ocssd.bin

oracle 4476 0.0 2.0 24048 4576 ? S May30 0:00 /home/oracle/product/10.2.0s/bin/evmlogger.bin -o /home/oracle/product/10.2.0s/evm/log/evmlogger.info -l /home/oracle/product/10.2.0s/evm/log/evmlogger.log

oracle 6989 0.0 0.1 2676 404 ? S May30 0:00 /home/oracle/product/10.2.0s/opmn/bin/ons -d

oracle 6990 0.0 1.9 93224 4280 ? S May30 0:00 /home/oracle/product/10.2.0s/opmn/bin/ons -d

oracle 7891 0.0 0.2 3676 660 pts/3 S 23:35 0:00 grep 10.2.0s

Also list the nodes you have available to your clusterware software:

$ olsnodes n

route66 1

The Oracle 10g install itself is very trivial, assuming you’ve installed

Oracle before. Use the -ignoreSysPrereqs flag if necessary to start

up the Oracle Universal Installer, and use the software-only option,

as we’ll be creating our RAC database by hand. Also select Enterprise

Edition and things should proceed smoothly. Oracle will recognize that

you have the clusterware installed, and let you know during the

installation.

5. setup the listener and an ASM instance

——————————————-

The listener.ora file is setup as usual, the only difference is you will

include both route66 and route66-vip.

SID_LIST_LISTENER =

(SID_LIST =

(SID_DESC =

(SID_NAME = PLSExtProc)

(ORACLE_HOME = /home/oracle/product/10.2.0)

(PROGRAM = extproc)

)

)

LISTENER =

(DESCRIPTION_LIST =

(DESCRIPTION =

(ADDRESS_LIST =

(ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC))

)

(ADDRESS_LIST =

(ADDRESS = (PROTOCOL=TCP)(HOST=route66)(PORT = 1521)(IP = FIRST))

)

(ADDRESS_LIST =

(ADDRESS = (PROTOCOL=TCP)(HOST=route66-vip)(PORT = 1521)(IP = FIRST))

)

)

)

You may also choose to use the actual IP addresses of these hostnames if

you like. One other difference is that you will use Oracle 10g’s new

srvctl utility to start th listener.

$ srvctl start nodeapps -n route66

Ok, on to the fun stuff. It’s time to configure our ASM instance.

Our instance name will be +ASM1, and we’ll set the ORACLE_SID as usual.

In the init+ASM1.ora file specify:

user_dump_dest=/home/oracle/admin/+ASM1/udump

background_dump_dest=/home/oracle/admin/+ASM1/bdump

core_dump_dest=/home/oracle/admin/+ASM1/cdump

large_pool_size=15m

instance_type=asm

asm_diskstring=’/dev/raw/raw1′, ‘/dev/raw/raw2′,’/dev/raw/raw3’

Then in sqlplus startup the ASM instance:

SQL> startup nomount

Next tell ASM how we want to utilize the space we have by creating disk

groups:

SQL> create diskgroup DBDATA external redundancy disk

‘/dev/raw/raw1’,’/dev/raw/raw2;

SQL> create diskgroup DBRECO external redundancy disk ‘/dev/raw/raw3’;

Now that you have diskgroups, you want to make a note of it in your

init.ora:

SQL> !echo “asm_diskgroups=’DBDATA’,’DBRECO'” >> init+ASM1.ora

Now startup your ASM instance:

SQL> startup force

Exit sqlplus and let srvctl know about the new ASM instance:

$ srvctl add asm -n route66 -i +ASM1 -o /home/oracle/product/10.2.0.1

Now you can shutdown in sqlplus, and startup with srvctl:

SQL> shutdown immediate

$ srvctl start asm -n route66

And lastly use the ps command to check for your new instance.

6. create an instance, start it, and register with srvctl

———————————————————

We’re getting to our clustered database slowly but surely. We’re

just getting over the mountains now, and the open road is ahead of

us.

We’re going to create the first of our two instances and call it

BEATNIK. Edit your initBEATNIK.ora as follows:

db_block_size=8192

db_multiblock_read_count=8

db_name=kerouac

BEATNIK.background_dump_dest=/home/oracle/admin/BEATNIK/bdump

BEATNIK.user_dump_dest=/home/oracle/admin/BEATNIK/udump

BEATNIK.core_dump_dest=/home/oracle/admin/BEATNIK/cdump

BEATNIK.instance_number=1

BEATNIK.instance_name=BEATNIK

BEATNIK.thread=1

BEATNIK.undo_tablespace=beatnikundo

HIPPY.background_dump_dest=/home/oracle/admin/BEATNIK/bdump

HIPPY.user_dump_dest=/home/oracle/admin/BEATNIK/udump

HIPPY.core_dump_dest=/home/oracle/admin/BEATNIK/cdump

HIPPY.instance_number=2

HIPPY.instance_name=HIPPY

HIPPY.thread=2

HIPPY.undo_tablespace=hippyundo

You’ll also have to create the /home/oracle/admin/BEATNIK/* and

/home/oracle/admin/HIPPY/* directories.

Now edit the file crKEROUAC.sql as follows:

CREATE DATABASE KEROUAC”

DATAFILE SIZE 250M EXTENT MANAGEMENT LOCAL

SYSAUX DATAFILE SIZE 125M

DEFAULT TEMPORARY TABLESPACE TEMP TEMPFILE SIZE 20M

UNDO TABLESPACE “beatnikundo” DATAFILE SIZE 200M

CHARACTER SET WE8ISO8859P1

LOGFILE GROUP 1 SIZE 10240K,

GROUP 2 size 10240k,

GROUP 3 size 10240k;

There are other parameters you can specify in this create statement, such as

maxinstances, maxlogmembers, and the sys password. However I’ve tried to

simplify it, to make it easier to review and understand. Check the Oracle

docs for details.

Now startup sqlplus and issue:

SQL> startup nomount pfile=/home/oracle/admin/BEATNIK/pfile/initBEATNIK.ora

SQL> @crKEROUAC.sql

Now get the names of your controlfiles from v$parameter and add them to

the initBEATNIK.ora file.

Now add a couple more parameters to your initBEATNIK.ora file:

*.cluster_database=true

*.cluster_database_instances=5

Use sqlplus to stop and start the db again:

SQL> shutdown immediate

SQL> startup force

Now register our new database:

$ srvctl add database d KEROUAC -o /home/oracle/product/10.2.0.1/

$ srvctl add instance -d KEROUAC -i BEATNIK -n route66

Now one more time, shutdown with sqlplus, and then use srvctl to

start the db. From now on srvctl can stop + start the db.

$ srvctl start instance -d KEROUAC -i BEATNIK

7. create a second instance & undo tablespace, & register it

———————————————————–

Since the database is already created, you don’t have to do that step

again. At this point all you have to do is create another instance.

First fire up sqlplus and create another undo tablespace:

SQL> create undo tablespace hippyundo datafile ‘+DBDATA’ size 100m;

Make a copy of your init.ora for the hippy instance like this:

$ cp initBEATNIK.ora initHIPPY.ora

Then set your ORACLE_SID and use sqlplus to startup:

SQL> startup

Finally register with srvctl:

$ srvctl add instance -i HIPPY -d KEROUAC -n route66

8. create data dictionary

————————–

SQL>@?/rdbms/admin/catalog.sql

SQL>@?/rdbms/admin/catclust.sql

10. Some Things to Understand

————————–

Automatic Storage Management

Global Cache Services

Global Enqueue Services

Clusterware procs crsd, evmd, ocssd,oprocd

RAC processes lms, lmd, lmon, lck0

GV$ data dictionary

11. Further Reading

—————–

Clusterware & RAC Install & Configuration Guide for Linux

Clusterware & RAC Administration Deployment Guide

Oracle Technology Network – http://otn.oracle.com

Please visit http://www.iheavy.com or email me at

Thanks to Amit Poddar & dizwell.com

Apress, Oracle Press books

Part 10: RAC/Linux/Firewire – Summary

Summary

We covered a lot of ground in this article and it should serve as an introduction to 9iRAC on cheap Linux hardware. There are plenty of other topics to dig into including tuning, backup, SQL*Net setup and Load Balancing, I/O Fencing, NIC Failover and so on.

9iRAC is *NOT* a silver bullet as any good DBA knows. It will protect you from a single instance failure because of memory, kernel panic, or an interconnect failure, but there are still cases where your database could go down, for instance if the cluster manager software fails, or you lose a datafile either from human error, or a storage subsystem problem. Further redundancy can help you, but there are risks associated with an accidentally deleted object, or even and Oracle software bug.

Take a look at some of the documents listed below for further reading.

Other References

Red Hat Linux Advanced Server 2.1 – docs

Oracle Technology Network’s Linux Center

Oracle Cluster Filesystem FAQ

Oracle Cluster File System docs

Internals of Real Application Clusters by Madhu Tumma

Oracle9i Real Application Clusters Concepts

Oracle9i Real Application Clusters Administration

Linux HOWTO Docs – Networking, kernel config, hardware compatability etc


Part 1 – Introduction

Part 2 – Basic Costs + Hardware Platform Outline

Part 3 – Software Requirements, Versions, etc

Part 4 – Initial Oracle Setup

Part 5 – Firewire + OCFS Setup

Part 6 – Cluster Manager Setup

Part 7 – Cluster Database Setup

Part 8 – Review of Clustered Features + Architecture

Part 9 – A quick 9iRAC example

Part 10 – Summary