All Technical Article

otop – diy monitoring of your Oracle database

Thank you all for attending my presentation at Metro Oracle Day.

Please go ahead and download otop here. You may also download the otop presentation here. The full otop article can be found here.

If you have any questions at all, or are interested in contributing, please email me at shull (at) iheavy (dot) com.


All iHeavy Newsletter

Open Insights 15 – Marketing About Technology

By Sean Hull

Founder and Senior Consultant

January 1, 2006

Happy New Year everyone, and welcome to another year of Heavyweight Internet Group’s monthly newsletter.

Please forward to interested friends and colleagues. Subscription information can be found at the end.

Please also visit our sister site Oracle and Open Source for more frequent updates.

In This Issue:

Feature: Marketing About Technology

On The Lighter Side

Past Issues

Technical Articles

Marketing About Technology

At a recent day long seminar on Oracle technology I sat in on a panel where the

audience asked questions, and the panel responded with their professional expe

riences on that technology. One question was on Real Application Clusters, and whether it was a recommended solution for High Availability in a given environment. Should we go with RAC?

What struck me at that moment was two very opposing visions. One was of a bore

d technologist, who, having implemented the perfect high availability solution has very little to do because it runs so well. He affectively lets his genius work him out of a job. The technology just runs, and he has nothing really to do! The other vision was of a marketing/sales team, who has just sold a company on the latest wizbang technology that will save the business millions. They’re very excited because they know how much revenue it will bring into the consulting company.

What’s wrong with these two pictures? Well obviously the former picture of the

technologist who has nothing to do, we all know (I hope) is a fantasy. Despite all the marketing in the world self-managing systems remain, and I think will forever remain, a figment of the imagination. The complexity of technology systems, whether from security issues, upgrades, bugs, patches, new features, or new application development, will forever keep the technology administrator busy. Now the latter picture, I hope, also sounds fantastical, that of the perfectly happy consulting group who get everything they want. The truth is with enough insight, investigation, and research you can and should educate yourself be forehand about what’s reasonable, and within reach.

But what I also think is important, and want to draw your attention to is the two distinct perspectives. The technologists perspective is, or at least should

be to simplify the technology so it’s boring. So it’s ordinary in a way that

makes the business run smoothly. That way the technology facilitates, and provides, but doesn’t hinder, or become a nuisance. Now we know all technologists may not be of the mind to simplify, but we can point in this direction. So for instance when we look at upgrading systems to the “latest and greatest” we might think twice, and imagine being on the oldest supported version, which has every known bug ironed out, and support has a good document on every issue already tweaked, honed, and polished for your use. This isn’t as glamourous as using the latest Agile Software or Grid technology, but it may well be the most reliable.

And what of the other perspective? The marketing and sales perspective is to build the business, and in the case of consulting, find challenges, and ever bigger problems for which experts will be needed. This is a natural consequence of perspective, to build that type of business.

What you as a client hiring a professional services firm should keep in mind is all of these perspectives, as they all come into play, and point different minds in different directions. Hopefully with enough foresight, you can hire the right group, with just the right balance that won’t end up creating more opportunities for themselves, than they solve for you.

On the lighter side

Everyone who enjoys the great technology of google, but also wonders from time to time if there is an end in site should check out this great piece over at the Onion Google Announces Plan to Destroy All Information It Can’t Index.

All iHeavy Newsletter

Open Insights 14 – The Tricky Database

By Sean Hull

Founder and Senior Consultant

December 1, 2005
The Tricky Database
Which of these statements fits your thinking?

“Well we want the best of the best, so we got Oracle.”

“We have expensive problems to solve so we spent handsomely on the solution.”

“We need to be up more than 99.999% of the time, so we surely need Oracle.”
The truth is technology experts can tell you exactly what the database is and does but may not have the most insight into when and how to use it most effectively. And as managers we often have the above insights about problems that need to get solved, and budgets and so on and so forth, but the two bits of intelligence are often separated by an abyss of understanding, leaving money badly spent, or real business problems half solved.
At Heavyweight Internet Group I sit in the unlikely position of having one foot in both camps. So hopefully I can extend some insight, and possibly shed a little light on some of these questions. Ok, here goes…
1. Why are databases such a complex component in the enterprise?
Well to put it mildly, everyone has their hand in there. The finance department keeps accounting, and business intelligence there, helping to answer big questions about running the business, hr, marketing, and sales all want to keep contact info there. It’s your business’s proverbial golden nugget. So it needs to be available all the time, like electricity, or the telephone system. Unfortunately it’s a much more complex beast than those technologies, and is constantly evolving too. There are backups, security, patches, and upgrades to worry about. Not to mention application tuning, when the logic behind those lengthly reports becomes unweildy, or your data volume grows.
2. What’s with database “tuning” anyway? Is Oracle the PINTO of software?
This is a very interesting question for me, primarily because I can see it from two very different angles. From the management side I see this hunk of technology that looks for all intents and purposes like a very expensive Pinto, a Jaguar automobile of old. It needs constant attention, the parts are expensive, and so are the mechanics. But when I put on my engineering cap, I can see a shining piece of engineering marvel. A machine which, when tuned properly (not an easy task I grant you) will outperform any other datastore in the world. Thousands and thousands of transactions a second can be performed, while hundreds and hundreds of users are all connected simultanously asking it their own questions.
Now I will grant you that the machine does not come out of the box tuned very well as a starter system, the principal reason for this is there are so many types of uses. There are datawarehouses, terabytes of archival data, and reports that run all night long on million row tables. And there are transactional (dubbed OLTP) systems, perhaps driving the backend of a website or ecommerce site. Databases run financial institutions, and small startup companies, each with a different profile of needs. And what also makes it complex is that each of those businesses will be running on different hardware, from mainframes, to Linux servers, and Windows XP, to Sun Solaris. Some will have EMC storage, while others will have a cheap Intel based RAID controller with six disks. Every system can be tuning precisely, but out-of-the-box Oracle doesn’t just plop down and do what you want.
3. Will monitoring save me? What about the Remote DBA?

Automated monitoring is a really excellent way to keep your systems in tip-top shape. You spend time and money at the outset, but you gain peace of mind that your infrastructure will keep rolling. There are many ways to get monitoring, from commercial software packages that you can install, which provide fancy graphical front-ends to the database, to various Open Source solutions like Nagios which provide all the power, functionality, and customizability you could ever want, but perhaps with a few less bells and whistles on the interface.
If you’re not comfortable with these solutions, you can outsource this aspect of your infrastructure. Sign a contract with a vendor who specializes in this type of service, with a specific service level agreement, guarenteed response. That way it will be in their interests to keep track of the things that could most threaten your day-to-day operations, from security concerns and backups, to rogue or errant queries that are impacting performance. Even hardware monitoring is available, so the loss of a harddrive in your RAID array is noticed well before it threatens your whole system.
4. Is 5×9 reasonable? Can we do HA?
The industry talks about 5 NINES, that is 99.999% uptime as the sort of gold standard of availability. But lets really think about that. With only three NINES, you have room for 10 minutes of downtime per week, four NINES gives you only 1 minute per week, and five NINES gives you a mere 6 seconds per week. Here’s a more detailed look at what five NINES really means.
To put that in perspective, the power grid that runs the Northeastern United States was out in August 2003 for 24 hours. New Yorkers will remember this well. The last big one was the blackout of 1965. If you do the math that’s a real outage of 38 minutes per year, or a little better than four NINES. If you took all the little power outages businesses experience, the picture gets worse. Now granted many datacenters have their own power generators, but the point remains in complex systems, even with plenty of redundancy, and elimination of human error from the mix, some downtime is inevitable.
What about High Availability? Oracle offers this in a couple of ways. One through Dataguard, formerly Standby Database, and the other through Real Application Clusters (RAC). Yes these technologies will bring you closer to five NINES, but a reasonable and real assessment of the technologies, and real-world test cases, and relative expenditures have to be considered to get a true sense of what is reasonable to expect. As more complex components are added to the mix, both hardware and software, you have more points that can fail, and more possible software bugs too.
5. What about the Open Source databases? Will they change everything?
The Open Source databases vying for your attention these days include Firebird, MySQL, Postgres, and Ingres. The question of Open Source databases becomes more and more relevant everyday. As new features are added, and they become more sophisticated and feature rich, more businesses will use them for their data. Many enterprises are sticking with the watch and wait plan, and letting others live on the bleeding edge. We covered this over at Oracle and Open Source in an article War of the Databases?.
On the lighter side

A colleague of mine recently forwarded this excellent illustration. Everyone in the business of technology projects, whether you hire outside resources, or have developers in house, would benefit from understanding this. It is a picture with ten panels, each illustrating a different perspective of a technology project, from “How the customer explained it” to “How the business consultant described it” and even “How the customer was billed”. Well worth a look, and hopefully to keep in mind as what to avoid.

All iHeavy Newsletter

Open Insights 13 – Oracle Heavy Lifting

Heavyweight Internet Group Newsletter

Issue 13 – Oracle Heavy Lifting

November 1, 2005

by Sean Hull

Founder and Senior Consultant

Heavyweight Internet Group

Welcome to our free monthly newsletter, discussing news, developments and business best practices at the intersection of Oracle and Open Source software.

Please forward to interested friends and colleagues. Subscription information can be found at the end.

We are now writing material regularly for our sister site Oracle + Open Source so check there for more frequent updates.

Oracle Heavy Lifting

In consulting, I have the unique opportunity to both think deeply about technology, how it works, and what it can do, and also about business questions, what does a business need to do, and how will technology help it achieve greater returns. I just finished pouring over Tom Kyte’s new Expert Oracle title. It is an excellent book, full of plenty of deep insights about Oracle technology. As I was reading the chapters, I noted in particular which features required enterprise edition, and which are included in standard edition. You can find my chapter by chapter review on Amazon.

As I’m engaged with many different types of clients and businesses, I have the opportunity to see Oracle in various real-world situations, solving real business problems. I came to a profound realization that most enterprises decide how much to spend on Oracle in proportion to (a) the business problem & use they are putting it to and (b) how expensive the hardware and consulting services are. In other words the expense is relative to the budget constraints, and in proportion to the problem it is solving. However, with my technology hat on I realized that though it might seem like a rational decision from the business side, you may not be doing all the heavy lifting you think you’re doing. Put another way Enterprise Edition may not be what you need, or at least more than you’re using.

Without going into a tremendous amount of technical detail, I just wanted to outline the situations where Enterprise Edition would likely be worth the money, and when it may be overkill. When Standard Edition sits at roughly 10-15% of Enterprise Edition, that’s certainly something a business should consider.

A traditional way to separate database types is into two categories, Data Warehouse and OLTP. A Data Warehouse is characterized by very few users, doing large long running queries. OLTP or OnLine Transaction Processing is characterized by hundreds of concurrent users, such as a database which serves as a backend datastore to a website. An Oracle database can be tuned to prioritize and use resources most efficiently for each type of database.

So what does this have to do with Enterprise or Standard Edition. Well as it turns out Enterprise Edition proves a very attractive option for Data Warehouses. There are features such as streams, transportable tablespaces, materialized views, advanced analytical functions, and compression to name a few. None of these features are included in Standard Edition. Additionally there are other EE options which are attractive for Data Warehouses including Partitioning, and various ways to parallelize operations to help a single session consume the resources of the entire machine. These parallel operations can speed up dataloads, rebuilds, batch jobs, and large queries against even terabytes of data.

But what about Transactional environments (OLTP)? Many businesses use Oracle as a backend datastore for their online presence, such as banks, bookstores, airlines, and just about any other dynamic website. Well there are some features such as Fine Grained Access, Virtual Private Database, and Secure Roles which aren’t available in Standard Edition. There are also the Tuning, Change Management, and Diagnostics packs that are add-on options only available with Enterprise Edition. But many folks are just looking for speed and high availability features. Will Enterprise Edition help in this area? Well you only have Data Guard in Enterprise Edition, but databases can be cloned, and that process can be scripted, and although not seemless, and an integrated feature as Data Guard is, something similar can be done in Standard Edition. So how about the Parallel Features, and the Partitioning option. Will those help with Transactional databases? Let’s start with partitioning, it is a feature which may or may not be heavily used in your environment, but if your application is OLTP, chances are you’re not getting dramatic query improvements, because your queries are characterized by small frequent accesses by lots of concurrent users. Now partitioning will help you with maintenance and availability, but in that sense it serves more as a DBA tool, than a performance boost overall. What about the parallel operations. Again mostly for DBA operations, one-off rebuilds and dataloads can be made to run quite a lot faster but the day to day operation of your database may not be dramatically impacted by this feature.

With Oracle’s recent announcement of it’s Express Edition of 10g as well as it’s purchase of Innobase a commercial component of MySQL providing solid transactional support, your options are wider and more complex than ever before. Your best bet navigating this complex landscape is to get an accessment of your current infrastructure, or do sufficient research before investing in that new project. Like buying a car, ask lots of questions, don’t believe everything you hear, and kick the tires before you buy. Both Standard and Enterprise Edition can be downloaded for single-instance development use & testing off of Oracle Technology Network.

Technical Articles at IHEAVY.COM

Coming soon… A Hitchhiker’s Guide to Oracle 10g RAC

Note: The following articles will be returning to soon. We’ve done some redesign and rebuilding, and have to add these again…

Tracking the Wily Proxy Hackers

Asterisk Calling Card Applications

MySQL Disaster Recovery

Dummy’s Guide to Linux firewalls

Wireless Truth or Dare

Mirroring for the Impatient

All iHeavy Newsletter

Open Insights 12 – What the Geeks Mean

Heavyweight Internet Group Newsletter
Issue 12 – What the Geeks Mean
October 4, 2005

by Sean Hull
<[email protected]>
Founder and Senior Consultant
Heavyweight Internet Group

Welcome to our free monthly newsletter, discussing news, developments and business best practices at the intersection of Oracle and Open Source software. 

Please forward to interested friends and colleagues.  Subscription information can be found at the end.

We’d also like you to take a look at our sister site where you’ll find other non-technical and related news and content.

What the Geeks Mean

I’m blessed for having gotten started with computers when I was very young.  Well that’s either blessed or cursed depending on how you look at it.  At any rate, having worked with computers for about 25 years, I know well what the geeks are talking about.  And having spent the last 10 years doing a lot of writing, and consulting for diverse businesses, I’ve learned to understand what the business folks are talking about.  I’ve learned business thinking, and learned about the concerns of business owners.  So this naturally leaves me in a comfortable position as International Interpreter for Geek to Suit communication. 

Well I’m just kidding of course, but seriously there is often a wide gap in understanding and I make it a point of constantly trying to bridge that gap.  So what I’m going to discuss this month are some of the big issues that come up on the technology end, and try to help you understand what your technology team is really talking about.  This will help you make better purchasing and budget allocation decisions, as some problems impact the business more seriously, or more immediately than others.  And don’t worry, we’re going to use a broad brush here, and not get too technical.  Those technical folks in the audience, be patient where analogies don’t precisely match the underlying issues.

The six sections below basically amount to areas where your systems may be encountering problems.  For each section I make a hopefully memorable and clear analogy, and then in parentheses I write the technical description of that problem that you may hear.

A. Our Disks Form a Single Lane Freeway
(Disk I/O Subsystem problems)

Disks, Disk Subsystems, or I/O Subsytems are basically the same thing.  On a PC that sits on your desk, or at home there is a normally a single harddrive which stores things while the computer is turned off.  When the system is booted, or a program is started, programs and data are loaded off that storage media into memory.  In server computing environments much the same thing happens, albeit there are many more parts.  It is often referred to as a storage subsystem because it may be composed of a stack of harddrives all managed together in one cabinet, and interfaced together in what is called a RAID.  Why should I be concerned as a manager you ask?  Well how these are configured (ie what RAID level) can dramatically impact the performance, and reliability of your SAN (Storage Area Network), effectively giving you a one lane highway in one direction (while writing to disk) while giving you a 5 lane highway in the other direction.  What’s more some configurations are more prone to crashing, and loss of a harddrive (crashing, and loss of a lane in our car analogy) than others.  

If your developers and system administrators are saying what they have is incorrect this may be because the particular server in question has a controller card which does not support the proper RAID type.  In that case new hardware would solve your problem.  External SAN/NAS solutions such as EMC or NetApp normally support all the various RAID types in whatever configuration and combinations you want.

Keep in mind that you want to keep the traffic flowing in both directions of your highway, and you want to keep the car crashes down.  Of course also want to get the most performance especially if you invested a lot on that expensive and fast car, Oracle.  Above all avoid the single lane freeway and its attendant accidents.

B. Our Network Is a Party Line
(Network + Database Security)

Some of us may recall back in the old days when you might have a party line running into your home for telephone calls.  When you picked up the telephone you might hear your next door neighbor on the telephone.  You would have to politely (or not so politely) ask them when they would be finished so you could make your call.  Beyond the inconvenience, the privacy concerns are obvious. 

In the technology world, much more of our communications are like a party line, or a postcard that you send through the mail, then you might imagine.  But how much?  And when and where should I be most concerned?

If you have a dedicated networking team, who manage the firewall, and keep patches up to date, you still may have to be concerned about wireless access to your network.  Believe it or not even encrypted WiFi connections can be breached within 10 to 15 minutes with the right tools.  You have no WiFi you say?  You only use a VPN to allow folks on the road to connect?  Well there can be vulnerabilities in all of that software, ones that can allow a hacker to breach the network, and steal valuable data. 

But even if you have all that covered, what about your Oracle database itself, the pot of gold hidden inside your network, and closely guarded.  Is it?  How safe is it from insiders?  Perhaps someone in the office wants to know the salaries of key people, or perhaps read and email or document about a potential merger or aquisition that may impact them.  There are many reasons you may want part or all of your database to be protected, even from certain internal folks. 

The best way to find out is to audit all of this.  Find out if the db has strong passwords, if data sent over the network is encrypted, and what if anything would or could be compromised, and how.

C. Trouble at the Interchange
(Middle-tier Problems)

Internet websites such as Amazon, Ebay, or your favorite bank as well as intranet applications (web based apps running in-house only) all use a three tier architecture today.  In simple terms the browser (Firefox, Internet Explorer, or Safari) running on your desktop or laptop is the client.  The middle tier is the webserver and/or application server such as Apache, JBoss, Websphere, and so on.  The backend piece is your database itself.  If your systems are sluggish it could very well point to problems in the middle tier or at the interchange.  If cars aren’t getting off at the right exit, you have problems.

These servers can often be replicated quite easily, and in fact that is what many companies like your favorite online bank or bookstore do.  They also add lots of memory, and fast harddrives for reading the data and returning it back to you.  All of these middle tier servers send their database requests to the same database on the backend. 

If you’re just starting to provision new servers for a project, and haven’t decided on platform (Windows, Linux, Solaris etc), memory, cpu and so on, be sure to run performance tests first.  If you’re choosing between Windows and Linux you’ll want to consider your IT expertise, but also seriously consider the performance and stability you can expect on each platform.

D. Alphabet Soup Tuning
(SQL Query Optimization)

Your developer, database administrator, or consultant just came up to you and told you that you are experiencing serious problems because of bad SQL.  What to do?  Sounds like something the doctor prescribed, and maybe you’re feeling like you’ll need to call one by the end of it.

Here’s some help.  It is difficult to find a page in a book without an index to what you’re searching for.  It would be difficult to find an address on Fifth Avenue if you don’t know what street it is near.  In fact, if they weren’t in order (which they are not in the database without sorting) you have to go through every address from Houston Street to 59th Street.  That’s going to take you a long time.  What’s more if someone comes later looking for the same or similar addresses you’ll have a hard time writing them down because you passed so many on your way up Fifth Avenue.  This is exactly what happens in a database when an index is missing.  Or when a developer asks the database for all the addresses in New York City, when they only wanted the first one, or one with the last name Smith. 

Essentially your Queries (SQL) are the key to these types of messes, and the key to cleaning them up as well.  Since your database is executing hundreds of requests from various middle-tier servers, you want it to return quickly, and efficiently only exactly what is necessary, and cache it for related queries in the future. 

This is very often a problem we identify in tuning applications, and database problems.  It is also one that often doesn’t show up in development, when you only have a couple of streets worth of addresses instead of the whole island of Manhattan!

E. We’ve Got A Leaning Tower of Pisa
(Architectural Problems)

When you’ve built a complex software system, sometimes problems and issues aren’t related to small fixes, those potholes in the road.  Sometimes issues are serious structural or architectural problems, which could not have been anticipated when the system was built.  Such a leaning tower can threaten to collapse, if you continue to patch, and provide short term fixes. 

What to do?  A system-wide assessment might be a good place to start.  This should definitely be by someone outside the organization who can provide you with feedback without threatening his or her own position.  Such an assessment can be like getting a second opinion from another doctor.  It can also identify serious bottlenecks possibly caused by platform decisions which though structural, can be remedied easier than rebuilding the tower.  It may be that such an assessment points to the need for a complete rewrite of the application, possibly porting it to another platform which is more scalable.  All of these conclusions though perhaps difficult to hear, are what you want to know as they are important for strategic decisions going forward.  You may hold back a launch, or upgrade to a new version until you’ve fixed the fundamental flaws.

Malcolm Gladwell, in his book “The Tipping Point” used this aphorism:  “People don’t change when you tell them there is a better option.  They change when they conclude that they have no other option.”  So make your own conclusions, but make them informed conclusions.

F. Have We Done A Firedrill?
Disaster Recovery, Backups etc

Disaster recovery has been on everyone’s mind since the September 11th tragedy, and even more so following hurricanes Katrina and Rita.  If your organization hasn’t run a firedrill, how can you be confident that you’re in good shape?  The answer is you can’t.

A firedrill much like the ones we all remember from elementary school, take an organization through EVERY STEP to recover the database, middle-tiers, and any other relevant systems from backups onto freshly setup servers.  If you don’t have enough beefy systems in-house to spare for such a firedrill, there are hosting services which can lease you usage of such servers on a short-term basis. 

In the process of this firedrill, you will learn some very important lessons.  First, you’ll learn if  you have all the pieces in place, or if you have to patch some things from the production systems to get it right.  That tells you if your backup is complete.  Second you will learn how long the process takes, and in running through it you document the process so you guarentee it’ll be faster in a real emergency.  Third and finally you will gain peace of mind because you can identify and fix the gaps before the impossible happens.

Oracle Open World – comments

We attended Oracle Open World this year, and have to say that it was one of the biggest in many years.  With all of Oracle’s recent acquisitions, many new vendors were there, and of course the media was there in attendance as well.

At our sister site Oracle and Open Source in a piece titled Restoring an Open World we discussed some of the announcements, and emphasis this year, on Open Source and Open Standards.  

All iHeavy Newsletter

Open Insights 11 – Google Wave or Tsunami

Heavyweight Internet Group Newsletter
Issue 11 – Google Wave or Tsunami?
September 1, 2005

by Sean Hull
<[email protected]>
Founder and Senior Consultant
Heavyweight Internet Group

Welcome to our free monthly newsletter, discussing news, developments and business best practices at the intersection of Oracle and Open Source software. 

Please forward to interested friends and colleagues.  Subscription information can be found at the end.

We are now writing material regularly for our sister site so check there for more frequent updates.

Google Wave or Tsunami?

With the recent announcement  of Google’s desktop messaging and VOIP client, the industry is full of analysis about how it will impact the playing field.  Some argue that it’s foray into VOIP will bring that much more attention to the amazing and destabalizing technology.  Some are also arguing interestingly that the new client could open up the instant messaging generally, which would be a big win for consumers.

Google’s new client Google Talk  is based on an Open Source application called Jabber  which is like a multi-lingual IM client, able to talk to AOL, ICQ, MSN and Yahoo, assuming you have  an account on all of those.  That they’ve embraced an Open platform, is further evidence of Google’s general trajectory of opening up areas of computing formerly held in rigid grip by commercial vendors, and proprietary protocols.

The discussion of Google Talk has also interestingly rekindled the rumors over a possible Google or Web OS.  Jason Kottke discusses this in depth as does Silicon Beat.  These discussions are no mere Slashdot scuttlebut, but ruminations by industry figures who likely have their finger on the Google pulse.

What would such an OS look like is anyone’s guess.  But if Tom Friedman has anything to say about it, the forces making the world a flatter playing field for business include work flow software, open protocols and Open Source software in general.   A flat playing field, like the internet, routes around monopolies, and finds ways to do business and computing cheaper, more efficiently and better.  

Open Source News

A comparison of TCO between Windows and Linux:

A Discussion of Open Source on Windows:

CTO of Athena Healthcare discusses open source integration

Success Story of Cendant Travel saves close to $100 million

A changing software landscape

Heavyweight Internet Group – Technical Articles

Tracking the Wily Proxy Hackers

Asterisk Calling Card Applications

MySQL Disaster Recovery

Dummy’s Guide to Linux Firewalls

Wireless Truth or Dare

Mirroring for the Impatient

Migrating a MySQL Database to Oracle 

Heavyweight Internet Group does Oracle integration with Open Source technologies, and has ten years experience in this space.  For more information visit our website at or call us toll free at 866 268-9448.


All iHeavy Newsletter

Open Insights 10 – Do You Arbitrage?

Heavyweight Internet Group Newsletter
Issue 10 – Do You Arbitrage?
August 1, 2005

by Sean Hull

<[email protected]>
Founder and Senior Consultant

Welcome to our free monthly newsletter, discussing news, developments and bus

iness best practices at the intersection of Oracle and Open Source software.

Subscriptions to this newsletter has been growing, we’re happy

to say, and we hope you all continue to forward on the email to anyone else you

think might benefit.  Subscription information can be found at the bottom.

We are now writing material regularly for our sister site so check there for more frequent updates.

Do You Arbitrage?

While traveling in Europe on business for the last couple of

weeks, I’ve been devouring Tom Friedman’s book The Lexus and the Olive Tree.  It’s subtitled “Understanding Globalization” but could have

just as well been “keeping up with the rush of technological change”.  Wha

tever your opinions on globalization & outsourcing, these forces of change,

along with disruptive technologies like the Internet and VOIP are flattening the

world and making it smaller and faster.  This both helps busines

s in providing a more efficient field on which to play, but also makes it more c

ompetitive as well.

In particular he discussed Arbitrage, that branch of finance about

profiting from disparities of information about markets.  Having

information or knowledge that some group of people or businesses don’t have isn

‘t enough, finding a way to put it to use helping people, that’s the tough part.

As technological change accelerates this becomes more and mo

re relevant in IT.  For example, companies are taking advantage of new tech

nologies such as Voice over IP to completely sidestep the traditional telephony

providers and saving a killing in the process.  Om Malik, the aut

hor of Broadbandits, discusses the rise of VOIP in this seminal blog entry

The Voice Over IP Insurrection.

Or take the example of Cendant Travel who has managed to save a bundle on an upgrage of their Orbitz and airline travel system by choosing not to go with a mainframe solution, but rather a 144 server Linux solution.
Of course these are two success stories, and just as with financial arbitrage, in

formation arbitrage can be risky.  Finding other businesses that can portray the trials and tribulations,  along with plenty of research, and testing

are crucial to avoiding a big waste of money.

Good consulting is all about this type of information arbitrage.  Knowledge of one area of IT is not enough.  You need experience with Operating Systems from Windows to Linux, MacOS to HP-UX.  You need a touch

of networking knowledge, and a good head about security.  And you need to have a sense of what troubles end users really struggle with.  On the other

hand you also need to juggle different disciplines, which gives you a very broa

d view of the business.  One day you are thinking about a proposal, and fra

ming and pricing that fits the clients budget.  The next day you are thinki

ng about technical problems and how to solve them within that structure you’ve p

ut together.  You’re thinking about promises, and timelines but also releva

nce and context.  There is always a bigger problem to  solve, always a

complex way to look at a problem.  The trick is often distilling a problem

down to it’s essence and focusing on that to bring out the solution as efficien

tly as possible.  And the truth is sometimes you make guesses.  Based

on all your years of experience, and diversity of encounters with problems at clients in various industries, you use your gut feeling to eliminate and focus on

the  relevant.  Of course there is plenty of science, and investigation, but the  efficiency is often a function of such intangibles.

Good consulting though is also about the story telling.  It is the non-technical side, conveying, distilling, and making analogies.  Information such as this helps business management make the right decisions based on budget, short and long-term expectations, and customer needs.

Consider all of these factors the next time you weigh and delibe

rate over outside resources for a project.  Although fixed fee projects sho

uld be fairly easy  to compare assuming you’ve spec’d out the bounds well,

hourly billing can be very misleading, a comparison of apples and oranges.

Also consider a wider net of  experience than just the particular need of

the moment.  Security vulnerabilities, or efficiencies of other components

of your infrastructure may be discovered, getting more for your money.

Technical Articles

Tracking the Wily Proxy Hackers

Asterisk Calling Card


MySQL Disaster Recovery

Dummy’s Guide to Linux Firewalls

Wireless Truth or Dare
Mirroring for the Impatient

Migrating a MySQL Database to Oracle

All iHeavy Newsletter

Open Insights 09 – IT Certifications

Heavyweight Internet Group Newsletter

Issue 09 – What About IT Certification?
July 2, 2005

by Sean Hull
<[email protected]>
Founder and Senior Consultant
Heavyweight Internet Group

Welcome to our free monthly newsletter, discussing news, developments and business best practices at the intersection of Oracle and Open Source software.

We are now writing material regularly for our sister site so check there for more frequent updates.

What About Certification?

From time to time the debate comes up again about certifications  in IT.  Are they important?  What do they indicate?  Should they be a deal breaker?  Which ones are relevant and which ones are less meaningful.  Depending on who you ask, you’ll get wildly  different answers.

Take for example a recent discussion over at Computerworld where Editor Don Tennant says he’s “Certifiably Concerned

Basically he covers a study in which it was found that folks WITH certifications were receiving smaller raises.  He found this very counterintuitive and managers may also.  But I’ll tell you with some confidence that most of your tech folks won’t be surprised.

There are a couple of different reasons for this.  The first one  sort of cuts to the heart of IT.  It is very much a moving target, that is to say that the set of skills, were they defined as a bunch of commands that work a certain way, are constantly in flux and changing.  Multiple choice tests especially focus on this type of wrote memorization of commands, not on problem  solving skills.  Due to the nature of technology being in constant flux, most good engineers don’t know all the commands.  It’s true ask one of your engineers the syntax for say doing an XOR with UNION in SQL.  The concepts are truely what is important, and that cannot be thoroughly tested in a certification exam.  It is more the job of a proper University education to teach that type of theory, and instill the problem solving skills to find the syntax and details when needed.

But we’re really hitting on a second point here, and that is that a Bachelor of Science degree from a good University is really the best certification you can get, besides real-world on-the-job  experience of course.  I’ve found in my experience that the ones most insistent on seeking out certifications are those who don’t  have Computer Science degrees in the first place but rather technical school training, and are looking to beef up the resume.  Barring perhaps the Cisco Certified Engineer which I understand is quite grueling with real-world problem solving with a real network, and real messes to cleanup, most certifications seem to be pretty  much a waste of time.

So where did this certification mania originate from then?  Well Law has the Bar Exam, Medical Doctors must go through residency, and get licensed to practice medicine, so why not apply the same rigors to the IT industry, albeit in more simplified form to provide the same sort of measures of skill and aptitude?  Good question, but as it turns out the IT industry is still too  disjointed, with no central authority governing it to provide  that type of certifications.  The best you have for such measures right now is the Bachelor of Science degree, hopefully from a  University which emphasizes engineering and has a good  reputation to go along with it.

And now you’re wondering, without certifications how do we find  good people?  Well I hope for one thing you take away from this that a certification might be misleading you in the first place on aptitude.  But really it means you have to do the legwork of finding the right people who have (a) some synergy with your company  (b) a degree from a good University (c) plenty of real-world  experience.  Run them by your best people to get their gut feel on the person, ask some tough questions, or how they might solve some problem in the enterprise. 

This type of selection process should go equally for full-timers as it does for the consultants you bring on board for your shorter term needs.  Don’t take the sales pitch, their parent company’s reputation, or other easy indicators at face value, and dig a  little deeper to find out if they’re worth your time, and money.

 Recent Technical Articles

Tracking the Wily Proxy Hackers

Asterisk Calling Card Applications

MySQL Disaster Recovery

Dummy’s Guide to Linux Firewalls

Wireless Truth or Dare

Mirroring for the Impatient

Migrating a MySQL Database to Oracle 

All iHeavy Newsletter

Open Insights 08 – The Devil Is In The Details

Heavyweight Internet Group Newsletter

Issue 08 – The Devil Is In The Details
June 1, 2005

by Sean Hull
<[email protected]>
Founder and Senior Consultant
Heavyweight Internet Group

Welcome to our free monthly newsletter, discussing news, developments and business best practices at the intersection of Oracle and Open Source software.

We are now writing material regularly for our sister site so check there for more frequent updates.

The Devil Is In The Details

Project planning has got to be one of the most difficult areas in computing. And yet often it is what is most overlooked. Popular insistence on hourly rates makes for easy comparisons of two resources, and allows management to avoid scope and specification, the exact opposite of what you want to do if you’re trying to watch costs. Also, without a fixed fee you cannot make an accurate ROI decision because you don’t know how long it will take to complete. Person A costs X, person B costs Y, the cheaper of the two is probably a better bet, even if they’re a little slower. Not only does this miss the impact of experience, which drives up rates naturally, but also an experienced technologist can make strategic and time-saving decisions up front if he knows what the project is comprised of. But if he is being directed this way and that he is more likely to waste time and therefore money on the wrong things.

Time and time again I’ve seen projects get started, with the most clear intentions in mind end up being a huge can of worms once they get underway. Take for example an upgrade we did for a large security firm in midtown Manhattan.

The project started out fairly straightforward. Migrate an existing Oracle 8i database running on a terribly slow single processor and single disk Sun server to a 4-processor Sun server with a 10 disk raid array. Since there was version compatability we had a good sense that (a) the application would continue to work the same way, (b) the optimizer would still work the same way, so performance of queries would be consistent, and related Operating System and backup scripts would continue to work as before with little change. Predictability is the key to scoping a project, which is of course key to coming up with a fixed cost at the outset.

As things unfolded, the management team decided that 9i was a key requirement, and that despite potential trouble along the way, the expected downtime was a sensible time to upgrade, that application functionality did not rely heavily on 8i features that might have changed, and cost-wise it would be better to do the two together. Nice on paper.

Of course a change like this completely eliminates the predictability for a project, quickly pushing us onto an hourly basis for work beyond a certain point. It is open ended because application changes are difficult to predict and changes in Oracle’s Cost Based Optimizer could impact performance as well.

After a period of almost 20 hours of downtime, and a lot of tired IT folks, we managed to get things running again. The biggest hurdle turned out to be getting the standby database working again as we encountered some Oracle bugs with the configuration, which were causing core dumps, and general panic as well.

All of this underlines the need for careful planning, testing, and then deployment. It’s like checking out track conditions and the route on the day of the race, it just makes good sense. Even if you have ten years experience running marathons as we do in Oracle IT, you still want to do your due diligence or the technology might bite back. Also try to lean towards well-scoped projects that are conducive to fixed fees, and avoid hasty comparisons of hourly rates between different resources.

Recent Technical Articles

Tracking the Wily Proxy Hackers

Asterisk Calling Card Applications

MySQL Disaster Recovery

Dummy’s Guide to Linux Firewalls

Wireless Truth or Dare

Mirroring for the Impatient

Migrating a MySQL Database to Oracle

All iHeavy Newsletter

Open Insights 07 – Open Source in the Enterprise

Open Source in the Enterprise

Heavyweight Internet Group is of course involved heavily in Oracle

database administration, setup, management and tuning. The other half

of our business involves Open Source integration, and development

using various technologies such as the LAMP platform, Linux, Apache,

MySQL and

PHP. We also make heavy use of Open Source technologies to run the

enterprise, and thought it might be illustrative to talk about that.

There are a whole host of technologies and applications we rely on

from day to day to run our business. When I actually sat down to

write this months newsletter, I was rather shocked at how long the

list was. Here’s a peek into what we use.

Sales + Marketing

For starters we use the spectacular SugarCRM (

{}) for sales and customer relationship

management. More than an addressbook, it manages opportunity lists,

open tasks, calls, notes, leads, and accounts. You can keep track of

prospects at every stage of the sales process from prospecting,

qualification, needs analysis, value proposition, and id-ing decision

makers to perception analysis, proposal, price quote, negotiation, and

deal closed. There is also a dashboard which displays excellent color

graphs of sales stage, opportunity size, and breakdowns by month and

lead source. For a business like ours which has run for years on

paper, and disjointed organizing methods, this application has had a

tremendous impact.


Invoicing we manage with an application called phpaga. Still under

development, it already offers a host of features such as HTML or PDF

invoices, quotes, varying billable rates by project, and by resource,

as well as various graph reports. Definitely worth a look.

( {})

Web Content Management

Next we use PostNuke ( {}) to

handle website content management automatically and easily. The

formating is not perfect, but it works quite well for our needs. We

have also started using Moveable Type (

{}) for managing our new weblog Oracle +

Open Source (

{}). All of these solutions use Apache

( {}) as a webserver, and

MySQL ( {}) as a database for basic

website needs. Though this tool is not strictly open source, it has

fairly loose restrictions on usage, and comes with source code, so we

include it here. Of course you need a browser to view all

this great content, and though we dabble with Galeon, Opera, and even

IE from time to time, we primarily use Firefox


{}) because it is fast, and

secure, saving all of our computing environment from the trouble of

spyware, adware, trojans, and other malware.

Desktop Publishing

Next we have started using OpenOffice (

{}) which provides fairly consistent

compatibility with Microsoft’s Office suite of tools. To be fair we

also occaisionally use the Windows versions too, as client comfort is

ultimately our goal. We’ve also begun using Scribus

( {}) for some

applications, which provides Quark-type document publishing, and can

generate nice PDF files as output.

Email, SPAM + List Management

Email lists we’ve just installed some new management software for

called phplist. ( {}) It provides

support for multiple lists, and easy subscribe/unsubscribe which we’ve

currently been doing manually! For email itself we use Postfix

( {}) Not email

solution would be complete if it couldn’t handle spam. For that we

use a whitelist solution called ASK (

{}) Not always perfect, it became an

absolute necessity when our junkmail started reaching into the

hundreds a day. Now we receive about one spam a week, from the odd

spammer that bothers to do an active reply.

Operating Systems

Of course no enterprise would be complete without an operating system

and we use Mandrake Linux 10.1 (

{}) in most cases, including the desktop (KDE

and GNOME) and on our mail + webservers. When we need to remotely

administer Windows machines, as is necessary from time to time, the

Open Source Cygwin tool comes in very handy. It provides a proper

command line interface, and openssh implementation for secure remote

logins. Most of the other standard Unix tools are there as well, such

as Emacs for editing.


If you think you’ve heard it all, think again. Our enterprise phone

PBX uses the superb Asterisk PBX and VOIP gateway solution

( {}). This allows integration

of traditional trunks from the phone company as well as VOIP lines

over our internet connection. It supports standard telephone

equipment, IP phones, and digital softphones all of which we use

actively. We also use Skype ( {}) to

coordinate with technologists, partners, and clients overseas.

Monitoring + Backups

No enterprise would be complete without monitoring and backup

solutions. For monitoring we use Nagios (

{}) for monitoring our own, and our client

systems, including web, mail, disk usage, load averages, Oracle

statistics and much more. It notifies us by email whenever there is a

problem. We manage our backups with rsync and rsnapshot

( {}) two great

tools that are fast and efficient whe combined with a secure shell

like OpenSSH ( {}). For

site statistics we have some integrated information which comes

directly from Postnuke mentioned above, but we also use Webalizer

which provides fancy graphs broken down by month and year for

detailed information directly from the webserver logfiles.


Lastly we use a few tools for project management. Since we coordinate

and manage the efforts of a number of developers we use CVS

( {}) for version

control. This allows everyone to be making changes to the code at the

same time without stepping on each others toes. We also have a trouble

ticket system called OTRS ( {}) which

helps us manage change requests, and bugs in these various

applications so the developers know what to work on and what has the

highest priority.

In an enterprise which handles 14 hosted domains, mostly for

non-profit organizations, as well as six active clients, we need to

be organized to remain efficient and effective. We use powerful

software to help us do that, and remain focused on pressing business