Heavyweight Internet Group Newsletter
Issue 12 – What the Geeks Mean
October 4, 2005
by Sean Hull
Founder and Senior Consultant
Heavyweight Internet Group
Welcome to our free monthly newsletter, discussing news, developments and business best practices at the intersection of Oracle and Open Source software.
Please forward to interested friends and colleagues. Subscription information can be found at the end.
We’d also like you to take a look at our sister site http://oracleopensource.com where you’ll find other non-technical and related news and content.
What the Geeks Mean
I’m blessed for having gotten started with computers when I was very young. Well that’s either blessed or cursed depending on how you look at it. At any rate, having worked with computers for about 25 years, I know well what the geeks are talking about. And having spent the last 10 years doing a lot of writing, and consulting for diverse businesses, I’ve learned to understand what the business folks are talking about. I’ve learned business thinking, and learned about the concerns of business owners. So this naturally leaves me in a comfortable position as International Interpreter for Geek to Suit communication.
Well I’m just kidding of course, but seriously there is often a wide gap in understanding and I make it a point of constantly trying to bridge that gap. So what I’m going to discuss this month are some of the big issues that come up on the technology end, and try to help you understand what your technology team is really talking about. This will help you make better purchasing and budget allocation decisions, as some problems impact the business more seriously, or more immediately than others. And don’t worry, we’re going to use a broad brush here, and not get too technical. Those technical folks in the audience, be patient where analogies don’t precisely match the underlying issues.
The six sections below basically amount to areas where your systems may be encountering problems. For each section I make a hopefully memorable and clear analogy, and then in parentheses I write the technical description of that problem that you may hear.
A. Our Disks Form a Single Lane Freeway
(Disk I/O Subsystem problems)
Disks, Disk Subsystems, or I/O Subsytems are basically the same thing. On a PC that sits on your desk, or at home there is a normally a single harddrive which stores things while the computer is turned off. When the system is booted, or a program is started, programs and data are loaded off that storage media into memory. In server computing environments much the same thing happens, albeit there are many more parts. It is often referred to as a storage subsystem because it may be composed of a stack of harddrives all managed together in one cabinet, and interfaced together in what is called a RAID. Why should I be concerned as a manager you ask? Well how these are configured (ie what RAID level) can dramatically impact the performance, and reliability of your SAN (Storage Area Network), effectively giving you a one lane highway in one direction (while writing to disk) while giving you a 5 lane highway in the other direction. What’s more some configurations are more prone to crashing, and loss of a harddrive (crashing, and loss of a lane in our car analogy) than others.
If your developers and system administrators are saying what they have is incorrect this may be because the particular server in question has a controller card which does not support the proper RAID type. In that case new hardware would solve your problem. External SAN/NAS solutions such as EMC or NetApp normally support all the various RAID types in whatever configuration and combinations you want.
Keep in mind that you want to keep the traffic flowing in both directions of your highway, and you want to keep the car crashes down. Of course also want to get the most performance especially if you invested a lot on that expensive and fast car, Oracle. Above all avoid the single lane freeway and its attendant accidents.
B. Our Network Is a Party Line
(Network + Database Security)
Some of us may recall back in the old days when you might have a party line running into your home for telephone calls. When you picked up the telephone you might hear your next door neighbor on the telephone. You would have to politely (or not so politely) ask them when they would be finished so you could make your call. Beyond the inconvenience, the privacy concerns are obvious.
In the technology world, much more of our communications are like a party line, or a postcard that you send through the mail, then you might imagine. But how much? And when and where should I be most concerned?
If you have a dedicated networking team, who manage the firewall, and keep patches up to date, you still may have to be concerned about wireless access to your network. Believe it or not even encrypted WiFi connections can be breached within 10 to 15 minutes with the right tools. You have no WiFi you say? You only use a VPN to allow folks on the road to connect? Well there can be vulnerabilities in all of that software, ones that can allow a hacker to breach the network, and steal valuable data.
But even if you have all that covered, what about your Oracle database itself, the pot of gold hidden inside your network, and closely guarded. Is it? How safe is it from insiders? Perhaps someone in the office wants to know the salaries of key people, or perhaps read and email or document about a potential merger or aquisition that may impact them. There are many reasons you may want part or all of your database to be protected, even from certain internal folks.
The best way to find out is to audit all of this. Find out if the db has strong passwords, if data sent over the network is encrypted, and what if anything would or could be compromised, and how.
C. Trouble at the Interchange
Internet websites such as Amazon, Ebay, or your favorite bank as well as intranet applications (web based apps running in-house only) all use a three tier architecture today. In simple terms the browser (Firefox, Internet Explorer, or Safari) running on your desktop or laptop is the client. The middle tier is the webserver and/or application server such as Apache, JBoss, Websphere, and so on. The backend piece is your database itself. If your systems are sluggish it could very well point to problems in the middle tier or at the interchange. If cars aren’t getting off at the right exit, you have problems.
These servers can often be replicated quite easily, and in fact that is what many companies like your favorite online bank or bookstore do. They also add lots of memory, and fast harddrives for reading the data and returning it back to you. All of these middle tier servers send their database requests to the same database on the backend.
If you’re just starting to provision new servers for a project, and haven’t decided on platform (Windows, Linux, Solaris etc), memory, cpu and so on, be sure to run performance tests first. If you’re choosing between Windows and Linux you’ll want to consider your IT expertise, but also seriously consider the performance and stability you can expect on each platform.
D. Alphabet Soup Tuning
(SQL Query Optimization)
Your developer, database administrator, or consultant just came up to you and told you that you are experiencing serious problems because of bad SQL. What to do? Sounds like something the doctor prescribed, and maybe you’re feeling like you’ll need to call one by the end of it.
Here’s some help. It is difficult to find a page in a book without an index to what you’re searching for. It would be difficult to find an address on Fifth Avenue if you don’t know what street it is near. In fact, if they weren’t in order (which they are not in the database without sorting) you have to go through every address from Houston Street to 59th Street. That’s going to take you a long time. What’s more if someone comes later looking for the same or similar addresses you’ll have a hard time writing them down because you passed so many on your way up Fifth Avenue. This is exactly what happens in a database when an index is missing. Or when a developer asks the database for all the addresses in New York City, when they only wanted the first one, or one with the last name Smith.
Essentially your Queries (SQL) are the key to these types of messes, and the key to cleaning them up as well. Since your database is executing hundreds of requests from various middle-tier servers, you want it to return quickly, and efficiently only exactly what is necessary, and cache it for related queries in the future.
This is very often a problem we identify in tuning applications, and database problems. It is also one that often doesn’t show up in development, when you only have a couple of streets worth of addresses instead of the whole island of Manhattan!
E. We’ve Got A Leaning Tower of Pisa
When you’ve built a complex software system, sometimes problems and issues aren’t related to small fixes, those potholes in the road. Sometimes issues are serious structural or architectural problems, which could not have been anticipated when the system was built. Such a leaning tower can threaten to collapse, if you continue to patch, and provide short term fixes.
What to do? A system-wide assessment might be a good place to start. This should definitely be by someone outside the organization who can provide you with feedback without threatening his or her own position. Such an assessment can be like getting a second opinion from another doctor. It can also identify serious bottlenecks possibly caused by platform decisions which though structural, can be remedied easier than rebuilding the tower. It may be that such an assessment points to the need for a complete rewrite of the application, possibly porting it to another platform which is more scalable. All of these conclusions though perhaps difficult to hear, are what you want to know as they are important for strategic decisions going forward. You may hold back a launch, or upgrade to a new version until you’ve fixed the fundamental flaws.
Malcolm Gladwell, in his book “The Tipping Point” used this aphorism: “People don’t change when you tell them there is a better option. They change when they conclude that they have no other option.” So make your own conclusions, but make them informed conclusions.
F. Have We Done A Firedrill?
Disaster Recovery, Backups etc
Disaster recovery has been on everyone’s mind since the September 11th tragedy, and even more so following hurricanes Katrina and Rita. If your organization hasn’t run a firedrill, how can you be confident that you’re in good shape? The answer is you can’t.
A firedrill much like the ones we all remember from elementary school, take an organization through EVERY STEP to recover the database, middle-tiers, and any other relevant systems from backups onto freshly setup servers. If you don’t have enough beefy systems in-house to spare for such a firedrill, there are hosting services which can lease you usage of such servers on a short-term basis.
In the process of this firedrill, you will learn some very important lessons. First, you’ll learn if you have all the pieces in place, or if you have to patch some things from the production systems to get it right. That tells you if your backup is complete. Second you will learn how long the process takes, and in running through it you document the process so you guarentee it’ll be faster in a real emergency. Third and finally you will gain peace of mind because you can identify and fix the gaps before the impossible happens.
Oracle Open World – comments
We attended Oracle Open World this year, and have to say that it was one of the biggest in many years. With all of Oracle’s recent acquisitions, many new vendors were there, and of course the media was there in attendance as well.
At our sister site Oracle and Open Source in a piece titled Restoring an Open World we discussed some of the announcements, and emphasis this year, on Open Source and Open Standards.