Way back in the heydays of the dot-com boom, the year is 1999.
Join 12,100 others and follow Sean Hull on twitter @hullsean.
I worked for a medium size internet startup called Method Five. When I came on board they were having a terrible time with their site performance.
Website crashing
When I first met the team, I was tasked with performance problems. After all their flagship web property kept crashing, and it didn’t look good to investors. As with most web properties in those days it was a home-grown datacenter in the back of the office, running on Sun Microsystems hardware, with Oracle on the backend and Apache serving webpages.
Also: Why a killer title can make or break your content efforts
Negotiating an acquisition
As it became clearer after day one, the project was particularly sensitive. They were negotiating a huge acquisition by a firm called Xceed Corp. The sticking point? Their crashing website did not sell their technology prowess in a particularly positive light. To say the least!
Read: Why high availability is so very hard to deliver
Investigation
As it turns out the site had all the right players, from systems administrators to a DBA who sat watch over the Oracle systems.
As I dug into the systems, I found a serious smoking gun. It seems the Oracle software was configured to use just 5M of memory out of about 256M free. Just like MySQL, the server must be configured to use available memory upon startup. There are myriad caches and buffers which need to be attended to. By today’s standards these numbers probably sound absurd. Nevertheless the DBA wasn’t familiar with the basic memory settings, and so the system was terribly bottlenecked.
Read this: Why a four letter word divides dev and ops
Problem Solved
We then ordered some urgent changes to the system, configuring all of Oracle’s caches to use up the precious memory available.
Immediate the website unlocks, transactions begin flowing, and webpages are returning quickly. End users pull their noggins off their keyboards, and the executives begin breathing a sigh of relief. The site was literally 1000x faster during peak.
Related: MySQL interview guide for managers and candidates alike
Acquisition
Shortly thereafter the acquisition goes through for a cool 5 million in cash and 80 million in stock.
Where’s my cut?! You might be asking that question. But my policy is almost always defer to something concrete and tangible, aka fees and real compensation. I did not negotiate any stock in the deal.
Another popular war story we wrote A CTO Must Never Do This….
Read: Why devops talent is in short supply
Lesson’s Learned
o Don’t believe received wisdom. Check and double check what’s really happening.
o Use the memory and resources you have available.
o Measure capacity, and isolate bottlenecks in the system
o Decouple services wherever possible
o Problems are as often people and process as they are with technology
Also: 5 more things deadly to scalability
Make it this far? Grab our newsletter!