Where’s my 80 million dollars?

Way back in the heydays of the dot-com boom, the year is 1999.

Join 12,100 others and follow Sean Hull on twitter @hullsean.

I worked for a medium size internet startup called Method Five. When I came on board they were having a terrible time with their site performance.

Website crashing

When I first met the team, I was tasked with performance problems. After all their flagship web property kept crashing, and it didn’t look good to investors. As with most web properties in those days it was a home-grown datacenter in the back of the office, running on Sun Microsystems hardware, with Oracle on the backend and Apache serving webpages.

Also: Why a killer title can make or break your content efforts

Negotiating an acquisition

As it became clearer after day one, the project was particularly sensitive. They were negotiating a huge acquisition by a firm called Xceed Corp. The sticking point? Their crashing website did not sell their technology prowess in a particularly positive light. To say the least!

Read: Why high availability is so very hard to deliver

Investigation

As it turns out the site had all the right players, from systems administrators to a DBA who sat watch over the Oracle systems.

As I dug into the systems, I found a serious smoking gun. It seems the Oracle software was configured to use just 5M of memory out of about 256M free. Just like MySQL, the server must be configured to use available memory upon startup. There are myriad caches and buffers which need to be attended to. By today’s standards these numbers probably sound absurd. Nevertheless the DBA wasn’t familiar with the basic memory settings, and so the system was terribly bottlenecked.

Read this: Why a four letter word divides dev and ops

Problem Solved

We then ordered some urgent changes to the system, configuring all of Oracle’s caches to use up the precious memory available.

Immediate the website unlocks, transactions begin flowing, and webpages are returning quickly. End users pull their noggins off their keyboards, and the executives begin breathing a sigh of relief. The site was literally 1000x faster during peak.

Related: MySQL interview guide for managers and candidates alike

Acquisition

Shortly thereafter the acquisition goes through for a cool 5 million in cash and 80 million in stock.

Where’s my cut?! You might be asking that question. But my policy is almost always defer to something concrete and tangible, aka fees and real compensation. I did not negotiate any stock in the deal.

Another popular war story we wrote A CTO Must Never Do This….

Read: Why devops talent is in short supply

Lesson’s Learned

o Don’t believe received wisdom. Check and double check what’s really happening.
o Use the memory and resources you have available.
o Measure capacity, and isolate bottlenecks in the system
o Decouple services wherever possible
o Problems are as often people and process as they are with technology

Also: 5 more things deadly to scalability

Make it this far? Grab our newsletter!