Tag Archives: apache

Scalability Happiness – A Quiet Query Log

Peter Van Allen - Pin Drop

Join 7500 others and follow Sean Hull on twitter @hullsean.

There’s a lot of talk on the web about scalability. Making web applications scale is not easy. The modern web architecture has so many moving parts. How can we grapple with the underlying problem?

Also: Why Are MySQL DBAs So Hard to Find?

The LAMP stack scales well

The truth that is half right. True there are a lot of moving parts, and a lot to setup. The internet stack made up of Linux, Apache, MySQL & PHP. LAMP as it’s called, was built to be resilient, dynamic, and scalable. It’s essentially why Amazon works. Why what they’re doing is possible. Windows & .NET for example don’t scale well. Strange to see Oracle mating with them, but I digress…

[quote]
Linux and LAMP that is built on top of it, are highly scalable and dynamic to begin with.
[/quote]

Also: AirBNB Didn’t Have to Fail During an AWS Outage

Ok, so what’s this got to do with MySQL? Well a LOT.

The webserver tier, the caching layers like memcache & varnish, as well as the search tier solr. These all scale fairly easily because their assets are fixed. Or almost so.

The database tier is different. So what affects performance of a database server? Server size? Main memory? Disk speed? The truth is all of those. But

Also check out: The Sexiest New Feature of AWS Speeds Up EBS

After you setup the server – set memory settings and so forth, it’s a fairly fixed object. True there are parameters to tweak but on the whole there isn’t a ton of day-to-day tuning to do.

Well if that’s true, why does performance take a hit?! As applications grow, the db server slows down, don’t we need to tweak server settings? Do we need new hardware?

Read this: A CTO Must Never Do This

The answer is possibly, but 9 times out of 10 what really needs to happen is queries must be tuned.

[quote]
In 17 years of consulting that is the single largest cause of scalability problems. Fix those queries and your problems are over.
[/quote]

The Elephant in the Room – Query Tuning

I was talking with a colleague today at AppNexus. He said, so should we do some of that work inside the application, instead of doing a huge UNION or a large JOIN? I said yes you can move work onto the application, but it makes the application more complex. On the flip side the webserver tier is easier to scale. So there are tradeoffs.

I said this:

[quote]
By and large, if scalability is our goal, we should work to quiet the activity in the slow query log. This is an active project for developers & DBAs. Keep it quiet and your server will run well.
[/quote]

Also: Top MySQL DBA Interview Questions for Candidates, Hiring Managers & Recruiters

Yet I still talk to teams where this is mysterious. It’s unclear. There’s no conviction there. And that’s where I think DBAs are failing. Because this is our subject matter expertise, and if we haven’t convinced developer teams of this, we’re not working together enough. API teams aren’t separate from DBA and operations. Siloing technology departments is a killer…

[mytweetlinks]

As you roll out new code, if some queries show up, then those need attention. Tweak the code until the queries drop out. This is the primary project of scalability.

When should I think about upgrading hardware?

If your code is stable, but you’re seeing a steady line rising on load average of the server, *THEN* go up in hardware. Load average means cpu & disk are being taxed. The server can’t keep up.

Related: Should I use RDS or build a MySQL server on AWS?

Devops means work together!

I close with a final point. Devops means bring dev & ops together! Don’t silo them off in different wings. Communicate. DBAs it’s your job to educate Developers about scalability and help with query tuning. Devs, profile new SQL code, test with large datasets & for god sakes don’t use an ORM – it’s one of 5 things toxic to scalability. Run explain and be sure to index all the right columns.

Together we can tackle this scalability thing!

Get some in your inbox: Exclusive monthly Scalable Startups. We share tips and special content. Here’s a sample

Caching – What is it and why is it important?

Caching keeps frequently accessed objects, images and data closer to where you need them, speeding up access to websites you hit often.

Your browser is the first layer of caching, keeping images and data from websites that you visit often.  Next the webserver itself has a caching layer, typically implemented by something like memcache, caching information that it would normally fetch from the database on the backend.  This avoids the network roundtrip, and also avoids the load and work of running the query to fetch that data again.

Furthermore you can install what’s called a reverse-proxy on the webserver, such as Varnish.  This can bring further speedups and performance benefits to your overall architecture.

On the database server you also do a lot of caching.  With MySQL you may configure the query cache, which caches query result sets inside of MySQL, eliminating the need to rerun those queries on subsequent calls.  And further the database server has various other caches such as the InnoDB buffer cache, to keep blocks of data in memory, reducing slower requests from disk.

On Quora, Sean Hull asks: What is caching and why is it important?

Backups – What are they and why are they important?

Backups are obviously a crucial component in any enterprise application.  Modern internet components are prone to failure, and backups keep your bases covered.  Here’s what you should consider:

  1. Is your database backed up, including object structures, data, stored procedures, grants, and logins?
  2. Is your webserver doc-root backed up?
  3. Is your application source code in version control and backed up?
  4. Are your server configurations backed up?  Relevant config files might include those for apache, mysql, memcache, php, email (postfix or qmail), tomcat, Java solr or any other software your application requires.
  5. Are your cron or supporting scripts and jobs backed up?
  6. Have you tested all of these components and your overall documentation with a fire drill?  This is the proof that you’ve really covered all the angles.

If you do your backups right, you should be able to restore without a problem.

Sean Hull asks on Quora – What are backups and why are they important?

Success Story–Media and Entertainment Conglomerate

The Business

A website aggregating twitter feeds for celebrities, with sophisticated search functionality.

The Problem

Having been recently acquired by a large media and entertainment conglomerate, their traffic had already tripled.  What’s more they expected their unique pageviews to grow by 20 to 30 times in the coming six months.

Our Process

We worked closely with the lead architect and designer of the site to understand some of the technical difficulties they were encountering.  We discussed key areas of the site, and where performance was most lacking.

Next we reviewed the underlying infrastructure with an eye for misconfigurations, misuse of or badly allocated resources, and general configuration best practices.  They used Amazon EC2 cloud hosted servers for the database, webserver, and other components of the application.

The Solution

Our first round of reviews spanned a couple of days.  We found many issues with the configuration which could dramatically affect performance.  We adjusted settings in both the webserver, and the database to optimally maximize the platform upon which they were hosted.  These initial changes reduced the load average on the server from a steady level of 10.0 to an average of 2.0.

Our second round of review involved a serious look at the application.  We worked closely with the developer to understand what the application was doing.  We identified those areas of the application causing the heaviest footprint on the server, and worked with the developer to tune those specific areas.  In addition we examined the underlying database structures, tables and looked for relevant indexes, adding those as necessary to support the specific requirements of the application.

After this second round of changes, tweaks, adjustments, and rearchitecting, the load average on the server was reduced dramatically, to a mere 0.10.  The overall affect was dramatic.  With 100 times reduction in the load on the server, the websites performance was snappy, and very responsive.  The end user experience was noticeably changed.  A smile comes on your face when you visit your favorite site, to find it working fast and furious!

Results

The results to the business were dramatic.  Not only were their short term troubles addressed, as the site was handling the new traffic without a hick up.  What’s more they had the confidence and peace of mind now to go forward with new advertising campaigns, secure in the knowledge that the site really could perform, and handle a 20 to 30 times increase in traffic with ease.