30 Days of Aspirin for your Ailing Management Headaches

Take Two & Call Me In the Morning

I like a book like Gerry Czarnecki’s even if I can’t pronounce his name. He’s laid out things for readers, a very busy bunch who are going to have trouble finding time in their day to read, but can sorely use the advice. Organized into 30 daily bites of probably 15-20 minutes, you can dig through on your commute to work, or on your lunch break.

Also check out Who Moved My Cheese a business self-help classic.

Czarnecki should know. As a 2nd Army Lieutenant then later heading up board of directors at organizations large and small he’s seen a lot. He digs up the best stories, and offers up lessons straight out of his experiences.

[quote]
On interviews – Be careful that you do not let the resume dominate your conversation. You will gain better insight if you listen to what the candidates want to talk about or what they think you want to talk about. – Gerry Czarnecki
[/quote]

Some of the topics you’ll touch on…

o dealing with mistaken hires that don’t work out
o finding a mentor
o career moves & pivots
o setting expectations with new hires
o hitting peak performance
o filling your bus with rock stars

For startups an all-time favorite of mine is REWORK – 37 Signals guys. These guys have taken the lean methodology and built a real business by being efficient. The pages resonate for me as a small business owner. I’ve found many of the same lessons work in the real world for me.

[mytweetlinks]

Also read
Thank you for arguing by Jay Heinrichs
. With some of the best advice on rhetoric, sales, presentations & persuasion, I re-read bits of the book often.

Get some in your inbox: Exclusive monthly Scalable Startups. We share tips and special content. Here’s a sample

3 Things Devops Can Learn from Aviation

I recent went on a flight with a pilot friend & two others. With only 3 passengers, you get up front and center to all the action. Besides getting to chat directly with the pilot, you also see all of the many checks & balances in place. They call it the warrior aviation checklist. It inspired this post on disaster recovery in the datacenter.

Join 8000 others and follow Sean Hull on twitter @hullsean.

1. Have a real plan B

Looking over the document you can see there are procedures for engine failure in flight! That’s right those small planes can have an engine failure and they glide. So for starters the technology is designed for failure. But the technology design is not the end of the story.

When you have procedures and processes in place, and training to deal with failure, you’re expecting and planning for it.

Also check out 5 Conversational Ways to Evaluate Great Consultants.

[quote]
Expect & plan for failure. Use technologies designed to fail gracefully and build software to do so. Then put procedures in place to detect & alert you so you can put those processes into place quickly.
[/quote]

2. Use checklists

Checklists are an important part of good process. We use them for code deploys. We can use them for disaster recovery too. But how to create them?

Firedrills! That’s right, run through the entire process of disaster recovery with your team & document. As you do so, you’ll be creating the master checklist for real disaster recovery. Keep it in a safe place, or better multiple places so you’ll have it when you really need it.

Read this Scalability Tips & Greatest Hits.

3. Trust your instruments

Modern infrastructures include 10’s or 100’s of servers. With cloud instances so simple to spinup, we add new services and servers daily. What to do?

You obviously need to use the machines to monitor the machines. Have a good monitoring system like Nagios, and metrics collection such as New Relic to help you stay ahead of failure.

Setup the right monitoring automation, and then trust the instruments!

Related Real disaster recovery lessons from Sandy.

Get some in your inbox: Exclusive monthly Scalable Startups. We share tips and special content. Here’s a sample

5 conversational ways to evaluate great consultants

Startups and more mature businesses alike, and those large and small, at some point will need to hire a consultant or two. Want to get the best bang for your buck? Ask some tough questions!

Join 8000 others and follow Sean Hull on twitter @hullsean.

1. Make sure they’re not going to quit

I’ve heard so many crash and burn stories over the years, it makes my head spin.

One client had hired a consultant who was supposed to be the best in NYC. After only a few weeks of working with the client, he explained that they were “doing it all wrong”. Furthermore he had a travel schedule to meet, with speaking engagements in South America.

So he basically dropped them! As the client retold this story to me, I wonder if they could hear my head shaking, I was stunned. Who would turn away from a customer in pain? And furthermore turn away from one who could clearly use the expertise of someone who had seen a lot before!?

Keep in mind the reasons why people leave consulting.

2. Be sure they have some war stories to tell

Any consultant who’s been in business for a while, surely has some good war stories to tell. Talk about those, and find out the battles they’ve been in.

I can tell a few myself. In one case I was only 12 hours from leaving for summer vacation when a long time colleague and former client called me. They were in a serious emergency. The big boys, the remote DBAs that many in the industry use, had broken their database. Replication was misconfigured, and they were running blind. I ended up on a SKYPE call fixing the database and troubleshooting problems on Virgin’s inflight wifi. You don’t forget that kind of firefighting.

[quote]
Be sure they won’t quit, ask about war stories, test for some push back and be sure they empathize with your business pain. But more importantly ask them to tell their own business story. You’ll learn a lot.
[/quote]

For another client, back in the dot-com days, their application was stalling completely. Customers were leaving, and so was an 80 million dollar buyout deal. Nothing a few Oracle parameters couldn’t fix!.

And there are always a few tales of woe between sales teams, and the engineers that then must deploy solutions in the real world. Beware the sales wolf in sheeps clothing and do your homework aka due diligence on technical solutions.

3. Ensure they can provide resistance & push back

Good consultants have to walk a tightrope all the time. On the one hand they are tasked with making their clients happy. At the end of the day, improving their position, business bottom line, yes these are crucial. Sure that means saying yes, that means trying to be a problem solver as much as possible.

But always saying yes, avoids hard truths that you have to share. I had one client whose primary Oracle dba went on vacation. Before he left we reviewed systems. Multi-master replication in Oracle is brittle by nature. We both agreed. And I agreed not to change the configurations lest it break other things down the line. Not one week into his vacation a mandate comes back from on high, this change absolutely has to happen now. There are millions of dollars at stake!

Applying strong resistance is necessary to avoid breaking something even bigger. And it was not easy to stand strong in the face of such pressure. But I assured the team that such changes would mean even bigger problems for the company.

4. Find out if they empathize with your pain

In one of the biggest ironic twists, consultants should understand the pain of building a business. Because they themselves have experienced when clients don’t pay so they understand cash flow problems themselves. That is what every small business struggles with, and most startups too!

5. Ask them how they built their business

For me, one of the least appreciated things about independent consulting is, that in the most important ways, it is about running a small business. So someone who has built and kept running a freelance or consulting business knows how to make hard decisions, and keep their eyes on the ball.

A consultant needs to know how to get business first and foremost. But then how to manage engagements carefully. Once you’ve got those two down, keep building your business incrementally.

Someone who has successfully run a real business for years can share the story of what they’ve done. What has worked, what hasn’t worked, how they have pivoted when necessary, how they have failed fast, and moved through it.

They can tell you how they stay cash flow positive, can deliver on time, can be likeable and communicate with teams, and really understand every side of a business.

Get some in your inbox: Exclusive monthly Scalable Startups. We share tips and special content. Here’s a sample

Scalability Tips & Greatest Hits

autoscaling MySQL

Join 8000 others and follow Sean Hull on twitter @hullsean.

In the past two years we’ve written a ton of material on scalability. Here’s the greatest hits…

Why Generalists Are Better at Scaling the Web

The internet stack is a complex infrastructure of interlocking components. An scalability engineer must be adept at Linux, plus webservers, caching servers, search servers, automation services, and relational databases on the backend. We think a generalist with a broad base of experience is most suited to the job of scalability engineer.

5 Things Toxic to Scalability

ORMs should keep you up at night, but so should coupled and locking processes, a single copy of your database, missing metrics and no deployment feature flags.

5 More Things Deadly to Scalability

A followup to the original, we touch on Disk I/O, RAID, queuing in the database (a no-no), full-text searching, insufficient or missing caching and lastly the dreaded technical debt.

Scalability Happiness

A Zen monk might ask what is the sound of one hand clapping? That’s the sound your servers will be making when you apply this one simple principal.

5 Ways to Boost MySQL Scalability

Deploying MySQL as your web-facing database? Here are a few key tips to boost speed & performance.

3 Ways To Boost Cloud Scalability

Building your startup in the Amazon Web Services cloud? There are 3 things you absolutely must do.

Why Your Cloud Is Speeding for a Scalability Cliff

The cloud may seem like the obvious place to build new applications & infrastructure, but there is a precipice hidden from sight…

Get some in your inbox: Exclusive monthly Scalable Startups. We share tips and special content. Here’s a sample

3 Simple Patterns for Tighter MySQL Code

Join 8000 others and follow Sean Hull on twitter @hullsean.

SQL is derided by many and for good reason. It’s key to scalability yet terribly difficult to write good code.

Here’s a few quick tips to write tighter queries in MySQL

1. Get rid of those Subqueries!

Subqueries are a standard part of SQL, unfortunately MySQL doesn’t handle them very well. Luckily there’s a sweet rewrite that can put you in the fast lane. Here’s how to speedup a MySQL subquery by rewriting as a join.

Note that another compelling reason to upgrade to MySQL 5.6 is that this tweak has been rolled into the optimizer. Hoorah!

Also: 5 Things Deadly to Scalability.

2. Repair those UNIONs

If your code uses the UNION construct in SQL, there are a few different ways to tune those queries. You can use UNION ALL or pushdown conditions can help you optimize UNION in MySQL.

Read this: MySQL DBA Hiring Guide for candidates, managers & recruiters

4. Better PAGING through datasets

Does your web application display pages of users, pages of orders or pages of items? If you’re using
LIMIT and OFFSET there are 3 good ways to optimize these in MySQL.

Check out: Scalability Happiness – A Quiet Query Log

Get some in your inbox: Exclusive monthly Scalable Startups. We share tips and special content. Here’s a sample

Scalability Happiness – A Quiet Query Log

Peter Van Allen - Pin Drop

Join 7500 others and follow Sean Hull on twitter @hullsean.

There’s a lot of talk on the web about scalability. Making web applications scale is not easy. The modern web architecture has so many moving parts. How can we grapple with the underlying problem?

Also: Why Are MySQL DBAs So Hard to Find?

The LAMP stack scales well

The truth that is half right. True there are a lot of moving parts, and a lot to setup. The internet stack made up of Linux, Apache, MySQL & PHP. LAMP as it’s called, was built to be resilient, dynamic, and scalable. It’s essentially why Amazon works. Why what they’re doing is possible. Windows & .NET for example don’t scale well. Strange to see Oracle mating with them, but I digress…

[quote]
Linux and LAMP that is built on top of it, are highly scalable and dynamic to begin with.
[/quote]

Also: AirBNB Didn’t Have to Fail During an AWS Outage

Ok, so what’s this got to do with MySQL? Well a LOT.

The webserver tier, the caching layers like memcache & varnish, as well as the search tier solr. These all scale fairly easily because their assets are fixed. Or almost so.

The database tier is different. So what affects performance of a database server? Server size? Main memory? Disk speed? The truth is all of those. But

Also check out: The Sexiest New Feature of AWS Speeds Up EBS

After you setup the server – set memory settings and so forth, it’s a fairly fixed object. True there are parameters to tweak but on the whole there isn’t a ton of day-to-day tuning to do.

Well if that’s true, why does performance take a hit?! As applications grow, the db server slows down, don’t we need to tweak server settings? Do we need new hardware?

Read this: A CTO Must Never Do This

The answer is possibly, but 9 times out of 10 what really needs to happen is queries must be tuned.

[quote]
In 17 years of consulting that is the single largest cause of scalability problems. Fix those queries and your problems are over.
[/quote]

The Elephant in the Room – Query Tuning

I was talking with a colleague today at AppNexus. He said, so should we do some of that work inside the application, instead of doing a huge UNION or a large JOIN? I said yes you can move work onto the application, but it makes the application more complex. On the flip side the webserver tier is easier to scale. So there are tradeoffs.

I said this:

[quote]
By and large, if scalability is our goal, we should work to quiet the activity in the slow query log. This is an active project for developers & DBAs. Keep it quiet and your server will run well.
[/quote]

Also: Top MySQL DBA Interview Questions for Candidates, Hiring Managers & Recruiters

Yet I still talk to teams where this is mysterious. It’s unclear. There’s no conviction there. And that’s where I think DBAs are failing. Because this is our subject matter expertise, and if we haven’t convinced developer teams of this, we’re not working together enough. API teams aren’t separate from DBA and operations. Siloing technology departments is a killer…

[mytweetlinks]

As you roll out new code, if some queries show up, then those need attention. Tweak the code until the queries drop out. This is the primary project of scalability.

When should I think about upgrading hardware?

If your code is stable, but you’re seeing a steady line rising on load average of the server, *THEN* go up in hardware. Load average means cpu & disk are being taxed. The server can’t keep up.

Related: Should I use RDS or build a MySQL server on AWS?

Devops means work together!

I close with a final point. Devops means bring dev & ops together! Don’t silo them off in different wings. Communicate. DBAs it’s your job to educate Developers about scalability and help with query tuning. Devs, profile new SQL code, test with large datasets & for god sakes don’t use an ORM – it’s one of 5 things toxic to scalability. Run explain and be sure to index all the right columns.

Together we can tackle this scalability thing!

Get some in your inbox: Exclusive monthly Scalable Startups. We share tips and special content. Here’s a sample

Tech Print Isn't Dead – Bloomberg Businessweek

Join 7100 others and follow Sean Hull on twitter @hullsean.

I usually review a book. But this time I thought I’d trumpet about the best weekly going right now. Nope it’s not a blog, not Wired or any of the other tech mag.

It’s the upstart Bloomberg Businessweek which got a major overhaul a couple years back by
creative director Josh Tyrangiel formerly of the Guardian.

Check out: Why Generalists Are Better at Scaling the Web

The covers are controversial and grab your attention, and the copy is top notch. It’s like reading the economist but with more tech & business heavy content. Work lifestyle articles, back-to-back with business book reviews as haiku? A solid dose of business & tech trends, startups, entrepreneurs and innovation. It’s all here.

Also: What Wouldn’t Google Do?.

If you’re looking for a good read, get a subscription right now.

Get some in your inbox: Exclusive monthly Scalable Startups. We share tips and special content. Here’s a sample