5 Superb blogs this week

Why wait for the new year to start something new? I come across a lot of great new blogs, while digging through the interwebs. So I thought I’d start a regular column to feature the best ones. We’ll including gems from web 2.0 industry, startups, business & management, and of course some technical devops & cloud computing ones.

Join 11,000 others and follow Sean Hull on twitter @hullsean.

1. Todd Hoff’s High Scalability

Todd Hoff’s High Scalability has been around for years, and offers up a cup of espresso for your infrastructure daily. From important topics like why you should avoid ORMs (Object Relational Modelers see post on technical debt) to regular scalability around the web posts to keep you on track.

He also features great articles under the title “real life architectures” from heavyweights such as facebook, twitter & youtube. These are the gold nuggets that are indispensable to devops and startups.

Read This: 5 Reasons Devops Should Blog

2. Albert Wenger’s Continuations

I was tipped off to Continuations using the Disqus commenting system’s discovery features. Click through to the community tab on say Fred Wilson’s AVC blog and you can find top commenters and where they blog at.

Wenger’s posts include such gems as Anatomy of a URL, giving a lay audience a little insight into the ubiquitous web paths and Computing Building Blocks which dissects the internet stack for everyone. As a partner at Union Square Ventures he’s obviously looped in with the big boys, but his writing style is so great he offers a model for technical bloggers everywhere.

Check out: A CTO Must Never Do This

3. Andrew Chen

Let’s face it Andrew Chen is the rock star I want to be! He’s got tons of organic followers on twitter, and reading his blog & newsletter it’s no surprise. He’s bright, and always provides Nate Silver style insights & new perspectives.

What is a minimal homepage, and how will it help me increase signups? Why can’t I seem to find a technical co-founder? What’s a minimum desirable product? You’ll see why Dave MacClure & Mitch Kapor work with him.

Read: AirBNB Didn’t Have to Fail – AWS Outage Postmortem

4. John Paul Aguiar

John’s website may appear a bit busy at first, but that’s just because it is so chock full of useful content. He offers very hands on, down in the trenches advice for bloggers & entrepreneurs. 150k followers on twitter, and articles that get retweeted hundreds of times, means he’s done the A/B testing, and learned to write clearly, and has great insights to share.

One thing he does is a weekly piece on entrepreneurs & users to follow on twitter. That great feature inspired this very post, not least because it offers a steady stream of things to write about, but because I was also featured there recently. I feel like I’ve hit the big time, thanks John!

Related: How to Hire a Developer That Doesn’t Suck

5. Krebs on Security

Brian Krebs is a bad boy. According to Bruce Schneier he apparently pissed someone off so bad, they had illegal substances sent to him through the mail in attempt to frame him.

Clearly his security research and writing is not appreciated by everyone. That said take a look at his website. You’d be shocked to learn what an ATM skimmer is, or what is the value of a hacked PC. Phishing, bots, email spam, gaming & reputation hijacking are just a few of the criminal activities that go on.

Also: The Myth of Five Nines

Get some in your inbox: Exclusive monthly Scalable Startups. We share tips and special content. Here’s a sample

Understand the High Cost of Technical Debt by Ward Cunningham (Video)

A week or two ago, I got into a conversation on Twitter about technical debt, and someone shared this superb video by Ward Cunningham (youtube). Here is Ward’s Interview website.

Join 10,000 others and follow Sean Hull on twitter @hullsean.

Waterfall, agile, developer or operations, devops, managers, CTOs… everyone should watch this video, for it cuts to the heart of the challenges we face doing modern software development, in a fast paced and always changing environment.

Though it’s not mentioned in the video, I would argue using an ORM is an expensive form of debt, one that is improperly calculated by teams, managers & startups, and one that bites hard into future scalability. Stay tuned for a post about that!

Enough said, here’s the video.

Here’s the transcript of much of the dialog. Appologies for any typos…

1. Metaphors & Thinking

I became interested in the way metaphors influence the way we think

after reading George Lakoff & Mark Johnson’s Metaphors We Live By (Amazon).

An important idea is that we reason by analogy through the metaphors that have entered our language.

Related: 5 Things Toxic to Scalability.

2. Coining Debt Metaphor

I coined the debt metaphor to explain the refactoring we were doing on a product.

This was an early product done in smalltalk. It was important to me that we accumulate the learnings we did about the application by modifying the program to look as if we had known what we were doing all along. And to look as if it had been easy to do in smalltalk.

The explanation I gave to my boss, and this was financial software, was a financial analogy I called the debt metaphor and that said that:

[quote]
If we failed to make our program align with what we then understood to be the proper way to think about our fin objects, then we were going to continue to stumble on that disagreement which is like paying interest on a loan. -Ward Cunningham
[/quote]

Also: AirBNB didn’t have to fail when Amazon went down.

3. Need for Speed

With borrowed money you can do something sooner than you might otherwise, but until you pay back that money you will pay interest.

I thought borrowing money was a good idea. I thought that rushing software out the door to get some experience with it was a good idea. But that of course you would eventually go back and as you learned things about that software you would repay that loan by refactoring the program to reflect your experience as you acquired it.

Read this: Why generalists are better at scaling the web.

4. Understand the Burden

I think that there were plenty of cases where people would rush things soft out the door & learn things, but never put that learning back into the program. That by analogy was borrowing money thinking you never had to pay it back. of course If you do that you know say with your credit card, evemtually all your income goes to interest & your purchasing power goes to zero.

By the same token if you dev a program for a long period of time, by only adding features, and never reorganizing it to reflect your understanding of those features, then eventually that program does not contain any understanding and all efforts to work on it take longer and longer. In other words the interest is total. You’ll make zero progress.

Check out: How to hire a developer that doesn’t suck.

5. Achieving Agility

a lot of bloggers at least have explained the debt metaphor and confused it i think with the idea that you could write code poorly with the intention of doing a good job later. and thinking that that was the pirmary source of deb.t I’m never in favor of writing code poorly. But I am in favor of writing a code to reflect your current understanding of a problem even if your understanding is partial.

If you want to be able to go into debt that way by dev soft you odn’t completely understand, you’re wise to make that software reflect your understanding as best you can. So that when it does come tiem to refactor it’s clear what your thinking was when you wrote it. and making it easier to refactor it to what your thinking is now.

in other words the whole debt metaphor or lets say the ability to pay back debt, and make the debt metaphor work for your advantage depends upon you writing code that is clean enough to be able to refactor as you come to understand your problem.

i think that’s a good methodology it’s at the heart of extreme programming. the debt metaphor is an explanation, one of many explanations as to why extreme programming works.

Read our popular interview guides for candidates, managers & recruiters: MySQL DBAAWS Expert (Amazon Cloud) and Cloud Deployment & Automation Engineer.

Don’t forget, hiring is a numbers game

Get some in your inbox: Exclusive monthly Scalable Startups. We share tips and special content. Here’s a sample

5 Reasons Devops Should Blog

Join 9500 others and follow Sean Hull on twitter @hullsean.

1. Stand up and be heard

Years ago I was sitting on an online forum chatting with an Oracle buddy of mine. This was circa 1998. We were working on an open source tool to interface with Oracle. There were all these libraries, for PHP & Perl, and a lot of developers starting to build tools. We hatched this hair brained idea to write a book about all of this, and pitched it to O’Reilly. They loved it and thus was born the book Oracle & Open Source in 2001.

Writing a book was, is and always will be a lot of work. It was a great learning experience too. Editors critique your writing, and this teaches you to speak to a broader audience, clarify your statements, and including illustrations and stories.

Along with this came opportunities to speak at conferences, user groups and meetups. It’s exhilerating, and professionally challenging, and I enjoyed all of it.

Blogging allows you to do all of the above in a more measured way. Write regularly, get your ideas out there, get feedback, rinse & repeat. What’s more over time you’ll build up a library of material some of which will draw solid, strong, repeat traffic. It may be what you are most passionate about, what ideas you’ve ironed out smoothly, or what material is most missing from the we already. Whatever the reason your analytics will show you the way.

Read this: Database expert interview questions for candidates, recruiters & managers

2. Share your lessons

In professional services engagements, you learn something new everyday. After a few years you’ll have some battle scars & war stories too. For example I used to have a strong distrust of sales. But through real lessons in the feast or famine world of running your own business, you learn some survival instincts. And knowing how to sell your services & expertise is an art all to itself.

Also: A CTO Must Never Do This

[quote]
Getting up and taking a stand isn’t easy. You’ll receive criticism, and likely feel professionally vulnerable at first. But that only makes us stronger engineers, willing to listen, and better communicators.
[/quote]

3. Get opinionated

Taking a stand on controversial topics, is it something you want to do? Is it something you can do confidently, but also being open to criticism, and seeing all sides.

It’s challenging, but in that process it will either open you to new ideas, or make your resolve stronger. And that process is great for your professional development.

Also check out: AirBNB didn’t have to fail – don’t go down with Amazon

4. Withstand a sh*tstorm

Audiences keep you honest. If I were to go out on a limb I’d say technically brilliant, engineering audiences even more so.

I remember a post I did a year ago, which referenced a feature based on a wrong software version. In other words that feature would not work based on my article. The readers tore me to pieces in the comments.

But listen closely now, I’m saying that’s a good thing. Yes criticism is a very good thing indeed. Get enough of it, and you’ll learn to weed out the folks who are just trolls, from the ones with genuine suggestions. And all that makes you stronger!

Learn to listen a bit, and that makes you an even better sysadmin or devop.

Related: 5 conversational ways to evaluate great consultants

5. Learn by doing

Developers, ops, DBAs and Big Data jockeys alike are doers. We sit and code, build & configure components, troubleshoot & tune. Writing is descriptive and often it’s difficult for us to step back and describe what we’re doing.

By writing we carefully sift through our own thought processes to break it down for novices, or a broader audience. This is a learning process for us too. It’s therapeutic. But also it hones our message and makes us better teachers. We literally learn by doing.

Also: The Myth of Five Nines

Get some in your inbox: Exclusive monthly Scalable Startups. We share tips and special content. Here’s a sample

30 Days of Aspirin for your Ailing Management Headaches

Take Two & Call Me In the Morning

I like a book like Gerry Czarnecki’s even if I can’t pronounce his name. He’s laid out things for readers, a very busy bunch who are going to have trouble finding time in their day to read, but can sorely use the advice. Organized into 30 daily bites of probably 15-20 minutes, you can dig through on your commute to work, or on your lunch break.

Also check out Who Moved My Cheese a business self-help classic.

Czarnecki should know. As a 2nd Army Lieutenant then later heading up board of directors at organizations large and small he’s seen a lot. He digs up the best stories, and offers up lessons straight out of his experiences.

[quote]
On interviews – Be careful that you do not let the resume dominate your conversation. You will gain better insight if you listen to what the candidates want to talk about or what they think you want to talk about. – Gerry Czarnecki
[/quote]

Some of the topics you’ll touch on…

o dealing with mistaken hires that don’t work out
o finding a mentor
o career moves & pivots
o setting expectations with new hires
o hitting peak performance
o filling your bus with rock stars

For startups an all-time favorite of mine is REWORK – 37 Signals guys. These guys have taken the lean methodology and built a real business by being efficient. The pages resonate for me as a small business owner. I’ve found many of the same lessons work in the real world for me.

[mytweetlinks]

Also read
Thank you for arguing by Jay Heinrichs
. With some of the best advice on rhetoric, sales, presentations & persuasion, I re-read bits of the book often.

Get some in your inbox: Exclusive monthly Scalable Startups. We share tips and special content. Here’s a sample

3 Things Devops Can Learn from Aviation

I recent went on a flight with a pilot friend & two others. With only 3 passengers, you get up front and center to all the action. Besides getting to chat directly with the pilot, you also see all of the many checks & balances in place. They call it the warrior aviation checklist. It inspired this post on disaster recovery in the datacenter.

Join 8000 others and follow Sean Hull on twitter @hullsean.

1. Have a real plan B

Looking over the document you can see there are procedures for engine failure in flight! That’s right those small planes can have an engine failure and they glide. So for starters the technology is designed for failure. But the technology design is not the end of the story.

When you have procedures and processes in place, and training to deal with failure, you’re expecting and planning for it.

Also check out 5 Conversational Ways to Evaluate Great Consultants.

[quote]
Expect & plan for failure. Use technologies designed to fail gracefully and build software to do so. Then put procedures in place to detect & alert you so you can put those processes into place quickly.
[/quote]

2. Use checklists

Checklists are an important part of good process. We use them for code deploys. We can use them for disaster recovery too. But how to create them?

Firedrills! That’s right, run through the entire process of disaster recovery with your team & document. As you do so, you’ll be creating the master checklist for real disaster recovery. Keep it in a safe place, or better multiple places so you’ll have it when you really need it.

Read this Scalability Tips & Greatest Hits.

3. Trust your instruments

Modern infrastructures include 10′s or 100′s of servers. With cloud instances so simple to spinup, we add new services and servers daily. What to do?

You obviously need to use the machines to monitor the machines. Have a good monitoring system like Nagios, and metrics collection such as New Relic to help you stay ahead of failure.

Setup the right monitoring automation, and then trust the instruments!

Related Real disaster recovery lessons from Sandy.

Get some in your inbox: Exclusive monthly Scalable Startups. We share tips and special content. Here’s a sample

5 conversational ways to evaluate great consultants

Startups and more mature businesses alike, and those large and small, at some point will need to hire a consultant or two. Want to get the best bang for your buck? Ask some tough questions!

Join 8000 others and follow Sean Hull on twitter @hullsean.

1. Make sure they’re not going to quit

I’ve heard so many crash and burn stories over the years, it makes my head spin.

One client had hired a consultant who was supposed to be the best in NYC. After only a few weeks of working with the client, he explained that they were “doing it all wrong”. Furthermore he had a travel schedule to meet, with speaking engagements in South America.

So he basically dropped them! As the client retold this story to me, I wonder if they could hear my head shaking, I was stunned. Who would turn away from a customer in pain? And furthermore turn away from one who could clearly use the expertise of someone who had seen a lot before!?

Keep in mind the reasons why people leave consulting.

2. Be sure they have some war stories to tell

Any consultant who’s been in business for a while, surely has some good war stories to tell. Talk about those, and find out the battles they’ve been in.

I can tell a few myself. In one case I was only 12 hours from leaving for summer vacation when a long time colleague and former client called me. They were in a serious emergency. The big boys, the remote DBAs that many in the industry use, had broken their database. Replication was misconfigured, and they were running blind. I ended up on a SKYPE call fixing the database and troubleshooting problems on Virgin’s inflight wifi. You don’t forget that kind of firefighting.

[quote]
Be sure they won’t quit, ask about war stories, test for some push back and be sure they empathize with your business pain. But more importantly ask them to tell their own business story. You’ll learn a lot.
[/quote]

For another client, back in the dot-com days, their application was stalling completely. Customers were leaving, and so was an 80 million dollar buyout deal. Nothing a few Oracle parameters couldn’t fix!.

And there are always a few tales of woe between sales teams, and the engineers that then must deploy solutions in the real world. Beware the sales wolf in sheeps clothing and do your homework aka due diligence on technical solutions.

3. Ensure they can provide resistance & push back

Good consultants have to walk a tightrope all the time. On the one hand they are tasked with making their clients happy. At the end of the day, improving their position, business bottom line, yes these are crucial. Sure that means saying yes, that means trying to be a problem solver as much as possible.

But always saying yes, avoids hard truths that you have to share. I had one client whose primary Oracle dba went on vacation. Before he left we reviewed systems. Multi-master replication in Oracle is brittle by nature. We both agreed. And I agreed not to change the configurations lest it break other things down the line. Not one week into his vacation a mandate comes back from on high, this change absolutely has to happen now. There are millions of dollars at stake!

Applying strong resistance is necessary to avoid breaking something even bigger. And it was not easy to stand strong in the face of such pressure. But I assured the team that such changes would mean even bigger problems for the company.

4. Find out if they empathize with your pain

In one of the biggest ironic twists, consultants should understand the pain of building a business. Because they themselves have experienced when clients don’t pay so they understand cash flow problems themselves. That is what every small business struggles with, and most startups too!

5. Ask them how they built their business

For me, one of the least appreciated things about independent consulting is, that in the most important ways, it is about running a small business. So someone who has built and kept running a freelance or consulting business knows how to make hard decisions, and keep their eyes on the ball.

A consultant needs to know how to get business first and foremost. But then how to manage engagements carefully. Once you’ve got those two down, keep building your business incrementally.

Someone who has successfully run a real business for years can share the story of what they’ve done. What has worked, what hasn’t worked, how they have pivoted when necessary, how they have failed fast, and moved through it.

They can tell you how they stay cash flow positive, can deliver on time, can be likeable and communicate with teams, and really understand every side of a business.

Get some in your inbox: Exclusive monthly Scalable Startups. We share tips and special content. Here’s a sample

Scalability Tips & Greatest Hits

autoscaling MySQL

Join 8000 others and follow Sean Hull on twitter @hullsean.

In the past two years we’ve written a ton of material on scalability. Here’s the greatest hits…

Why Generalists Are Better at Scaling the Web

The internet stack is a complex infrastructure of interlocking components. An scalability engineer must be adept at Linux, plus webservers, caching servers, search servers, automation services, and relational databases on the backend. We think a generalist with a broad base of experience is most suited to the job of scalability engineer.

5 Things Toxic to Scalability

ORMs should keep you up at night, but so should coupled and locking processes, a single copy of your database, missing metrics and no deployment feature flags.

5 More Things Deadly to Scalability

A followup to the original, we touch on Disk I/O, RAID, queuing in the database (a no-no), full-text searching, insufficient or missing caching and lastly the dreaded technical debt.

Scalability Happiness

A Zen monk might ask what is the sound of one hand clapping? That’s the sound your servers will be making when you apply this one simple principal.

5 Ways to Boost MySQL Scalability

Deploying MySQL as your web-facing database? Here are a few key tips to boost speed & performance.

3 Ways To Boost Cloud Scalability

Building your startup in the Amazon Web Services cloud? There are 3 things you absolutely must do.

Why Your Cloud Is Speeding for a Scalability Cliff

The cloud may seem like the obvious place to build new applications & infrastructure, but there is a precipice hidden from sight…

Get some in your inbox: Exclusive monthly Scalable Startups. We share tips and special content. Here’s a sample

3 Simple Patterns for Tighter MySQL Code

Join 8000 others and follow Sean Hull on twitter @hullsean.

SQL is derided by many and for good reason. It’s key to scalability yet terribly difficult to write good code.

Here’s a few quick tips to write tighter queries in MySQL

1. Get rid of those Subqueries!

Subqueries are a standard part of SQL, unfortunately MySQL doesn’t handle them very well. Luckily there’s a sweet rewrite that can put you in the fast lane. Here’s how to speedup a MySQL subquery by rewriting as a join.

Note that another compelling reason to upgrade to MySQL 5.6 is that this tweak has been rolled into the optimizer. Hoorah!

Also: 5 Things Deadly to Scalability.

2. Repair those UNIONs

If your code uses the UNION construct in SQL, there are a few different ways to tune those queries. You can use UNION ALL or pushdown conditions can help you optimize UNION in MySQL.

Read this: MySQL DBA Hiring Guide for candidates, managers & recruiters

4. Better PAGING through datasets

Does your web application display pages of users, pages of orders or pages of items? If you’re using
LIMIT and OFFSET there are 3 good ways to optimize these in MySQL.

Check out: Scalability Happiness – A Quiet Query Log

Get some in your inbox: Exclusive monthly Scalable Startups. We share tips and special content. Here’s a sample

Scalability Happiness – A Quiet Query Log

Peter Van Allen - Pin Drop

Join 7500 others and follow Sean Hull on twitter @hullsean.

There’s a lot of talk on the web about scalability. Making web applications scale is not easy. The modern web architecture has so many moving parts. How can we grapple with the underlying problem?

Also: Why Are MySQL DBAs So Hard to Find?

The LAMP stack scales well

The truth that is half right. True there are a lot of moving parts, and a lot to setup. The internet stack made up of Linux, Apache, MySQL & PHP. LAMP as it’s called, was built to be resilient, dynamic, and scalable. It’s essentially why Amazon works. Why what they’re doing is possible. Windows & .NET for example don’t scale well. Strange to see Oracle mating with them, but I digress…

[quote]
Linux and LAMP that is built on top of it, are highly scalable and dynamic to begin with.
[/quote]

Also: AirBNB Didn’t Have to Fail During an AWS Outage

Ok, so what’s this got to do with MySQL? Well a LOT.

The webserver tier, the caching layers like memcache & varnish, as well as the search tier solr. These all scale fairly easily because their assets are fixed. Or almost so.

The database tier is different. So what affects performance of a database server? Server size? Main memory? Disk speed? The truth is all of those. But

Also check out: The Sexiest New Feature of AWS Speeds Up EBS

After you setup the server – set memory settings and so forth, it’s a fairly fixed object. True there are parameters to tweak but on the whole there isn’t a ton of day-to-day tuning to do.

Well if that’s true, why does performance take a hit?! As applications grow, the db server slows down, don’t we need to tweak server settings? Do we need new hardware?

Read this: A CTO Must Never Do This

The answer is possibly, but 9 times out of 10 what really needs to happen is queries must be tuned.

[quote]
In 17 years of consulting that is the single largest cause of scalability problems. Fix those queries and your problems are over.
[/quote]

The Elephant in the Room – Query Tuning

I was talking with a colleague today at AppNexus. He said, so should we do some of that work inside the application, instead of doing a huge UNION or a large JOIN? I said yes you can move work onto the application, but it makes the application more complex. On the flip side the webserver tier is easier to scale. So there are tradeoffs.

I said this:

[quote]
By and large, if scalability is our goal, we should work to quiet the activity in the slow query log. This is an active project for developers & DBAs. Keep it quiet and your server will run well.
[/quote]

Also: Top MySQL DBA Interview Questions for Candidates, Hiring Managers & Recruiters

Yet I still talk to teams where this is mysterious. It’s unclear. There’s no conviction there. And that’s where I think DBAs are failing. Because this is our subject matter expertise, and if we haven’t convinced developer teams of this, we’re not working together enough. API teams aren’t separate from DBA and operations. Siloing technology departments is a killer…

[mytweetlinks]

As you roll out new code, if some queries show up, then those need attention. Tweak the code until the queries drop out. This is the primary project of scalability.

When should I think about upgrading hardware?

If your code is stable, but you’re seeing a steady line rising on load average of the server, *THEN* go up in hardware. Load average means cpu & disk are being taxed. The server can’t keep up.

Related: Should I use RDS or build a MySQL server on AWS?

Devops means work together!

I close with a final point. Devops means bring dev & ops together! Don’t silo them off in different wings. Communicate. DBAs it’s your job to educate Developers about scalability and help with query tuning. Devs, profile new SQL code, test with large datasets & for god sakes don’t use an ORM – it’s one of 5 things toxic to scalability. Run explain and be sure to index all the right columns.

Together we can tackle this scalability thing!

Get some in your inbox: Exclusive monthly Scalable Startups. We share tips and special content. Here’s a sample

Tech Print Isn't Dead – Bloomberg Businessweek

Join 7100 others and follow Sean Hull on twitter @hullsean.

I usually review a book. But this time I thought I’d trumpet about the best weekly going right now. Nope it’s not a blog, not Wired or any of the other tech mag.

It’s the upstart Bloomberg Businessweek which got a major overhaul a couple years back by
creative director Josh Tyrangiel formerly of the Guardian.

Check out: Why Generalists Are Better at Scaling the Web

The covers are controversial and grab your attention, and the copy is top notch. It’s like reading the economist but with more tech & business heavy content. Work lifestyle articles, back-to-back with business book reviews as haiku? A solid dose of business & tech trends, startups, entrepreneurs and innovation. It’s all here.

Also: What Wouldn’t Google Do?.

If you’re looking for a good read, get a subscription right now.

Get some in your inbox: Exclusive monthly Scalable Startups. We share tips and special content. Here’s a sample