31 Essential Blogs for Startups & Scalability

So many blogs, so little time! Here’s our list of the best we’ve found. Currently our favorite reader is Pulse pictured left. Starting to play around with flipboard too.

Nuts & Bolts Technical

Slashdot
One of the original tech blogs, that still covers lots of breaking news, and difficult topics. Very technical, with probing commentary. Beware the actual comments though, as they’re often full of immature and childish rants.

Planet Mysql
An aggregator of many MySQL blogs, it hits on topics from benchmarking, and advanced tuning, to new technologies on the horizon. Drupal and LAMP topics are often also covered.

mysql performance blog
Percona’s technical blog never disappoints. There are endless posts about a myriad of topics related to deploying, tuning and optimizing MySQL and all it’s variants.

Hacker News
You may not like Paul Graham, he’s easy not to. But his YCombinator News site is an awesome collection of always surprising technical topics that are sure to keep you busy.

Netflix Tech Blog
You might have read about Chaos Monkey before, that

Our very own Scalable Startups
You’re already reading us regularly of course! Why not grab our newsletter?

Programmable Web
Mashups & APIs. What more do you want? Very cool stuff here.

Also take a look at our best of compilation.

Business & Economics

HBR Blog
If you’ve ever read Harvard Business Review, you know how in depth and on point the material is. More thorough discussions than many other blogs, and excellent discussions in the comments.

Marginal Revolution
Tyler Cowen’s endlessly interesting and provocative take on the world through the eyes of economics. Like using science to analyze and solve the worlds ails, this blog always has a reasoned take on things.

NPR Planet Money
I’ve been listening to this podcast religiously since the financial crisis of 2008. It continues to intrigue and educate me in ways that college finance never did. You’ll learn a lot.

Bloomberg Businessweek
BBW despite it’s name is like Wired back in the 90’s before it got taken over by Conde, and the cutting edge writers and risk takers left. That’s right this magazine is full of analysis, creativity, and color. It’s what you’re looking for in a print magazine. One of my favorites.

Inc. Magazine
Real articles for your small business needs today. Thoughtful and topical.

Forbes Magazine
Banking, finance, politics, news.

You might also check out our Scalable Startups newsletter archives.

Venture

A VC
Fred Wilson’s iconic blog is always on the cusp, with a thoughtful and participating audience of readers.

Infochachkie
John Greathouse is a VC with a very readable blog on startups and investing.

Chris Dixon
This guy invested in tons of great startups that are household names now. With a very readable blog to match, he’s a man with ideas that we all benefit from.

Springwise
As they call it, your “Essential Fix of Entrepreneurial Ideas”.

Feld Thoughts
Brad Feld is another big VC with an excellent blog on topics relevant to Venture Capital & Startups.

Social

Andrew Chen
Consumer internet, metrics, and user growth. Brilliant idea guy. I learn from this guy’s blog everytime I check it.

Problogger
The smarties behind the book of the same name, this is essential reading for bloggers who wanna make a dent in the world.

Blog Tyrant
How to build successful blogs that make real money. Learn from Ramsay Taplin who’s done it already. Whether your blog sells products, widgets or services, there’s stuff for you here.

Kissmetrics Marketing Blog
Very good stuff on marketing, twitter, facebook and all the other good social topics.

Mixergy Blog
Business tips & startup advice with a bent towards marketing and social.

Mark Schaefer Marketing
Mark’s the brains behind the great book Return on Influence which we reviewed. His Businesses Grow blog is full of helpful ideas and insights.

Figaro Speech
You may have read my review of Word Hero and seen the earlier review of Thank You For Arguing. His blog is a real gem, extending on the wonders and lessons of word hero, you’ll be writing witty and memorable one-liners and titles that will go viral tomorrow!

Industry

Gigaom
Om Malik started out writing about the bandwidth boom and bust of the 2000’s. His blog has grown wildly to cover the industry as a whole, and contrary to the stuff you get on business insider, this is quality journalism.

AllThingsD
Another industry site with a great selection of journalists writing on the internet & startup industries.

ReadWriteWeb
Another excellent industry blog with slightly overlapping coverage to gigaom and allthingsd, but worth scanning each of them for different perspectives.

Venturebeat
Possibly a bit more venture and investment oriented than the others, but still mainly an industry coverage blog site.

Entrepreneur
Slightly more focus on business, and entrepreneurs, but also internet & startup industry topics.

Adweek
Trying to broaden my horizons by adding this one into the mix. Some very interesting topics, and plenty of overlap with internet industry and startups.

If you read this far, grab our newsletter!

5 Things You Overlooked with MySQL Dumps

1. Point In Time Recovery

If you’ve never done point in time recovery, it’s time to take a second look. With a standard mysqldump you restore your database to the time when the backup happened. Only do them once a day, then you can lose as much as 24 hours of data.

Enter point-in-time recovery, and you have the option to restore all those transactions that occured since your backup last night. Those changes, INSERTs, UPDATEs, DELETEs & even ALTER TABLEs are all stored in the binary logs. That’s right, all those statements are preserved for you!

The tool to fetch those statements with is called mysqlbinlog. When you use it, you’ll need to specify the binlog file & offset to start at. Uh oh, you don’t have those?

That’s where the mysqldump option –master-data comes in. We recommend using –master-data=2 option. That includes this important nuggets of information in the dump file as a comment.

Now you’ll know where to start applying transactions. Simply dig this information out of the dump file.

What I like to do is dump the statements into a text file first, then apply them.

Here’s how you fetch all the transactions since your dump:

$ mysqlbinlog --offset=12345 /var/lib/mysql/binlog.000005 > my_point_in_time_data.mysql

Here’s how you would then apply them:

$ mysql < my_point_in_time_data.mysql

Also checkout: Replication setup with no downtime - using hotbackups!

2. Locking & Consistency

MySQL can be confusing when it comes to concurrency and locking. Do we need to lock tables during a backup, yes, no? Sometimes? The confusion is because MyISAM tables are non-transactional. So sometimes locking all tables is required to get a consistent dump.

--lock-all-tables

This option is only required if your application works on tables across schemas. So if any one application hits multiple schemas and there are MyISAM tables in both, you'll need to use this option. It's equivalent to FLUSH TABLES WITH REACH LOCK.

--lock-tables

This option locks all tables in one schema, performs backup, then moves on to the next schema. You only need it if you have MyISAM tables. It's on by default and with --opt flag.

--single-transaction

This is the least invasive way to do a dump of your data. However it only works if all your tables are InnoDB. Keep in mind it won't error out if you have MyISAM tables, but your backup will be inconsistent. The plus side is that from your applications perspective there is really no blocking locks happening. Very cool indeed!

For a more general guide see Accidental DBAs Guide to MySQL.

3. Backups of stored code

If you have stored procedures, triggers or functions, you're not capturing them in your dumps by default. Really! Use the --routines option to include them. Better yet do a dump with just your stored code. This line will do so, and add a date timestamp for you. Be sure to have the [client] or [mysql] section in your .my.cnf file with user & password specified. Then you don't need them on command line.

$ mysqldump --no-data --no-create-info --all-databases --routines > db_code_`/bin/date +%m%d%Y`.mysql

Are you verifying replication slaves? You should know they regularly drift out of sync without throwing errors. Read more to verify MySQL replication..

4. Separate schema & data dumps

Sometimes it's useful to have a schema only backup. That is a dump of the objects in your database (CREATE statements) without any of the data. Why you ask? For starters you could use the unix "diff" command between a schema dump today, and one you did last month. That's a cool use case, but there are others. Suppose you want to build a test server, without all the production data or some subset of it? Again, a schema only dump is your friend.

o Use unix diff to compare objects in two different databases
o Audit schema changes over time
o Build test servers without all the production data
o Parallel loading of data
- use multiple mysql processes to import data
- break up the data only portion of your backup

Here's the syntax to get the schema only dump:

$ mysqldump --no-data --all-databases > db_schema_`/bin/date +%m%d%Y`.mysql

Here's how you get the data only dump:

$ mysqldump --no-create-info --all-databases --complete-insert --skip-quote-names > db_data__`/bin/date +%m%d%Y`.mysql

Looking for a MySQL DBA position? Check our our interview questions guide.

5. A few more options

--complete-insert

By default mysqldump includes INSERT statements built to *follow* create statements. So the assume all the columns match. If you break up your schema & data you could potentially have a mismatch. This option makes sure the INSERT statemens included in the dump specify column names. So if they don't match, you'll get a proper error message.

--skip-quote-names

MySQL's default dump settings will quote everything under the sun. That's to protect if you use certain reserved words or worse if you have objects with spaces in the names. But of course best practices advice against *both* of those in your database, so be advised. This option will make your dump much more readable and easier to edit.

--hex-blob

We had a customer who was using the cross-platform media publishing system K4. When we moved data from one server to another, we had a terrible time at first. That's because this enterprise application stores data in MySQL blobs. Unfortunately the default mysqldump options didn't account for certain special characters which were getting munged. The way around that is to use this option. If you have blobs, you really need this.

Read this far? Grab our scalable startups newsletter.

Beware the sales wolf in sheep suits

Recently a colleague called me up to get my opinion.

[quote]We’re in the process of standardizing our systems on Red Hat Linux, but management and higher ups are convinced we should deploy Oracle on Oracle’s own Linux distribution. Which is better?[/quote]

Therein lies the eternal drama in organizations, the push & pull between dollars and technology best practices.

We had a similar experience with a MySQL deployment, and solution framed by Oracle sales.

Battle lines are drawn

Clearly the battle lines are drawn now. Between director of operations & team versus management & business stakeholders, between high level and the trenches, or between the systems that support your business and day-to-day running of them.

Business units & management are tasked with budgets, cost management, and long term thinking about trajectory and what’s best for the business. Operations teams are tasked with the day-to-day stability, the command line perspective.

What is the sales team’s position?

Sales guys at Oracle have a job to sell licenses. This isn’t good or bad, it’s their driver. Understanding all the drivers will help us align them.

Sales guys sell to management, so they will likely frame all their stories to management concerns. Also Oracle’s history here is fairly clear. Get customers locked into Oracle up and down the stack, and they become more and more beholden to you as their primary provider. As customers become more dependent, they will begin to squeeze more and more out of them.

Nothing personal, this is how money is made. But understand the goal.

How do OS choices affect the business bottom line?

Standardizing across the enterprise reduces costs & reduces operational complexity. This can reduce risks to operator error & other downtime that increase with more heterogeneous environment.

On the Oracle distribution side, you likely have tweaks to make Oracle run better. However don’t forget the profit motive. Some tweaks may be conveniently “overlooked” in favor of profit. For example for many years the Oracle installer would not complete without error on many Linux systems. Imagine all the professional services that are sold around running through a complex install. Streamlining such an install would *reduce* profits. Don’t laugh.

What happens on the front lines?

On the front lines of course are the ops teams & DBAs, actually installing and supporting enterprise software. Let’s not forget these guys are at the command line. They know inordinately more about what’s really happening down in the trenches. You may find them repeatedly rolling their eyes at salesmen claims.

However they are not the colorful storytellers or communicators that salesmen are, so they may

Want to hire a DBA? Here’s our MySQL interview hiring guide. We also wrote a similar one for Oracle DBA Interview questions.

Align each division’s interests

Despite cultural differences, business management & operations teams should work hard to connect, and align with one another.

Operations should make an effort to better understand the business bottom line. Money doesn’t grow on trees as they say, and choices have to be based on budget, and real-world needs. We’d all like to sit in a university and program or build things just to create something new, but in a business there are market pressures. All teams should reflect on those.

Management should also make an effort to understand ops teams needs. Why are my ops teams telling me a different story than they Oracle sales guys? Fight the urge to bond with the sales folks, despite their smooth delivery, great suits and peer positioning.

Weigh short and long term tradeoffs

List out advantages & tradeoffs on all sides. These should be technical and business bullet points. Brainstorming a full list like this, and having the whole team discuss the list openly will help the team together come up with a more realistic outcome. Some questions to ask…

1. What are the advantages & disadvantages of having multiple providers for your technology stack?
2. Which solutions are open and which are proprietary? What are the tradeoffs there?
3. What does your team have subject matter expertise in?
4. Are there real technical advantages to one solution or the other?
5. Are there real cost advantages to one solution or the other?
6. Are there expertise advantages & training savings to go one direction?
7. Is the technology widely used in your industry? Will additional or replacement operations experts be easy or hard to find?

Read this far? Grab our scalable startups newsletter.

Juggling apples & oranges in the datacenter


In which a few choice words become one serious accident…

The Backstory

More than five years ago now, I worked for a shop in the business of news & information around the legal and real estate sectors. It was a fairly large organization with a number of Oracle and MySQL backed applications. The whole place ran on Sun servers, with a team of systems administrators, developers, and of course editors & content folks.

I was the primary database administrator for almost an entire year back then. I reported directly to the CTO. She was bright, competent and great to work for.

Although she had a technical background, she often spoke about products and gave very high level directives when making requests. This was made more confusing as the environment lacked naming conventions. So often product names didn’t match server or database names.

I tended to take the very paranoid approach. I’d ask over and over for clarification, and let some time pass before actually executing on a request.

A Changing of the Guard

After many months as a contractor DBA, the firm finally located a fulltime guy to replace me. It’s no easy task finding a DBA these days, especially for MySQL.

He was a very bright guy with a lot of technical knowledge. A bit green behind the ears, but fully capable to manage an enterprise database shop.

Looking for a top-notch DBA? Here’s our MySQL interview questions & hiring guide. We also have one for hiring an Oracle DBA.

Nuking the database

After two weeks on the job, something unpleasant happened.

Imagine a chef working with cooks & confusing dishes with vegetables.

[quote]
Chef says, “Toss the avocado”
Cook throws the avocado salad in the trash thinking it’s rotten.
Chef comes back later asking quizzically, “I wanted you to mix it up!”.
[/quote]

In the datacenter the conversation went something like this…

[quote]
CTO: Drop the journal database & rebuild.
DBA: Ok. Give me a few minutes
CTO: What did you do? The whole application is offline now!
[/quote]

From there scrambling ensued. After nearly six hours of screaming, and firefighting, everything is finally restored from backups and the application brought back online.

Naming – product or components?

Semantics is very important. Those in the trenches tend to take requests word-for-word while those managing the troops tend to make requests in terms of products, divisions & the vantage point of the business.

That’s why naming conventions can be so important. Don’t want to be talking about apples when you really mean oranges.

Living with dysfunction

As environments grow over years and years, they tend to evolve into a spaghetti of confusing names & relationships. It’s the nature of enterprise environments.

– big confusion can mean big mistakes
– check & recheck – be risk averse and a bit paranoid
– check yourself, your shell, your hostname, your login
– ask questions & clarify repeatedly
– let some time pass before executing a destructive command

Made it this far? Grab our newsletter.