Upcoming for Scalable Startups

Just back from the Labor Day holiday, and ready to dive back in.

I thought this would be a great time to outline some of our upcoming topics so here goes…

1. Why Oracle usability sucks

– a rant about Oracle’s weak points

In the meantime take a peek at our piece on why we wrote the book on Oracle & Open Source. We ruminate on trends in the datacenter and take a stab at Oracle’s future.

2. Why relational databases don’t scale

– Is there any such thing as automatic scalability?
– What blocks scalability?
– Are NoSQL databases magic?

Also one of our articles that went viral – 5 things toxic to scalability

3. Eternal tension between dev & operations

– origin in different job roles & priorities
– balance found in each appreciating the others point of view
– hiring the best or building the right culture

You might enjoy a wildly popular piece we wrote a few months back How to hire a developer that doesn’t suck.

4. MySQL Query Tuning Cheatsheet

– SQL queries are hellish to tune, that we know.
– An outline of some of the common patterns that don’t work will help you identify and avoid them.

5. Differentiation in professional services

– A commodity does not stand out in the services business
– Differentiation is about personality, relationships & how you solve problems for the business

Also in the meantime take a look at our professional services 101 guide.

Read this far? Grab our monthly scalable startups newsletter.

31 Essential Blogs for Startups & Scalability

So many blogs, so little time! Here’s our list of the best we’ve found. Currently our favorite reader is Pulse pictured left. Starting to play around with flipboard too.

Nuts & Bolts Technical

Slashdot
One of the original tech blogs, that still covers lots of breaking news, and difficult topics. Very technical, with probing commentary. Beware the actual comments though, as they’re often full of immature and childish rants.

Planet Mysql
An aggregator of many MySQL blogs, it hits on topics from benchmarking, and advanced tuning, to new technologies on the horizon. Drupal and LAMP topics are often also covered.

mysql performance blog
Percona’s technical blog never disappoints. There are endless posts about a myriad of topics related to deploying, tuning and optimizing MySQL and all it’s variants.

Hacker News
You may not like Paul Graham, he’s easy not to. But his YCombinator News site is an awesome collection of always surprising technical topics that are sure to keep you busy.

Netflix Tech Blog
You might have read about Chaos Monkey before, that

Our very own Scalable Startups
You’re already reading us regularly of course! Why not grab our newsletter?

Programmable Web
Mashups & APIs. What more do you want? Very cool stuff here.

Also take a look at our best of compilation.

Business & Economics

HBR Blog
If you’ve ever read Harvard Business Review, you know how in depth and on point the material is. More thorough discussions than many other blogs, and excellent discussions in the comments.

Marginal Revolution
Tyler Cowen’s endlessly interesting and provocative take on the world through the eyes of economics. Like using science to analyze and solve the worlds ails, this blog always has a reasoned take on things.

NPR Planet Money
I’ve been listening to this podcast religiously since the financial crisis of 2008. It continues to intrigue and educate me in ways that college finance never did. You’ll learn a lot.

Bloomberg Businessweek
BBW despite it’s name is like Wired back in the 90’s before it got taken over by Conde, and the cutting edge writers and risk takers left. That’s right this magazine is full of analysis, creativity, and color. It’s what you’re looking for in a print magazine. One of my favorites.

Inc. Magazine
Real articles for your small business needs today. Thoughtful and topical.

Forbes Magazine
Banking, finance, politics, news.

You might also check out our Scalable Startups newsletter archives.

Venture

A VC
Fred Wilson’s iconic blog is always on the cusp, with a thoughtful and participating audience of readers.

Infochachkie
John Greathouse is a VC with a very readable blog on startups and investing.

Chris Dixon
This guy invested in tons of great startups that are household names now. With a very readable blog to match, he’s a man with ideas that we all benefit from.

Springwise
As they call it, your “Essential Fix of Entrepreneurial Ideas”.

Feld Thoughts
Brad Feld is another big VC with an excellent blog on topics relevant to Venture Capital & Startups.

Social

Andrew Chen
Consumer internet, metrics, and user growth. Brilliant idea guy. I learn from this guy’s blog everytime I check it.

Problogger
The smarties behind the book of the same name, this is essential reading for bloggers who wanna make a dent in the world.

Blog Tyrant
How to build successful blogs that make real money. Learn from Ramsay Taplin who’s done it already. Whether your blog sells products, widgets or services, there’s stuff for you here.

Kissmetrics Marketing Blog
Very good stuff on marketing, twitter, facebook and all the other good social topics.

Mixergy Blog
Business tips & startup advice with a bent towards marketing and social.

Mark Schaefer Marketing
Mark’s the brains behind the great book Return on Influence which we reviewed. His Businesses Grow blog is full of helpful ideas and insights.

Figaro Speech
You may have read my review of Word Hero and seen the earlier review of Thank You For Arguing. His blog is a real gem, extending on the wonders and lessons of word hero, you’ll be writing witty and memorable one-liners and titles that will go viral tomorrow!

Industry

Gigaom
Om Malik started out writing about the bandwidth boom and bust of the 2000’s. His blog has grown wildly to cover the industry as a whole, and contrary to the stuff you get on business insider, this is quality journalism.

AllThingsD
Another industry site with a great selection of journalists writing on the internet & startup industries.

ReadWriteWeb
Another excellent industry blog with slightly overlapping coverage to gigaom and allthingsd, but worth scanning each of them for different perspectives.

Venturebeat
Possibly a bit more venture and investment oriented than the others, but still mainly an industry coverage blog site.

Entrepreneur
Slightly more focus on business, and entrepreneurs, but also internet & startup industry topics.

Adweek
Trying to broaden my horizons by adding this one into the mix. Some very interesting topics, and plenty of overlap with internet industry and startups.

If you read this far, grab our newsletter!

Opportunity a day – career risk at bay

Free Agent. Stress Test. Avoid Sameness

As the globalization juggernaut rolls on, it continues to create more Detroits. Skills and perspectives quickly become obsolete.

What to do in the face of such change?

[quote]Small fires prevent the big burn[/quote]

So there’s your quick answer. Get the book if you want more!

Some related material: why is it so hard to find a mysql dba?.
Consulting 101 Guide – Finding Business :: Completing Engagements :: Growing business

Your Mentors

On this tour, a free agent needs mentors. Hoffman & Casnocha provide you with plenty from stories & lessons from some of the startup industry’s finest. Jack Dorsey, Mark Andreesen, Cheryl Sandberg, Rick Warren, Paul Graham, Jeff Bezos, Joi Ito and a few of their own running Paypal & Linkedin.

What you’ll love

Each chapter closes with concrete actionable advice. The authors carefully craft marching orders for you in the next day, next week and next month. Go ahead, give them a try.

[quote]Safe is the new risky – Phil Simon[/quote]

An executive summary of Startup of You

1. develop your strengths
– what do you find easy that others find difficult?
– diversify asset mix aka learn new skills

2. plan to be nimble
– pivot as you learn more
– always prepare a lifeboat contingency plan

3. work & develop your network
– hangout with those already on the road
– domain experts, people who know you & smart people

4. hustle for breakout opportunities

5. Embrace baby steps of risk
– bounds of unemployment – shocks that motivate
– adjust your strategy & pivot if need be

What’s next?

Had a taste and want more? If you’re a MySQL DBA we wrote an interview guide. Also check out our Oracle dba interview questions.

Want more? Check out our best of content compilation.

[quote]Only the paranoid survive. – Andy Grove[/quote]

Read this far? Grab our newsletter. We cover all sorts of great topics for free agents, consultants, and those who want to hire them.

Sometimes… let things break a little

Have you ever started a new project, just into it you realize that maybe there aren’t technical problems to solve? It starts to dawn on you the real crux of the problem boils down to people & processes?

It’s happened to me on a number of occasions, but once in particular really stands out for me.

I was working for a firm in the education space, in particular around test preparations.

Asked to automate a publish process

The environment had a mix of relational databases, from SQL*Server to MySQL for some applications. The web facing database however used Oracle on the backend.

Their career DBA was real old guard Oracle, he had his ways of doing things, and didn’t want to rock the boat. In particular he managed the process for publishing changes to the website. Publishing amounted to running a few hand rolled scripts and each step was a manual one.

With the process setup this way, the editors had to work closely with engineering each time they wanted to move content to the website. Slow, cumbersome, and not very workable.

The real problem, siloed departments & infighting

As I worked closely with the DBA, quite a few things became clearer. For one he was sometimes a grumpy fellow & he had a strong accent which was sometimes hard to cut through. Knowing the other team members, I knew this all contributed to the trouble. But he maintained quite a bit of resistance to automation of the process no matter what. His view was, if he hands over the reigns to editors who don’t understand the technology, they’ll screw something up, make a mess, and that would ultimately create more work for him. After all that he’d be doing it manually anyway!

Further attempts to communicate between teams or even between the managers and this guy went nowhere.

It’s not easy to find a good DBA. We wrote a MySQL interview guide to help and one for an Oracle interview too. The mythical dbas remain in rather short supply.

We weren’t trained for this in engineering school

When you’re looking for a technical solution and you realize the bigger issue is a people problem, what do you do? You can gently bring it up with the higher ups, but they may have a different style, or prefer the shout orders and bark mode of management.

Things continued to go around in circles, and attempts to get further information from this DBA didn’t prove fruitful. He was protective of his domain, and fought tooth & nail to open up.

Things come to a head

The bigger boss, the one above my direct report, one day called me into his office. Actually it was an off day I was just stopping by to check in on progress.

He pulls me into his office. Since I rarely interacted with the guy, my guard was very much down. I thought we’d have a chat about the weather or perhaps who was going to win the world cup.

He proceeds to tear into me without warning. Practically screaming, he’s giving me a piece of his mind and not stopping to hear what I have to say. Where is this project going, why is there no progress, we’ve got serious deadlines, you’re pushing us right up to the wall … that kind of thing.

As I listened to him fire away at me, I realized some of this had gotten filtered through some sources, who didn’t completely understand the blocking issues either. That it wasn’t technical challenges, but rather people and processes that weren’t working. As I began to explain this, he stopped me and said:

[quote]Sean, your job is to push, push, push and push us more. Rock the boat if necessary. When I was a constultant I was constantly running around making sure everyone was talking to eachother[/quote].

A few things ran through my mind at that point. One was well he’s not a consultant, so either he couldn’t last in it, or the life didn’t appeal to him. Or perhaps his skill sets ran truer to management in a large firm, a different albeit tougher role to master.

But it also occurred to me that different folks have very different styles, some like to push, and prefer confrontation & believe that leads to resolution. While others are more listeners and find there way around a problem by giving everyone a chance to voice their positions.

My style is the latter, while his was clearly the former.

The fallout

What ended up happening is a project manager also got assigned to the project, as well as another manager. The PM liked working with me very much, and as things unfolded, much of the departmental siloing began to dissolve, and the bigger communication problems began to surface. From there solutions followed.

Lessons learned

– beware the status quo – some don’t really want to rock the boat
– communicate your position, but beware that others may have a different style
– you may be asked to support a path you see as the wrong one
– let things “break a little bit” so everyone learns the hard lessons
– getting burned can be a lesson for the whole team, and lead to new solutions

Made it this far huh? Grab our newsletter.

Where’s my 80 million dollars?

Way back in the heydays of the dot-com boom, the year is 1999.

Join 12,100 others and follow Sean Hull on twitter @hullsean.

I worked for a medium size internet startup called Method Five. When I came on board they were having a terrible time with their site performance.

Website crashing

When I first met the team, I was tasked with performance problems. After all their flagship web property kept crashing, and it didn’t look good to investors. As with most web properties in those days it was a home-grown datacenter in the back of the office, running on Sun Microsystems hardware, with Oracle on the backend and Apache serving webpages.

Also: Why a killer title can make or break your content efforts

Negotiating an acquisition

As it became clearer after day one, the project was particularly sensitive. They were negotiating a huge acquisition by a firm called Xceed Corp. The sticking point? Their crashing website did not sell their technology prowess in a particularly positive light. To say the least!

Read: Why high availability is so very hard to deliver

Investigation

As it turns out the site had all the right players, from systems administrators to a DBA who sat watch over the Oracle systems.

As I dug into the systems, I found a serious smoking gun. It seems the Oracle software was configured to use just 5M of memory out of about 256M free. Just like MySQL, the server must be configured to use available memory upon startup. There are myriad caches and buffers which need to be attended to. By today’s standards these numbers probably sound absurd. Nevertheless the DBA wasn’t familiar with the basic memory settings, and so the system was terribly bottlenecked.

Read this: Why a four letter word divides dev and ops

Problem Solved

We then ordered some urgent changes to the system, configuring all of Oracle’s caches to use up the precious memory available.

Immediate the website unlocks, transactions begin flowing, and webpages are returning quickly. End users pull their noggins off their keyboards, and the executives begin breathing a sigh of relief. The site was literally 1000x faster during peak.

Related: MySQL interview guide for managers and candidates alike

Acquisition

Shortly thereafter the acquisition goes through for a cool 5 million in cash and 80 million in stock.

Where’s my cut?! You might be asking that question. But my policy is almost always defer to something concrete and tangible, aka fees and real compensation. I did not negotiate any stock in the deal.

Another popular war story we wrote A CTO Must Never Do This….

Read: Why devops talent is in short supply

Lesson’s Learned

o Don’t believe received wisdom. Check and double check what’s really happening.
o Use the memory and resources you have available.
o Measure capacity, and isolate bottlenecks in the system
o Decouple services wherever possible
o Problems are as often people and process as they are with technology

Also: 5 more things deadly to scalability

Make it this far? Grab our newsletter!

You're Too Young To Be My Boss

About a year ago I engaged with a firm to do some operations work on their site. They provided services to colleges and universities.

When they first reached out to me, they were rather quick to respond to my proposal. They seemed to think the quote was very reasonable. I also did some due diligence of my own, checking the guy’s profile on the about page. I noticed he was 25, rather young, but I didn’t think much else of it.

We discussed whether they wanted fixed hours. Since those would limit my availability we both agreed a more flexible approach made sense. This worked well for me as I tend to shift and schedule time liberally, so I can be efficient & flexible with clients, but still have a life too.

Trouble Brewing

As we began to interact the first week, I sensed something amiss. My thought was that the first week you work with a client, they feel you out. They see how you work, when you work, how much gets done and so forth. This provides a benchmark with which to measure you. If either party is unhappy with how things are going, they discuss and make adjustments accordingly.

What was happening in this case was the guy started pestering me. I began to get incessant messages on instant messenger asking for updates. I had none. I explained that I would contact him as things were completed, or if I had questions.

This was only two days into the project. I’d barely gained access to the servers!

The Fever Pitch

After discussing my concerns on the phone, the gentleman kind of glossed them over. From there the pestering continued. I explained that I could not be available to him any hour of the day, while the engagement only provided for one half of a week. This began to interrupt me from other client work, so I had to signoff of instant messenger. Not good.

The Pot Boils Over

We spoke again on Monday briefly, and decided to connect the following day. From there the pestering began anew, and I began to lose my patience. I insisted that we speak on the phone before work would continue. I felt the problem was deteriorating and discussing over text would only make things worse.

He emailed me back as I was then offline. In his email he ordered me to come online. While he sat in a meeting, he explained, he could not take a call! Nevertheless he insisted we resolve it during the meeting. Distracted no less.

[quote]It was then that I started receiving text messages on my personal mobile phone from the guy, pestering me to get online so we could resolve our communication problem! You can’t make this stuff up![/quote]

The Fallout

Eventually we did both get on the phone, and I explained I had reached wits end. After only ten short days of working together, we had both set strong precedents and they were obviously not compatible. He asked if I would stay on longer, and reconsider working together, and I said I would think about it.

I chose not to dig a deeper hole, and let him know I wouldn’t be invoicing for previous the weeks work.

The Lessons

o beware age differences – in our case an 18 year gap
o pay attention to management styles – self-starters don’t need micromanaging
o be patient & keep communicating
o allow for an exit strategy that is amenable to both parties

Read this far? You’ll love our newsletter. Get Scalable Startups. No Spam. No Selling..

A CTO Must Never Do This…

A couple years back I was contacted to look at a very strange problem.

The firm ran flash sales. An email goes out at noon, the website traffic explodes for a couple of hours, then settles back down to a trickle.

Of course you might imagine where this is going. During that peak, the MySQL database was brought to its knees. I was asked to do analysis during this peak load, and identify and fix problems. Make it go faster, please!

First day on the job I’m working with a team of outsourced DBAs. I was also working with a sort of swat team chatting on SKYPE, while monitoring the systems closely.

Then up popped one comment from a gentlemen I hadn’t worked with. He insisted there was contention for a little known MySQL resource called the AUTO_INC lock. Since I wanted to know more, I asked who the guy was and to my surprise he turned out to be the CTO.

[quote]The CTO was tuning and troubleshooting the database![/quote]

Wow, that’s a first. I thought I’d seen it all. A CTO is normally overseeing technology & the team rather than crawling around in the trenches on the front line.

This all raised some important points

1. The app was having major growing pains
2. Current architecture was not scaling
3. Amazon elasticity was not helping at the database layer
4. People & process were also failing, hence the CTOs hands on approach

It was shocking to see a problem deteriorate to this point, but when you consider the context its understandable. A company like this is struggling with hypergrowth to such a degree, that each day seems like a hurricane storm. With emergency meetings, followed by hardware & application emergencies, trouble seems constant. It can be very difficult to step back and see the larger picture.

The takeaway from this experience…

o Amazon EC2 can’t do it all – consider physical servers for disk intensive apps
o MySQL still has some real scalability limitations
o use technology for its intended purpose – MySQL isn’t great for queueing
o A CTO tuning the database means problems have deteriorated too far

Read all the way to the end? Grab our newsletter – scalable startups.

Half of NYC Fastest Growing Firms In Tech


If there was ever any question about the exploding tech scene in New York City circa 2012, check out Crains recent 50 fastest growing firms.

#3 Thrillist

City guide & lifestyle site for men.

#5 Usablenet

Provides a powerful platform for content delivery to mobile & tablets as well as laptops.

#8 Admarketplace

A leader in search syndication.

#9 Yodle

From the bloodbath of newspaper classifieds brought on by disrupters like Craigslist & eBay, comes a new breed of online advertiser catering to customers and the digital space.

#15 Telx

Provides datacenter services such as high speed connections, private clouds, servers & networking equipment.

#16 Complex Media

An online style and lifestyle magazine catering to young men.

#18 Wavsys

Finding talented mobile engineers isn’t easy. This firm provides contractors for the big wireless firms.

#19 Return Path

Helps firms with email delivery, by routing around and through spam filters.

#20 M5 Networks

Cloud Based phone systems.

#22 Mitchell Martin

Tech staffing and consulting solutions

#25 Artech Information Systems

Staffing for technology and project management requirements.

#27 Infinity Consulting Solutions

Recruiters for the technology industry as well as finance and accounting firms.

#28 Vicom Computer Services Inc.

Providing high end IBM enterprise solutions to datacenters.

#35 Direct Agents Inc.

A fast growing digital marketing firm.

#37 TTI of USA

Another staffing & consulting solutions firm for technology firms.

#41 Net@work

Bringing better customer service to technology consulting & services industry.

#44 Insys Group

Technology consulting and services.

#46 Travelclick

Provides technology solutions that help hotels increase revenues.

#48 Tekserve

Apple products repair & servicing.

#49 Axispoint Inc.

Business technology services and solutions.

#50 Paradysz Inc.

Email & social media marketing company

What Recession? – New York Tech Hiring Frenzy

startups hiringAccording to Crains, New York is digital jobs central.  ZocDoc, Thrillist, Foursquare and 10gen are just a few of the hot shots located at 568 Broadway, a tech hub in Soho.  Each of these firms is looking to double their headcount in the coming year.  Not bad considering the rough shape much of the economy is in.

Don’t Forget the Big Boys

Let’s not forget the huge tech firms that are also on a hiring binge,  Google, Facebook, Zynga, Twitter & Skype are all hiring.  Bloomberg Businessweek describes it literally as Silicon Valley setting up shop here!

This interactive map of the tech scene in New York is striking.

Population Growth

New York is also experiencing serious population growth.  The city expanded by 70,000 people in the last 15 months according to Gothamist.

According to another Crains article, what we’re seeing is the best job gains in 60 years!

Increased Competition for Talent

All this means increased competition for the best talent.  These days only the biggest or the coolest firms can snag the smartest people.

We wrote some hiring guides that might help: How to Hire a Great Software Developer and another about filling the Mythical MySQL DBA role.

A two part article – How to Hire a MySQL DBA and Part two.

All This and Fun Too!

And hey, what other city has spawned the European inspired Lunch Beat?

 

 

Ask Me Questions – Scalability, Performance, Cloud Computing

 

 

I blog a lot about various topics near and dear to me.  So I thought I’d turn the tables a bit, and offer the microphone up to readers.  Do you have questions on any of these topics?  Feel free to drop a note in the comments.

 

Business & Consulting

Are you a freelancer or independent consultant?  Struggling with some part of the business?  Or are you a CTO or Director looking to hire short term talent?  Glad to offer up advice and suggestions, just fire away!

Scalability

The goal of every hypergrowth company, from Pinterest to Facebook, Zynga to FourSquare.  How do they achieve it?  What architecture decisions make those applications grow effortlessly to meet user demand?  I’m sure you have some questions!

Performance

It’s on everyone’s mind, especially in the context of deploying in the cloud.  How best to achieve good performance?  Are you struggling with something specific?

High Availability

We want our systems to be available 24x7x 3million days a year!  Is it feasible?  What should we aim for in the real world?  Does our technology mix and hosting factor into the equation?  Ask away.

Startup Challenges

I’ve worked with a lot of startups over the years.  From the dot-com hey days to the more sober field we see before us today.  Many struggle with similar challenges.  Am glad to offer up comments and suggestions from what I’ve learned over the years.

Cloud Computing

What really works in the cloud? Is Amazon EC2 the only way to go?  What are the pros and cons of going with a Rackspace type provider that also has great service and the physical data center option?  How will I get good performance?  These and other questions are on everyone’s lips.  Feel free to comment if you have some concerns.