Category Archives: CTO/CIO

Are startup CEO’s hiding their scalability problems?

Russian_Dolls

Join 27,000 others and follow Sean Hull on twitter @hullsean.

Your site is running fine right? You have 1000 customers, and it usually runs smoothly. Just this one lingering question, why does it take five high performance EC2 instances to run the database, all on flash drives? Goood question!

The truth is one of the highest trafficed sites I managed, pulled in 100 million uniques a month, and only used three backend databases. That site was one of these wildly popular celebrity gossip sites, the ultimate guilty pleasure when you’re at the office and can’t watch reality tv!

Snickers aside, this is huge traffic. And all of the above was built on Drupal, with no ORM in the mix. It could even run, albeit noticeably slower, while memcache was disabled.

1. Servers with solid state drives

I’m very excited to see Amazon introduce servers with SSD drives. They can bring you 100x improvement of disk I/O, and that my friends is the end all and be all for databases. So why complain?

If you deploy on these boxes right out of the gates, it may be like using a crutch. You become dependent on it, and ignore real performance tuning. Solid state drives still won’t obviate that ORM middleware you’re using.

Also: Do managers & CEO’s underestimate operational costs?

2. Memcache saving your bad queries

Memcache is also a powerful tool. It sits between the database and your webservers, reducing load on the database by as much as 10x. That’s a great way to get better response time, and reduce drag on your db tier. But it’s still worthwhile performance tuning without it.

Why? If you can get your site to run without caching, it will run blazingly fast *with* it. Don’t use it as a crutch, use it as rocket fuel for your well tuned site.

Read this: Do startups need techops?

3. A legion of read slaves

I’ve seen smaller sites, using a ton of read slaves. All of it deployed to cover up slow & redundant queries pouring out of an ORM middleware layer, in this case Cake PHP.

Again, read slaves are great, but tune & test with less hardware, and get the performance up the hard way. With elbow grease!

Related: Howto automate MySQL query analysis with Amazon RDS

4. Really really big memory

64G, 128G, 256G of main memory? If I wax on about the days when you’d get excited by 64k, I’ll sound like an old timer. But with those extreme limitations, you had to write tight code. Otherwise it just wouldn’t do anything.

Really really big memory of today’s servers allows us to get lazy. I hear developers say “Hey, the database is 10G of data, and we have 64G main memory, so the whole thing will fit in memory. Problem solved!”

Duhhh… No. Why not? Because you still have to slice and dice that data. You still have to scan through for bits & pieces that aren’t indexed, then sort, and organize that into temporary memory space. In DBA speak, you’re still doing a ton of logical IOs.

Picture it another way, imagine the days when you’re on horseback, riding across the west. You travel light cause frankly your horse can carry only so much. Then along come cars, and you start loading up the trunk. You add the kitchen sign, and the rear tires are hanging on the ground. All seems fine until you hit a steep mountain, and you’re car is almost stalling at 20mph. If you had only carried the same load as you did on horseback, you’d be speeding across the country at lightning pace.

Read: Is Amazon RDS hard to manage?

5. Deploying poor code

Deadlines are looming, and new features must be deployed. So performance testing can wait until later. The code works after all.

Been there, done that. Code gets deployed and all of a sudden there are spikes on server load in the evening. Ops & DBA teams are screaming, “Who wrote this code?”.

Load testing should be a part of everyday QA & test. It’s the only way to avoid growing scalability problems.

Check this: Are SQL databases dead?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Why I ask clients for a deposit

Editor & writer in friendly dialog

Join 25,000 others and follow Sean Hull on twitter @hullsean.

1. It indicates both parties are serious

A common refrain when discussing terms of a project, and reviewing statement of work – “when shall we get started?”. The answer should be, “I’m ready to get started anytime you like. Would you like to use paypal or ACH for deposit?”.

The deposit signals to the vendor that it’s time to get working. This client has the budget and is serious about moving forward today.

Read: Why Fred Wilson is wrong about Apple

2. It protects against scope changes

Startups & seasoned businesses alike have changing needs. That’s why they may choose a situational resource to begin with.

If the winds change, and we don’t need you tomorrow, a deposit defrays the final invoice, and or discounts you may have applied.

Related: Is Dave Eggers right about risks of social media?

3. Insurance if business fizzles

Fizzling business, is a nice way to say the market has changed. Perhaps the startup has decided to pursue other opportunities. In close to twenty years of business I’ve only had this happen twice.

Once I was working with a rewards card business. They were already having trouble meeting payroll. Turns out businesses have a legal obligation to meet payroll. That’s another way of saying they’re at the top of the who-gets-paid list. And vendors may be closer to the bottom. They owner went back to being a lawyer, his profession before the startup.

All in all, a deposit provides some insurance in these cases.

Read this: 5 cloud ideas that aren’t actually true

4. Signals your maturity to client

This is a hard one for some freelancers and consultants to stomach. “I really want to get going with consulting, and don’t want to turn away this client.” The thinking goes. But consulting is a peer relationship, where vendor and client need to be on an equal footing.

Your need for a deposit, and willingness to walk away without one, says to the client you are professional and have been in business for some time.

Also: If you’re using MySQL in the Amazon cloud, you need to ask yourself this question

5. Protection from early termination

That sounds ominous but it doesn’t have to be. In the world of freelance and consulting, a client can decide they no longer need your services tomorrow.

Why? Perhaps they hired a fulltime resource? Perhaps their needs changed? Perhaps the storm of site outages have passed and the urgency has changed.

Whatever the reason, projects change. If you’ve offered a discount for three months of work, but only end up with one month of work, your full fees may apply. In that case, the deposit should be the discount amount.

Check this: Do managers underestimate operational costs?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Why Fred Wilson is wrong about Apple

apple_android

If you’ve followed the tech news recently, you may have heard Fred Wilson’s comments about Apple. In essence he believes Apple is too reliant and rooted in hardware, and that hardware isn’t viable in the long term. Mobile hardware, is becoming a commodity.

Join 25,000 others and follow Sean Hull on twitter @hullsean.

To be sure Androids have come a long way, and they may yet improve a lot by 2020 as Fred says. But the aftermarket value of iPhones really does speak volumes. See below.

1. iPhone has never had the best hardware

If you’ve ever watched a Samsung ad, or talked to someone with the phone you probably know this already. Bigger screens, faster processors, first phones with fingerprint readers, or untethered syncing. The list goes on and on.
Also: 5 Cloud ideas that aren’t actually true

Yes Apple is rooted in the hardware business, but not in a way that a commodity can disrupt it. They’re rooted in the hardware business only in as much as it helps them deliver polish. If it helps them deliver a seemless experience, and a device that Jean Luc Picard would appreciate, then they are in that business. . Just “make it so!”.

2. Users are seduced by simplicity

So how is it possible that an inferior piece of hardware could sell more?

Easy. Those users don’t think that way. They aren’t buying hardware. What do I mean?

I would argue many iPhone users buy for the experience, the simplicity, the ease of use. Designers call this User Experience or User Interface, but end users don’t know these terms. What they know is they don’t have a headache. They’re not frustrated trying to move an image from one app to another, or copy/pasting etc.

User interface is that invisible force that just makes everything on the device better. Call it polish, but it’s much more than a pretty face.

Related: Are SQL Databases Dead?

3. Most users don’t care about “open”

Another benefit touted on the Android side is it being “open”. The OS is open-source, and then extended by manufacturers. While this surely brings down costs to them, it may be all be irrelevant to end users and consumers.

Yes open standards are great for competition, great for markets, and ultimately great for users. But Microsoft is a great case study in why consumers often still choose a closed solution.

Read: Do managers underestimate operational costs?

4. Apple is Sexy

That may sound fanboyish, but seriously. Look at the accessories market for blinging your phones.

If that’s not enough, look at the aftermarket value. iPhones retain their value, Samsung’s don’t.

Read: Five things I learned from David Maisterabout trust and advising clients

5. Android is still broken

From where I’m standing, and a lot of experts agree, the Android ecosystem is broken.

For one the AppStore, being historically unregulated, is chock full of malware and dangerous downloads. Most users aren’t computer experts, not good at evaluating security risks, and pay the price.

What’s more many Android phones come stocked with bloatware, slowing down the device, and reducing reliability from day one.

Read: Why the Android ecosystem is broken

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

5 cloud ideas that aren’t actually true

storm coming

Join 20,000 others and follow Sean Hull’s scalability, startup & innovation content on twitter @hullsean.

Cloud computing is heralding us into a wonderful era where computing can be bought in small increments, like a utility. This changes the whole way we plan, manage budgets, and accelerates startups making them more agile.

But it’s not all wine & roses up there. I’ve heard a few refrains from clients over the years, and thought I’d share some of the most common.

1. Scaling is automatic

Rather recently I was working with a client on building some sophisticated reports. They needed to slice & dice customer data, over various time series, and summarize with invoices & tracking data. Unfortunately their dataset was large, in the half terabyte range.


Client: Can we just load all this data into the cloud?
Me: Yes we can do that. Build a system in Amazon public cloud, can support large datasets.
Client: I want it to scale easily. So we won’t have these slow reports. And as we add data, it’ll just manage it easily for us.
Me: Well it’s a little bit more complicated than that, unfortunately.

Unfortunately this is a rather familiar conversation that I have quite often. A lot of the press around cloud scalability, centers around auto-scaling, Amazon’s renowned & superb virtualization feature. Yes it’s true you can roll out webservers to scale out this way, but that’s not the end of the story. Typically web applications have a lot of components, from caching servers, to search servers, and of course their backend datastore.

But can we scrap our relational database, such as MySQL and go with one that scales out of the box like Riak, Cassandra or Dynamodb?

Those NoSQL solutions are built to be distributed from the start, it’s true. And they lend themselves to that type of architecture. However, if you’ve built up a dataset in MySQL or Oracle, and more so an application around that, you’ll have to migrate data into the NoSQL solution. That process will take some time.

Like teaching a fish to fly, it make take some time. They do well in water, but evolution takes a bit longer.

Related: RDS or MySQL 10 use cases

2. Disaster recovery is free

In the traditional datacenter, when you want DR, you setup a parallel environment. Hopefully not in the same room, same city or same coast even. Preferrably you do so in a different region. What you can’t get around is dishing out cash for that second datacenter. You need the servers, just in case.

In the cloud, things are different. That’s why we’re here, right? In amazon you have regions already setup & available for plugin-n-play use. Setup your various components, servers, software & configure. Once you’ve verified you can failover to the parallel environment you can just turn off all those instances. Great, no big charges for all that iron that you’d pay for to keep the rooms warm in an old-school datacenter. Or do you?

As it turns out, since you don’t have this environment running all the time, you’ll want to test it more often, run fire drills to bring the servers back online. That’ll incur some costs in terms of manpower. You’ll also want to include in there some scripts to start those servers up, and/or some detailed documentation on how to do that. And don’t lose that documentation, either will you?

You may also want to build some infrastructure as code unit tests. Things change, code checkouts evolve, especially in the agile & continuous integration world. Devops beware!

Read this: Why a killer title can make or break your content efforts

3. Machines are fast

Fast, fast, fast. That’s what we expect, things keep getting faster, right? Hard to believe then that the world of computing took a big step backward when it jumped into the cloud. Something similar happened when we jumped to commodity Linux a decade ago.

In amazon, it’s a multi-tenant world. And just like apartment buildings, popular restaurants, or busy highways you must share. When things are quiet you may have the road to yourself, but it’ll never be as quiet as a dirt road in the country!

Amazon is making big strides though. They now offer memory optimized & storage optimized instances. And an even bigger development is the addition of the most important feature for performance & scalability. That said the network & EBS can still be a real bottleneck.

Also: What is a relational database & why is it important?

4. Backups aren’t necessary

I’ve experienced a few horror stories over the years. I wrote about one noteworthy one When fat fingers take down your business.

True EBS snapshots make backing up your whole server, well a snap! That said a few extra steps have to happen (flush the filesystem & lock tables) to make this work for a relational database like MySQL or Oracle. And suddenly you have a verification step that you also need to perform. You see no backups are valid until they’ve been restored, remember?

But even with these wonderful disk snapshots, you’ll still want to do database dumps, and perhaps table dumps. Operator error, deleting the wrong data, or dropping the wrong tables, will always be a risk. Ignore backups at your own peril!

Check this: Why CTOs underestimate operational costs

5. Outages won’t happen

In an ideal world, everything is redundant, and outages will be a thing of the past. We’ll finally reach five nines uptime and devops everywhere will be out of work. :)

It’s true that Amazon provides all the components to build redundancy into your architecture, and very cutting edge firms that have taken netflix’s approach with chaos monkey are seeing big improvements here. But AirBNB did fail and at root it was an Amazon outage that shouldn’t ever happen.

Read: Why Oracle won’t kill MySQL

Get more. Monthly insights about scalability, startups & innovation.. Our latest Are SQL Databases Dead?

Why managers & CTO’s underestimate operational costs

too much inventory

Join 19k others and follow Sean Hull on twitter @hullsean.

1. Technology choices & talent shortage

I worked at one firm evaluating their technology stack. When we got to the programming language, I paused in my tracks. “Haskell” I asked? “Oh you haven’t heard of it? It’s a really cool functional programming language, and we found it had some cool features that we really wanted to use”.

I had to fight the urge to roll my eyes. Yes I’d heard of the language, sitting in the club with scheme, lisp & prolog, you study them at university. They’re certainly an interesting bunch and to be sure, can do some things that imperative programming languages can’t. But did it belong in the stack of this run-of-the-mill internet startup?

In this case the developers had full reign to choose any technologies they liked, adding more & more to the mix almost daily. But what are some of the ramifications here?

Two years, three years, or five years down the line, this team will be long gone, and another team will be picking up the pieces. Will you as a manager be able to find a lot of Haskell experts? What’s more operationally will you be able to support those choices? Will updates be made often enough to have a secure stack for years to come?

Also: 5 things toxic to scalability

2. Scalability & server costs

Server costs are easier than ever to estimate. Build your application to serve your first 10,000 customers on Amazon with a couple webservers and a database server. Growing 100x to a million customers, just vertically scale your db, scale out your webservers and you’re good. Or are you?

What happens when you hit a wall? Did you build your application on ORM technology or take on technical debt? I’ve seen firm after firm struggle with technologies like hibernate, eating up precious resources, and being helpless to eliminate the problem. Tread carefully on these types of questions.

Related: Why you’re not hitting five nines uptime

3. Patching, fixing bugs & managing security

Another long term cost of an application will be minor repairs and bug fixes. Those might appear in a slow steady trickle over the years, but security may loom larger. Cross-site scripting, SQL injection and many other threats can be a real headache.

What’s more fixes may involve the libraries your application sits on top of. And when they are upgraded, your application will require tweaks too. It’s all basic stuff when you’re knee deep in development, but when your application has been deployed, the original team is long gone, and you’re supporting it years later, it can surely get messy.

Read: The four-letter-word dividing dev & ops

4. missing operational switches

When building a web application, all eyes are on features. Which ones to include, and which are a priority. Pressure is heavy to build functions that can be sold to customers. Pleasing customers is of obvious importance.

So it’s no surprise that backend switches are often missing. But they can be a real boon for operations team. Suppose you roll out a new feature to support star-ratings on certain pieces of content. An operational switch can be built to allow that feature to be disabled as necessary. If the site is loaded, or trouble is brewing, you may desperately want some switches to disable parts of the site, without the whole thing going down. I talk about this in AirBNB didn’t have to fail.

Another useful thing is a browse only mode. This allows your site to operate, even when writing to the database is not possible. If you’ve ever tried to update on a social network like twitter, facebook or instagram, perhaps late and nite and gotten a “please try again later” message, you’ll understand the value. Here users can’t make changes, but otherwise the site appears to be working, and browsing works normally.

Check this: Are SQL Databases Dead?

5. Consider bitcoin

Mt. Gox, the Japanese exchange handling bitcoin failed in a spectacular fashion. 500 million of the digital currency was stolen. And what’s more since it’s all frictionless currency, untraceable, there’s no marked bills to try and track down. Ooops!

How does this relate to operational costs? The failure was squarely with the operations department. Functionally the site worked fine. But security wasn’t handled well enough, intrusion detection wasn’t employed, and “unspecified weaknesses” were to blame.

Security is one of those things that can be ignored without pain. Until something goes wrong. What’s more if it is being handled well, it’s invisible, and unappreciated besides.

Read this: Why Oracle won’t kill MySQL

Get more. Grab our exclusive monthly Scalable Startups. We insights on scalability, startups & innovation. Our latest Why I don’t work with recruiters

Why I can’t raise the bar at every firm

Screen-shot-2012-08-02-at-1.28.35-PM

Join 17,000 others and follow Sean Hull on twitter @hullsean. Also check out: Who is Sean Hull?

It may seem counterintuitive. If I am not the best solution provider, why on earth would I highlight it?

I believe by pointing those cases out, I also underline the clients and problems that I’m particularly well suited too, and for which I can really provide value. Read on!

1. People Problems

Sometimes, you’re hired to solve a particular problem which is framed as a technical one. Some process needs to be reworked, recoded or retooled. It’s framed as a technical problem, yet as things unfold the client already has the expertise in-house to solve & write the code. What then?

It may be that the right people aren’t communicating, project managers aren’t seeing the issues, or part of the human systems are gummed up. We can’t raise the technical bar, but we can help getting those folks talking.

I wrote about this before in When You’re Hired to Solve a People Problem.

Also: Why are oil spills & financial instability related to datacenter outages?

2. ORM Usage & Technical Debt

If you’ve read my blog you know I am not very fond of Object Relational Modelers.

I would also argue as Ward Cunningham does so elloquently that technical debt can be a real and pressing problem.

Here we would help identify and frame the problem, though the work of raising the bar technically involves the longer process of retooling & refactoring your code base.

Related: Why database choices are tricky

3. Where Commodity & Offshore Works

Some firms are already making use of odesk or offshoring resources, where you might pay as little as $150/day. If you have a very technical manager or CTO, such a solution may work well for you.

At the other end of the spectrum are the high priced senior consultants from firms like Oracle, Percona or Pythian. Yes they may set you back as much as $3500/day.

In those cases a scalability & performance review may make sense. Here’s how.. Although specialists are necessary, remember to ask yourself Why generalists are better at scaling the web.

We sit in the sweet spot between the two options. With low overhead, our prices are more affordable. At the same time you’re getting a whole lot more than a commodity solution. We’ll communicate in plain language with folks at every technical level. And for many firms that in itself is a value add.

Check this: Does Oracle Aim to Kill MySQL?

4. Existing team did their homework

Believe it or not, I’ve gone into consulting engagements where the existing team has really really done their homework.

In those cases it becomes much harder to raise the bar technically. In those cases I can help when existing team missed something. But more importantly, I can validate a correct setup, or identify technical debt.

Having an outside perspective then, can provide reassurance. As I see ten to fifteen new environments per year, I’ve seen hundreds in the past decade & a half. That’s helpful perspective in itself.

Read: Does Oracle Aim to Kill MySQL?

5. Availability & Uptime Are Already High

I wrote in depth about high availability in the Myth of Five Nines

At the end of the day, availability can only approach perfection, not actually reach it. That’s a property of complex systems. If your uptime is already extremely high, again we can validate your environment, review and provide & summarize findings. But we may not be able to raise the bar.

If that’s you, it’s a good problem to have!

Also: Why AirBNB Didn’t Have to Fail

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Why Scalability Is Big Business

Russian_Dolls

Join 16,500 others and follow Sean Hull on twitter @hullsean.

1. Complexity Is Growing

Despite automation & the mass migration to the cloud, or perhaps because of it, complexity continues to grow. Back in the dot com era a typical infrastructure included a load balancer, a couple web servers, one oracle database, and that was pretty much it.

Now that has multiplied. Pile on top of that three to five more webservers, a search server, a page cache, an object cache, one or more slave databases and more. You may have a utility server with jenkins for continuous automation, monitoring applications like nagios and cacti, your source code repository and perhaps configuration management like Puppet or Chef.

That’s not only more moving parts, it’s a wider swath of skills and technologies to understand. That’s one reason Generalists Are Better At Scaling The Web.

Also: Are SQL Databases Dead?

2. Developer Mandate: Features

The pressure to build features that can directly be monetized is obvious. Startups especially have the pressure to grow fast and grow now. So security, technical debt, and scalability often take a back seat. What’s more in small scrappy and lean startups, ops sometimes falls on the shoulders of one competent but overworked developer.

Related: Why Oracle Won’t Kill MySQL

3. Startups Growing Pains

With hyper growth, startups can go from 100 customers to millions overnight. That kind of popularity is a good problem to have. But if your app hits a wall and suddenly falls over, everyone is scrambling. The pressure builds, as fear of losing that traction mounts, and heads are put on the chopping block.

Read: AirBNB Didn’t Have To Fail

4. Missing Browse-only Mode & Feature Flags

Ever been browsing for airline tickets, then go to order and get an error? Try again later? If so you’re familiar with a browse-only mode. This is a very powerful addition to any web application but is very often left out. Some mistakenly believe it won’t work for their application, as users will always be changing data.

Ever visited a website that has star ratings, only to find them missing? Or temporarily unable to edit your rating for a piece of content? This amounts to what’s called a feature flag. These powerful switches give operations teams the ability to disable heavy features, while the side is under tremendous load. They can take a huge burden off the shoulders of your servers when you hit that scalability cliff.

Check this: Why I Don’t Work With Recruiters

5. Operations as an afterthought

I outlined some of the top reasons Why Startups Desperately Need Techops. It is a repeating refrain. Priorities of a growing startup often involve taking on technical debt. But if that isn’t managed carefully you’ll run into some of the problems that Ward Cunningham Warns Us About.

Also: 5 Things Are Toxic To Scalability

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

25 Rumsfelds Rules for Startups

RumsfeldsRules

Join 12,100 others and follow Sean Hull on twitter @hullsean.

While we are still deep in the woods of a government shutdown, I thought it would be interesting to sum up some of our former Defense Secretary’s words of wisdom.

Rumsfeld may not have done everything right, but some of his quotes are priceless. What’s more they appeal to Startups quite nicely…

1 If you are working from your inbox, you are working on other people’s priorities.

Click To Tweet

2 Men count up the faults of those who keep them waiting.

Click To Tweet

3 In unanimity, there may well be either cowardice or uncritical thinking.

Click To Tweet

4 Test ideas in the marketplace. You learn from hearing a range of perspectives.

Click To Tweet

5. You can’t recover a fumble unless you’re on the field. Get out there.

Click To Tweet

Read this: Why the Twitter IPO mentioned scalability

6. First law of holes. If you get in one, stop digging.

Click To Tweet

7. Talent hits a target no one else can hit. Genius hits a target no one else can see.

Click To Tweet

8. You pay the same price for doing something halfway as for doing it completely so you might as well do it completely.

Click To Tweet

9. It is difficulties that show what men are.

Click To Tweet

10. What you measure, improves.

Click To Tweet

Also: Why I don’t work with recruiters

11. If you are lost, “climb, conserve and confess”

Click To Tweet

12. It is not the strongest species that survives, nor the most intelligent, but the one most responsive to change.

Click To Tweet

13. Luck is what happens when preparation meets opportunity.

Click To Tweet

14. People don’t spend money earned by others with the same care that they spend their own.

Click To Tweet

15. Disagreement is not disloyalty.

Click To Tweet

Related: Why a CTO must never do this

16. A lie travels halfway around the world before the truth gets its shoes on.

Click To Tweet

17. It is easier to convince someone they’re right, than to convince them they’re wrong.

Click To Tweet

18. Your best question is often why.

Click To Tweet

19. Simply because a problem is shown to exist, it doesn’t necessarily follow that there is a solution.

Click To Tweet

20. The world is run by those who show up.

Click To Tweet

Read: Who is Sean Hull?

21. Don’t panic. Things may be going better than they seem from the inside.

Click To Tweet

22. Know that the amount of criticism you receive may correlate closely to the amount of publicity you get.

Click To Tweet

23. Sunshine is a weather report, a flood is news.

Click To Tweet

24. If you expect people to be in on the landing, include them in the takeoff.

Click To Tweet

25. If a problem cannot be solved, enlarge it.

Click To Tweet

Read this: Why a killer title can make or break your content efforts

Get more in your inbox: Exclusive monthly Scalable Startups. We share tips and special content. Here’s a sample

5 conversational ways to evaluate great consultants

Startups and more mature businesses alike, and those large and small, at some point will need to hire a consultant or two. Want to get the best bang for your buck? Ask some tough questions!

Join 8000 others and follow Sean Hull on twitter @hullsean.

1. Make sure they’re not going to quit

I’ve heard so many crash and burn stories over the years, it makes my head spin.

One client had hired a consultant who was supposed to be the best in NYC. After only a few weeks of working with the client, he explained that they were “doing it all wrong”. Furthermore he had a travel schedule to meet, with speaking engagements in South America.

So he basically dropped them! As the client retold this story to me, I wonder if they could hear my head shaking, I was stunned. Who would turn away from a customer in pain? And furthermore turn away from one who could clearly use the expertise of someone who had seen a lot before!?

Keep in mind the reasons why people leave consulting.

2. Be sure they have some war stories to tell

Any consultant who’s been in business for a while, surely has some good war stories to tell. Talk about those, and find out the battles they’ve been in.

I can tell a few myself. In one case I was only 12 hours from leaving for summer vacation when a long time colleague and former client called me. They were in a serious emergency. The big boys, the remote DBAs that many in the industry use, had broken their database. Replication was misconfigured, and they were running blind. I ended up on a SKYPE call fixing the database and troubleshooting problems on Virgin’s inflight wifi. You don’t forget that kind of firefighting.

[quote]
Be sure they won’t quit, ask about war stories, test for some push back and be sure they empathize with your business pain. But more importantly ask them to tell their own business story. You’ll learn a lot.
[/quote]

For another client, back in the dot-com days, their application was stalling completely. Customers were leaving, and so was an 80 million dollar buyout deal. Nothing a few Oracle parameters couldn’t fix!.

And there are always a few tales of woe between sales teams, and the engineers that then must deploy solutions in the real world. Beware the sales wolf in sheeps clothing and do your homework aka due diligence on technical solutions.

3. Ensure they can provide resistance & push back

Good consultants have to walk a tightrope all the time. On the one hand they are tasked with making their clients happy. At the end of the day, improving their position, business bottom line, yes these are crucial. Sure that means saying yes, that means trying to be a problem solver as much as possible.

But always saying yes, avoids hard truths that you have to share. I had one client whose primary Oracle dba went on vacation. Before he left we reviewed systems. Multi-master replication in Oracle is brittle by nature. We both agreed. And I agreed not to change the configurations lest it break other things down the line. Not one week into his vacation a mandate comes back from on high, this change absolutely has to happen now. There are millions of dollars at stake!

Applying strong resistance is necessary to avoid breaking something even bigger. And it was not easy to stand strong in the face of such pressure. But I assured the team that such changes would mean even bigger problems for the company.

4. Find out if they empathize with your pain

In one of the biggest ironic twists, consultants should understand the pain of building a business. Because they themselves have experienced when clients don’t pay so they understand cash flow problems themselves. That is what every small business struggles with, and most startups too!

5. Ask them how they built their business

For me, one of the least appreciated things about independent consulting is, that in the most important ways, it is about running a small business. So someone who has built and kept running a freelance or consulting business knows how to make hard decisions, and keep their eyes on the ball.

A consultant needs to know how to get business first and foremost. But then how to manage engagements carefully. Once you’ve got those two down, keep building your business incrementally.

Someone who has successfully run a real business for years can share the story of what they’ve done. What has worked, what hasn’t worked, how they have pivoted when necessary, how they have failed fast, and moved through it.

They can tell you how they stay cash flow positive, can deliver on time, can be likeable and communicate with teams, and really understand every side of a business.

Get some in your inbox: Exclusive monthly Scalable Startups. We share tips and special content. Here’s a sample

NYC Tech Firms Are Hiring – Map

Made In NY - Startups Hiring

If you haven’t noticed how much the NYC tech scene has grown recently, I’m afraid you’ve been hiding under a rock. It’s simply incredible.

Take a look at Mapped In NY a google maps mashup of the growing list popularized by the NY Tech Meetup called Made In New York.

Join 5000 others and follow Sean Hull on twitter @hullsean.

[mytweetlinks]

Having been around during the first dot-com boom back in the late 1990′s this is even more exciting to see. Despite the recession, New York’s economy is truly thriving!

[quote]
New York’s Startup scene is truly thriving with a whopping 1263 firms, many of which are hiring.
[/quote]

Why is database administration talent in short supply? They are the Mythical MySQL DBAs

Also take a look at: Why Generalists are Better at Scaling the Web

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Here’s a sample