Category Archives: CTO/CIO

5 cloud ideas that aren’t actually true

storm coming

Join 20,000 others and follow Sean Hull’s scalability, startup & innovation content on twitter @hullsean.

Cloud computing is heralding us into a wonderful era where computing can be bought in small increments, like a utility. This changes the whole way we plan, manage budgets, and accelerates startups making them more agile.

But it’s not all wine & roses up there. I’ve heard a few refrains from clients over the years, and thought I’d share some of the most common.

1. Scaling is automatic

Rather recently I was working with a client on building some sophisticated reports. They needed to slice & dice customer data, over various time series, and summarize with invoices & tracking data. Unfortunately their dataset was large, in the half terabyte range.


Client: Can we just load all this data into the cloud?
Me: Yes we can do that. Build a system in Amazon public cloud, can support large datasets.
Client: I want it to scale easily. So we won’t have these slow reports. And as we add data, it’ll just manage it easily for us.
Me: Well it’s a little bit more complicated than that, unfortunately.

Unfortunately this is a rather familiar conversation that I have quite often. A lot of the press around cloud scalability, centers around auto-scaling, Amazon’s renowned & superb virtualization feature. Yes it’s true you can roll out webservers to scale out this way, but that’s not the end of the story. Typically web applications have a lot of components, from caching servers, to search servers, and of course their backend datastore.

But can we scrap our relational database, such as MySQL and go with one that scales out of the box like Riak, Cassandra or Dynamodb?

Those NoSQL solutions are built to be distributed from the start, it’s true. And they lend themselves to that type of architecture. However, if you’ve built up a dataset in MySQL or Oracle, and more so an application around that, you’ll have to migrate data into the NoSQL solution. That process will take some time.

Like teaching a fish to fly, it make take some time. They do well in water, but evolution takes a bit longer.

Related: RDS or MySQL 10 use cases

2. Disaster recovery is free

In the traditional datacenter, when you want DR, you setup a parallel environment. Hopefully not in the same room, same city or same coast even. Preferrably you do so in a different region. What you can’t get around is dishing out cash for that second datacenter. You need the servers, just in case.

In the cloud, things are different. That’s why we’re here, right? In amazon you have regions already setup & available for plugin-n-play use. Setup your various components, servers, software & configure. Once you’ve verified you can failover to the parallel environment you can just turn off all those instances. Great, no big charges for all that iron that you’d pay for to keep the rooms warm in an old-school datacenter. Or do you?

As it turns out, since you don’t have this environment running all the time, you’ll want to test it more often, run fire drills to bring the servers back online. That’ll incur some costs in terms of manpower. You’ll also want to include in there some scripts to start those servers up, and/or some detailed documentation on how to do that. And don’t lose that documentation, either will you?

You may also want to build some infrastructure as code unit tests. Things change, code checkouts evolve, especially in the agile & continuous integration world. Devops beware!

Read this: Why a killer title can make or break your content efforts

3. Machines are fast

Fast, fast, fast. That’s what we expect, things keep getting faster, right? Hard to believe then that the world of computing took a big step backward when it jumped into the cloud. Something similar happened when we jumped to commodity Linux a decade ago.

In amazon, it’s a multi-tenant world. And just like apartment buildings, popular restaurants, or busy highways you must share. When things are quiet you may have the road to yourself, but it’ll never be as quiet as a dirt road in the country!

Amazon is making big strides though. They now offer memory optimized & storage optimized instances. And an even bigger development is the addition of the most important feature for performance & scalability. That said the network & EBS can still be a real bottleneck.

Also: What is a relational database & why is it important?

4. Backups aren’t necessary

I’ve experienced a few horror stories over the years. I wrote about one noteworthy one When fat fingers take down your business.

True EBS snapshots make backing up your whole server, well a snap! That said a few extra steps have to happen (flush the filesystem & lock tables) to make this work for a relational database like MySQL or Oracle. And suddenly you have a verification step that you also need to perform. You see no backups are valid until they’ve been restored, remember?

But even with these wonderful disk snapshots, you’ll still want to do database dumps, and perhaps table dumps. Operator error, deleting the wrong data, or dropping the wrong tables, will always be a risk. Ignore backups at your own peril!

Check this: Why CTOs underestimate operational costs

5. Outages won’t happen

In an ideal world, everything is redundant, and outages will be a thing of the past. We’ll finally reach five nines uptime and devops everywhere will be out of work. :)

It’s true that Amazon provides all the components to build redundancy into your architecture, and very cutting edge firms that have taken netflix’s approach with chaos monkey are seeing big improvements here. But AirBNB did fail and at root it was an Amazon outage that shouldn’t ever happen.

Read: Why Oracle won’t kill MySQL

Get more. Monthly insights about scalability, startups & innovation.. Our latest Are SQL Databases Dead?

Why managers & CTO’s underestimate operational costs

too much inventory

Join 19k others and follow Sean Hull on twitter @hullsean.

1. Technology choices & talent shortage

I worked at one firm evaluating their technology stack. When we got to the programming language, I paused in my tracks. “Haskell” I asked? “Oh you haven’t heard of it? It’s a really cool functional programming language, and we found it had some cool features that we really wanted to use”.

I had to fight the urge to roll my eyes. Yes I’d heard of the language, sitting in the club with scheme, lisp & prolog, you study them at university. They’re certainly an interesting bunch and to be sure, can do some things that imperative programming languages can’t. But did it belong in the stack of this run-of-the-mill internet startup?

In this case the developers had full reign to choose any technologies they liked, adding more & more to the mix almost daily. But what are some of the ramifications here?

Two years, three years, or five years down the line, this team will be long gone, and another team will be picking up the pieces. Will you as a manager be able to find a lot of Haskell experts? What’s more operationally will you be able to support those choices? Will updates be made often enough to have a secure stack for years to come?

Also: 5 things toxic to scalability

2. Scalability & server costs

Server costs are easier than ever to estimate. Build your application to serve your first 10,000 customers on Amazon with a couple webservers and a database server. Growing 100x to a million customers, just vertically scale your db, scale out your webservers and you’re good. Or are you?

What happens when you hit a wall? Did you build your application on ORM technology or take on technical debt? I’ve seen firm after firm struggle with technologies like hibernate, eating up precious resources, and being helpless to eliminate the problem. Tread carefully on these types of questions.

Related: Why you’re not hitting five nines uptime

3. Patching, fixing bugs & managing security

Another long term cost of an application will be minor repairs and bug fixes. Those might appear in a slow steady trickle over the years, but security may loom larger. Cross-site scripting, SQL injection and many other threats can be a real headache.

What’s more fixes may involve the libraries your application sits on top of. And when they are upgraded, your application will require tweaks too. It’s all basic stuff when you’re knee deep in development, but when your application has been deployed, the original team is long gone, and you’re supporting it years later, it can surely get messy.

Read: The four-letter-word dividing dev & ops

4. missing operational switches

When building a web application, all eyes are on features. Which ones to include, and which are a priority. Pressure is heavy to build functions that can be sold to customers. Pleasing customers is of obvious importance.

So it’s no surprise that backend switches are often missing. But they can be a real boon for operations team. Suppose you roll out a new feature to support star-ratings on certain pieces of content. An operational switch can be built to allow that feature to be disabled as necessary. If the site is loaded, or trouble is brewing, you may desperately want some switches to disable parts of the site, without the whole thing going down. I talk about this in AirBNB didn’t have to fail.

Another useful thing is a browse only mode. This allows your site to operate, even when writing to the database is not possible. If you’ve ever tried to update on a social network like twitter, facebook or instagram, perhaps late and nite and gotten a “please try again later” message, you’ll understand the value. Here users can’t make changes, but otherwise the site appears to be working, and browsing works normally.

Check this: Are SQL Databases Dead?

5. Consider bitcoin

Mt. Gox, the Japanese exchange handling bitcoin failed in a spectacular fashion. 500 million of the digital currency was stolen. And what’s more since it’s all frictionless currency, untraceable, there’s no marked bills to try and track down. Ooops!

How does this relate to operational costs? The failure was squarely with the operations department. Functionally the site worked fine. But security wasn’t handled well enough, intrusion detection wasn’t employed, and “unspecified weaknesses” were to blame.

Security is one of those things that can be ignored without pain. Until something goes wrong. What’s more if it is being handled well, it’s invisible, and unappreciated besides.

Read this: Why Oracle won’t kill MySQL

Get more. Grab our exclusive monthly Scalable Startups. We insights on scalability, startups & innovation. Our latest Why I don’t work with recruiters

Why I can’t raise the bar at every firm

Screen-shot-2012-08-02-at-1.28.35-PM

Join 17,000 others and follow Sean Hull on twitter @hullsean. Also check out: Who is Sean Hull?

It may seem counterintuitive. If I am not the best solution provider, why on earth would I highlight it?

I believe by pointing those cases out, I also underline the clients and problems that I’m particularly well suited too, and for which I can really provide value. Read on!

1. People Problems

Sometimes, you’re hired to solve a particular problem which is framed as a technical one. Some process needs to be reworked, recoded or retooled. It’s framed as a technical problem, yet as things unfold the client already has the expertise in-house to solve & write the code. What then?

It may be that the right people aren’t communicating, project managers aren’t seeing the issues, or part of the human systems are gummed up. We can’t raise the technical bar, but we can help getting those folks talking.

I wrote about this before in When You’re Hired to Solve a People Problem.

Also: Why are oil spills & financial instability related to datacenter outages?

2. ORM Usage & Technical Debt

If you’ve read my blog you know I am not very fond of Object Relational Modelers.

I would also argue as Ward Cunningham does so elloquently that technical debt can be a real and pressing problem.

Here we would help identify and frame the problem, though the work of raising the bar technically involves the longer process of retooling & refactoring your code base.

Related: Why database choices are tricky

3. Where Commodity & Offshore Works

Some firms are already making use of odesk or offshoring resources, where you might pay as little as $150/day. If you have a very technical manager or CTO, such a solution may work well for you.

At the other end of the spectrum are the high priced senior consultants from firms like Oracle, Percona or Pythian. Yes they may set you back as much as $3500/day.

In those cases a scalability & performance review may make sense. Here’s how.. Although specialists are necessary, remember to ask yourself Why generalists are better at scaling the web.

We sit in the sweet spot between the two options. With low overhead, our prices are more affordable. At the same time you’re getting a whole lot more than a commodity solution. We’ll communicate in plain language with folks at every technical level. And for many firms that in itself is a value add.

Check this: Does Oracle Aim to Kill MySQL?

4. Existing team did their homework

Believe it or not, I’ve gone into consulting engagements where the existing team has really really done their homework.

In those cases it becomes much harder to raise the bar technically. In those cases I can help when existing team missed something. But more importantly, I can validate a correct setup, or identify technical debt.

Having an outside perspective then, can provide reassurance. As I see ten to fifteen new environments per year, I’ve seen hundreds in the past decade & a half. That’s helpful perspective in itself.

Read: Does Oracle Aim to Kill MySQL?

5. Availability & Uptime Are Already High

I wrote in depth about high availability in the Myth of Five Nines

At the end of the day, availability can only approach perfection, not actually reach it. That’s a property of complex systems. If your uptime is already extremely high, again we can validate your environment, review and provide & summarize findings. But we may not be able to raise the bar.

If that’s you, it’s a good problem to have!

Also: Why AirBNB Didn’t Have to Fail

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Why Scalability Is Big Business

Russian_Dolls

Join 16,500 others and follow Sean Hull on twitter @hullsean.

1. Complexity Is Growing

Despite automation & the mass migration to the cloud, or perhaps because of it, complexity continues to grow. Back in the dot com era a typical infrastructure included a load balancer, a couple web servers, one oracle database, and that was pretty much it.

Now that has multiplied. Pile on top of that three to five more webservers, a search server, a page cache, an object cache, one or more slave databases and more. You may have a utility server with jenkins for continuous automation, monitoring applications like nagios and cacti, your source code repository and perhaps configuration management like Puppet or Chef.

That’s not only more moving parts, it’s a wider swath of skills and technologies to understand. That’s one reason Generalists Are Better At Scaling The Web.

Also: Are SQL Databases Dead?

2. Developer Mandate: Features

The pressure to build features that can directly be monetized is obvious. Startups especially have the pressure to grow fast and grow now. So security, technical debt, and scalability often take a back seat. What’s more in small scrappy and lean startups, ops sometimes falls on the shoulders of one competent but overworked developer.

Related: Why Oracle Won’t Kill MySQL

3. Startups Growing Pains

With hyper growth, startups can go from 100 customers to millions overnight. That kind of popularity is a good problem to have. But if your app hits a wall and suddenly falls over, everyone is scrambling. The pressure builds, as fear of losing that traction mounts, and heads are put on the chopping block.

Read: AirBNB Didn’t Have To Fail

4. Missing Browse-only Mode & Feature Flags

Ever been browsing for airline tickets, then go to order and get an error? Try again later? If so you’re familiar with a browse-only mode. This is a very powerful addition to any web application but is very often left out. Some mistakenly believe it won’t work for their application, as users will always be changing data.

Ever visited a website that has star ratings, only to find them missing? Or temporarily unable to edit your rating for a piece of content? This amounts to what’s called a feature flag. These powerful switches give operations teams the ability to disable heavy features, while the side is under tremendous load. They can take a huge burden off the shoulders of your servers when you hit that scalability cliff.

Check this: Why I Don’t Work With Recruiters

5. Operations as an afterthought

I outlined some of the top reasons Why Startups Desperately Need Techops. It is a repeating refrain. Priorities of a growing startup often involve taking on technical debt. But if that isn’t managed carefully you’ll run into some of the problems that Ward Cunningham Warns Us About.

Also: 5 Things Are Toxic To Scalability

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

25 Rumsfelds Rules for Startups

RumsfeldsRules

Join 12,100 others and follow Sean Hull on twitter @hullsean.

While we are still deep in the woods of a government shutdown, I thought it would be interesting to sum up some of our former Defense Secretary’s words of wisdom.

Rumsfeld may not have done everything right, but some of his quotes are priceless. What’s more they appeal to Startups quite nicely…

1 If you are working from your inbox, you are working on other people’s priorities.

Click To Tweet

2 Men count up the faults of those who keep them waiting.

Click To Tweet

3 In unanimity, there may well be either cowardice or uncritical thinking.

Click To Tweet

4 Test ideas in the marketplace. You learn from hearing a range of perspectives.

Click To Tweet

5. You can’t recover a fumble unless you’re on the field. Get out there.

Click To Tweet

Read this: Why the Twitter IPO mentioned scalability

6. First law of holes. If you get in one, stop digging.

Click To Tweet

7. Talent hits a target no one else can hit. Genius hits a target no one else can see.

Click To Tweet

8. You pay the same price for doing something halfway as for doing it completely so you might as well do it completely.

Click To Tweet

9. It is difficulties that show what men are.

Click To Tweet

10. What you measure, improves.

Click To Tweet

Also: Why I don’t work with recruiters

11. If you are lost, “climb, conserve and confess”

Click To Tweet

12. It is not the strongest species that survives, nor the most intelligent, but the one most responsive to change.

Click To Tweet

13. Luck is what happens when preparation meets opportunity.

Click To Tweet

14. People don’t spend money earned by others with the same care that they spend their own.

Click To Tweet

15. Disagreement is not disloyalty.

Click To Tweet

Related: Why a CTO must never do this

16. A lie travels halfway around the world before the truth gets its shoes on.

Click To Tweet

17. It is easier to convince someone they’re right, than to convince them they’re wrong.

Click To Tweet

18. Your best question is often why.

Click To Tweet

19. Simply because a problem is shown to exist, it doesn’t necessarily follow that there is a solution.

Click To Tweet

20. The world is run by those who show up.

Click To Tweet

Read: Who is Sean Hull?

21. Don’t panic. Things may be going better than they seem from the inside.

Click To Tweet

22. Know that the amount of criticism you receive may correlate closely to the amount of publicity you get.

Click To Tweet

23. Sunshine is a weather report, a flood is news.

Click To Tweet

24. If you expect people to be in on the landing, include them in the takeoff.

Click To Tweet

25. If a problem cannot be solved, enlarge it.

Click To Tweet

Read this: Why a killer title can make or break your content efforts

Get more in your inbox: Exclusive monthly Scalable Startups. We share tips and special content. Here’s a sample

5 conversational ways to evaluate great consultants

Startups and more mature businesses alike, and those large and small, at some point will need to hire a consultant or two. Want to get the best bang for your buck? Ask some tough questions!

Join 8000 others and follow Sean Hull on twitter @hullsean.

1. Make sure they’re not going to quit

I’ve heard so many crash and burn stories over the years, it makes my head spin.

One client had hired a consultant who was supposed to be the best in NYC. After only a few weeks of working with the client, he explained that they were “doing it all wrong”. Furthermore he had a travel schedule to meet, with speaking engagements in South America.

So he basically dropped them! As the client retold this story to me, I wonder if they could hear my head shaking, I was stunned. Who would turn away from a customer in pain? And furthermore turn away from one who could clearly use the expertise of someone who had seen a lot before!?

Keep in mind the reasons why people leave consulting.

2. Be sure they have some war stories to tell

Any consultant who’s been in business for a while, surely has some good war stories to tell. Talk about those, and find out the battles they’ve been in.

I can tell a few myself. In one case I was only 12 hours from leaving for summer vacation when a long time colleague and former client called me. They were in a serious emergency. The big boys, the remote DBAs that many in the industry use, had broken their database. Replication was misconfigured, and they were running blind. I ended up on a SKYPE call fixing the database and troubleshooting problems on Virgin’s inflight wifi. You don’t forget that kind of firefighting.

[quote]
Be sure they won’t quit, ask about war stories, test for some push back and be sure they empathize with your business pain. But more importantly ask them to tell their own business story. You’ll learn a lot.
[/quote]

For another client, back in the dot-com days, their application was stalling completely. Customers were leaving, and so was an 80 million dollar buyout deal. Nothing a few Oracle parameters couldn’t fix!.

And there are always a few tales of woe between sales teams, and the engineers that then must deploy solutions in the real world. Beware the sales wolf in sheeps clothing and do your homework aka due diligence on technical solutions.

3. Ensure they can provide resistance & push back

Good consultants have to walk a tightrope all the time. On the one hand they are tasked with making their clients happy. At the end of the day, improving their position, business bottom line, yes these are crucial. Sure that means saying yes, that means trying to be a problem solver as much as possible.

But always saying yes, avoids hard truths that you have to share. I had one client whose primary Oracle dba went on vacation. Before he left we reviewed systems. Multi-master replication in Oracle is brittle by nature. We both agreed. And I agreed not to change the configurations lest it break other things down the line. Not one week into his vacation a mandate comes back from on high, this change absolutely has to happen now. There are millions of dollars at stake!

Applying strong resistance is necessary to avoid breaking something even bigger. And it was not easy to stand strong in the face of such pressure. But I assured the team that such changes would mean even bigger problems for the company.

4. Find out if they empathize with your pain

In one of the biggest ironic twists, consultants should understand the pain of building a business. Because they themselves have experienced when clients don’t pay so they understand cash flow problems themselves. That is what every small business struggles with, and most startups too!

5. Ask them how they built their business

For me, one of the least appreciated things about independent consulting is, that in the most important ways, it is about running a small business. So someone who has built and kept running a freelance or consulting business knows how to make hard decisions, and keep their eyes on the ball.

A consultant needs to know how to get business first and foremost. But then how to manage engagements carefully. Once you’ve got those two down, keep building your business incrementally.

Someone who has successfully run a real business for years can share the story of what they’ve done. What has worked, what hasn’t worked, how they have pivoted when necessary, how they have failed fast, and moved through it.

They can tell you how they stay cash flow positive, can deliver on time, can be likeable and communicate with teams, and really understand every side of a business.

Get some in your inbox: Exclusive monthly Scalable Startups. We share tips and special content. Here’s a sample

NYC Tech Firms Are Hiring – Map

Made In NY - Startups Hiring

If you haven’t noticed how much the NYC tech scene has grown recently, I’m afraid you’ve been hiding under a rock. It’s simply incredible.

Take a look at Mapped In NY a google maps mashup of the growing list popularized by the NY Tech Meetup called Made In New York.

Join 5000 others and follow Sean Hull on twitter @hullsean.

[mytweetlinks]

Having been around during the first dot-com boom back in the late 1990′s this is even more exciting to see. Despite the recession, New York’s economy is truly thriving!

[quote]
New York’s Startup scene is truly thriving with a whopping 1263 firms, many of which are hiring.
[/quote]

Why is database administration talent in short supply? They are the Mythical MySQL DBAs

Also take a look at: Why Generalists are Better at Scaling the Web

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Here’s a sample

Don't Miss Percona Live 2013

The biggest event on the MySQL calendar is the yearly Percona Live and it’s just around the corner.

This year you’ll be able to pick from a whopping 110 technical sessions by 90 different speakers from companies like Facebook, Amazon, Google and Linkedin. Learn what’s happening at the cutting edge of open source database deployments. Besides the technical sessions, there’s plenty of hobnobbing with fellow DBAs, developers, and industry folks. A great opportunity all around.

It’s just a couple of weeks away, so don’t waste any time. You can register here and enjoy a 20% discount code “SH-Live”!

Get to it!

Want more? Grab our Scalable Startups monthly for more tips and special content. Here’s a sample

When You Have to Take the Fall

Also find Sean Hull’s ramblings on twitter @hullsean.

One of the biggest jobs in operations is monitoring. There are so many servers, databases, webservers, search servers, backup servers. Each has lots of moving parts, lots that can go wrong. Typically if you have monitoring, and react to that monitoring, you’ll head off bigger problems later.

A problem is brewing

We, myself & the operations team started receiving alerts for one server. Space was filling up. Anyone can relate to this problem. You fill up your dropbox, or the drive on your laptop and all sorts of problems will quickly bubble to the surface.

Also check out – Why generalists are better at scaling the web.

As we investigated over the coming days, a complicated chain of processes and backups were using space on this server. Space that didn’t belong to them.

Dinner boils over

What happened next was inevitable. The weekly batch jobs kicked off and failed for lack of space. Those processes were not being monitored. Business units then discovered missing data in their reports and a firestorm of emails ensued.

Hiring? Get our MySQL DBA Interview Guide for managers, recruiters and candidates alike.

Why weren’t these services being monitored, they wanted to know.

Time to shoot the messenger

Having recently seen a changing of the guard, and a couple of key positions left vacant, it was clear that the root problem was communication.

Looking for talent? Why is it so hard to find a mythical MySQL DBA or devops expert these days?

I followed up the group emails, explaining in polite tone that we do in fact have monitoring in place, but that it seemed a clear chain of command was missing, and this process fell through the cracks.

I quickly received a response from the CTO requesting that I not send “these types of emails” to the team and to direct issues directly to him.

You might also like: A CTO Must Never Do This

A consultants job

As the sands continued to shift, a lead architect did emerge, one who took ownership of the products overall. Acting as a sort of life guard with a higher perch from which to watch, we were able to escalate important issues & he would then prioritize the team accordingly.

Are you a startup grappling with scalability? Keep in mind these 5 things toxic to scalability

Sometimes things have to break a little first.

What’s more a consultants job isn’t necessarily to lead the pack, nor to force management to act. A consultant’s job is to provide the best advice possible & to raise issues to the decision makers. And yes sometimes it means being a bit of a fall guy.

Those are the breaks of the game.

Want more? Grab our Scalable Startups monthly for more tips and special content. Here’s a sample

When Clients Don't Pay – Consulting War Stories

Also find Sean Hull’s ramblings on twitter @hullsean.

It’s a cold cold winter out there, so as they say you need money to keep the lights on? Yep, that’s true whether you’re a small business or a consultant. Everyone has to pay their bills. Or do they?

It’s an unfortunate fact of life in business, but sometimes there are differences. Disagreements about deliverables, timelines, milestones, and deadlines. But when all the work is done, there are still sometimes differences over dollars.

Over the years most of those I’ve managed to work out with clients, but there were a few that went sour. One case was with a large entertainment firm. The music business is one that I hadn’t interacted with much before. For these guys I’d done a few days work in the past, and was paid promptly. Now they were in a bit of a jam. They called me up and asked if I could help.

Related article: A CTO Must Never Do This.

These types of emergencies often come at the worst times, and I explained that I was already juggling a few other things. They pleaded for help, and I relented. I carved out a full day of time for them, explaining the day rate and so forth. While on the phone though, I expressed caution.

[quote]I understand that your issue is urgent now, what happens if your needs change in the next 48 hours, I asked? Not a problem they replied, we can use your help anyway, so we’ll book you either way. Fast forward 48 hours, work canceled & client won’t take my call.[/quote]

Great I thought, verbal assurances. That works for me, I thought.

Fast forward 32 hours, and I receive an email saying the problem is resolved, and offering a “kill fee” of which I knew nothing, and which was never negotiated or discussed.

Along comes the day of reconning, and I call the client. They don’t take my call. Shortly there after I receive additional emails. I reply and explain we should talk on the phone. Still the client can’t find the time to pickup the call.

So I judiciously put together an invoice for the work. I email it directly to finance, and CC all parties. From there I get responses ranging from disinterest to denial. Over the coming months I periodically resend the invoice, but to no avail.

Or was it? I actually feel that this experience is to great avail.

1. Small disagreements foreshadow larger ones down the road
2. A relationship between client and vendor is a mutual one. If parties can only pickup the phone when they need something, then things are out of balance already.

Sure I lost a day & the fee associated. But I gained a lesson.

Patience and polite persistence

I firmly believe that being patient and persistent wins in the end. Sometimes clients have hiccups in payroll or budgets. Keep communication lines open.

Freelancers & consultants: Grab my Consulting 101 Guide.

Appealing to fairness

If you maintain a healthy relationship with your client, then appeals to fairness are normally heeded.

Appealing to promises

Emails are important to keep a paper trail of agreements. Communicate clearly and often so you know when you get derailed, and can refer back to what was agreed previously.

Popular: AirBNB didn’t have to failed – AWS hosting outages.

Setting precedents & expectations up front

This is an important one, that freelancers and consultants alike sometimes forget. Setting and agreeing on expectations is key. Often details are in the fine print or left out completely. So ongoing communication can iron out those differences or bring them to light.

Want to hire the best? Read our DBA Interview Guide.

Getting a protective deposit

If you haven’t worked together before, a deposit makes a lot of sense. Executing on this is more than a show of faith. It underlines that accounts payable is on board with your hiring, and you are now in the payments system.

Related: Hiring is a numbers game

Sizing each other up

Websites provide the first representation of you and your client to each other. How you carry yourself and how they feel meeting them face to face is important as well. Ideally you’ll meet each other at the client’s offices, where you may meet others on the team, and put names to faces. At the very least a skype call will go a long way as well.

Managing spend – communication along the way

Keep a close eye on invoices. If a client is getting behind, resolve it before continuing to work. Deadlines are mutual in a business relationship. Yours to complete work by a certain date, and theirs to pay by an equally agreed upon time.

Read also: Real Disaster Recovery Lessons from hurricane Sandy.

Never go to court – defer to the handshake & gentleman’s agreement

I know the lawyers out there may think I’m naive. But I’ve been in business a long time, and I believe a handshake means and says a lot. Also common sense language & contracts, in the form of emails and so forth are better than heavily legalese ones that no one but a lawyer can understand.

What’s more going to court has a huge cost in time & stress. Don’t go that route. I also think it appeals to clients knowing that you’re not the litigious sort.

Want more? Grab our Scalable Startups monthly for more tips and special content. Here’s a sample