Category Archives: iHeavy Newsletter

MySQL for Devs, DBAs and Debutantes

Join 6500 others and follow Sean Hull on twitter @hullsean.

I just received my copy of the 5th Edition of Paul DuBois’ MySQL tomb. Weighing in at 1153 pages, it’s a solid text, with a very thorough introduction to the topic of administering MySQL databases.

Buy the book here: MySQL 5th Edition by Paul Dubois

A book for a broad audience

When I say debutantes, it’s a nod to beginners, for this book forges a very solid and complete introduction to the topic of MySQL. Start with installing the software & setting up your environment, and then move on to really understanding the SQL language, from commands to create objects, to ones for adding & modifying data, and then writing code around it.

See also: 5 more things deadly to Scalability

There’s a thorough discussion of datatypes, stored procedures, functions and views.

[quote]
Paul Dubois’ definitive reference makes a excellent compliment to High Performance MySQL. They should sit alongside eachother on your database bookshelf.
[/quote]

For developers there are chapters on writing applications in C, another for Perl and a third for PHP.

For DBAs there are chapters on security, backups, replication, understanding the data directory and general server administration. There is also good coverage of both 5.5 and the newly released 5.6 of MySQL.

What I like about this book

You can think of this book as a definitive reference to MySQL. It includes much of the online documentation that you would find at Oracle’s site, such as command & variable reference, and detailed explanation of how to use the client tools.

Dubois also goes beyond the online documentation though, giving you a bit of a background around concepts, a broader more complete discussion.

Read this: Two Part DBA Interview Guide for Managers & Candidates alike

He also lays out the material in a very logical stepwise way, so for someone new to the MySQL world and the time on their hands, the 1153 pages could be read straight through.

Why No Mention of Percona Toolkit?

I have to admit I was a bit surprised there was no mention of Percona Toolkit. Perhaps it was buried in some dark corner of the text I missed, but it made no mention in the index at all.

Percona Toolkit of course is a tool that every DBA should be familiar with. It is really an essential toolkit and fills the gaps that the prepackaged tools can’t help you with.

Want to checksum your tables to compare data on master & slave? pt-table-checksum does the trick.

Check this: AirBNB didn’t have to fail during the Amazon AWS Outage

Want to find out how far your slaves *really* are behind? pt-heartbeat is your friend.

Want to analyze your slow query log to produce a useful summary report? pt-query-digest to the rescue.

I also see no mention of innotop, which I would also say is an essential tool. These aren’t really advanced topics, so It’s unclear why they are missing. In the real world you need these tools to do your job.

Other Criticisms

My more general criticism is where the book lacks real-world advice from a seasoned DBA. At times the writing feels a bit more of the official line on how things work. But in day-to-day devops and operations, things can be quite different.

Also: Bulletproofing MySQL Replication with Checksums

For example, stored procedures. In MySQL they are there, however using them brings real performance challenges. They’re not always compatible with replication. Given all of that, why include a whole chapter with endless discussion of them without strong reservations. It would lead a novice user or developer to incorporate them into an application only to be shocked and surprised at the problems they bring.

Another example, looking through the system variables reference, I see the sync_binlog option. There is a short caution “…lower values provide greater safety in the event of a crash, but also affect performance more adversely”. Now reading this as a novice DBA I might think great, crash protection. But having tried this parameter in production, I found a huge impact on performance and had to disable it. What’s the advice here? It’s a bit confusing.

Conclusions

This is a really great book as an introduction to MySQL, and delving into intermediate topics. I would sit it on your bookshelf along side High Performance MySQL. What this book lacks in advice, you can turn to the latter book, and what High Performance MySQL lacks in terms of introductory material this book covers in spades. They make a great compliment to each other.

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Here’s a sample

Five More Things Deadly to Scalability

The.Rohit - Flickr

The.Rohit – Flickr

Join 19k others and follow Sean Hull on twitter @hullsean.

1. Slow Disk I/O – RAID 5 – Multi-tenant EBS

Disk is the grounding of all your servers, and the base of their performance. True with larger and larger main memory, much is available in cache, a server still needs to constantly read from disk and flush things from memory. So it’s a very very important component to performance and scalability.

What’s wrong with Raid 5?

Raid 5 was designed to give you more space, using fewer disks. It’s often used in a server with few slots or because ops misunderstand how bad it will impact performance. On a database server it can be particularly bad.

All writes see a performance hit. What’s worse is if you lose a disk, the RAID though technically still on line, will perform SO SLOWLY as to be offline. And a rebuild takes many hours. Worse still is the risk to lose another drive during that rebuild. What if you have order a drive and it takes a couple of days?

RAID 10 is the solution. Mirror each set of two drives, then stripe over those. Even with only four slots available, it’s worth it. Good read performance, good write performance, and protection.

What the heck is multi-tentant?

In the cloud, you share servers, network & disk just like you do apartments in a building. Hence the name. Amazon’s EBS or elastic block storage, extends this metaphor, offering you the welcome flexibility of a storage network. But your bottleneck can be fighting with other tenants on that same network.

Default servers do have this problem, however AWS has addressed this serious problem with a little known but VERY VERY useful feature called Provisioned IOPS. It’s a technical name, but means you can lock in reliable disk I/O. Just what the scalability doctor ordered.

Check out our original post: 5 Things Toxic to Scalability

2. Using the database for Queuing

MySQL is good at a lot of things, but it’s not ideal for managing application queues. Do you have a table like JOBS in your database, with a status column including values like “queued”, “working”, and “completed”? If so you’re probably using the database to queue work in your application.

It’s not a great use of MySQL because of locking problems that come up, as well as the search and scan to find the next task.

Luckily their are great solutions for developers. RabbitMQ is a great queuing solution, as is Amazon’s SQS solution. What’s more as external services they’re easier to scale.

[quote]
Scalability becomes key to your business, as you customer base grows. But it doesn’t have to be impossible. Disk I/O, caching, queuing and searching are all key areas where you can make a big dent, in a manageable way. Juggle your technical debt too, and you’re golden!
[/quote]

Also take a look at: Why Generalists are Better at Scaling the Web

3. Using Database for full-text searching

Oracle has full text search support, why shouldn’t we assume the same in MySQL? Well MySQL *does* have this, but in many versions only with the old MyISAM storage engine. It has it’s set of corruption problems, and isn’t really very performant.

Better to use a proven search solution like Apache Solr. It is built specifically for search, includes excellent library support for developers of most modern web languages and best of all is easy to SCALE! Just add more servers on your network, or distributed globally.

For folks interested in the bleeding edge, Fulltext is coming to Innodb crash safe & transactional storage engine in 5.6. That said you’re still probably better off going with an external solution like Solr or Sphinx and the MySQL Sphinx SE plugin.

[mytweetlinks]

How to find A Mythical MySQL DBA

4. Insufficient Caching at all layers

Cache, cache, and cache some more. Your webservers should use a solid memcache or other object cache between them & the database. All those little result sets will sit in resident memory, waiting for future web pages that need them.

Next use a page cache such as varnish. This sits in front of the webserver, think of it as a mini-webserver that handles very simple pages, but in a very high speed way. Like a pack of motorbikes riding down an otherwise packed freeway, they speed up your webserver to do more complex work.

Browser caching is also important. But you can’t get at your customers browsers, or can you? Well not directly, but you can instruct them what things to cache. Do that with proper expires headers. Have your system administrator configure apache to support this.

Also: Tweaking Disqus to Find Experts & Drive Traffic

5. Too much technical debt

Technical debt can bite. What is it? As you’re developing an new idea, you’ll build prototypes. As those get deployed to customers, change gets harder, and past things you glossed over because problems. One team leaves, and another inherits the application, and the problems multiple. Overtime you’re building your technical debt as your team spends more time supporting old code and fixing bugs, and less time building new features. At some point a rewrite of problem code becomes necessary.

Also: How I increased my blog pagerank to 5

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Here’s a sample

The Needle in Big Data Noise

Join 5500 others and follow Sean Hull on twitter @hullsean.

[mytweetlinks]

Also take a look at: I hacked Disqus Digests to discover new blogs

Who the heck is Bayes

Thomas Bayes was a scientist & thinker, Fellow of the Royal Society, and back in 1763 author of “An Essay toward Solving a Problem in the Doctrine of Chances”. His method advocated learning by approximation, to get closer and closer to the truth by gathering more information, and factoring that into probabilities & predictions.

[quote]
What isn’t acceptable under Bayes’s theorem is to pretend that you don’t have any prior beliefs. You should work to reduce your biases, but to say you have none is a sign that you have many.
[/quote]

Why should you care?

From hurricane & earthquake warnings, to financial storms or terrorism, prediction is more important than ever. Epidemiologists can make use of Bayesian techniques to protect populations, gamblers can use it in sport, and investors for markets.

See also Amp up your blog traffic by improving your pagerank

Why Nate Silver is different

Nate is famous for predicting the 2012 presidential election with uncanny accuracy. So the book is an in depth look at how he thinks, and how he works with data. He talks of Hedgehogs – those who believe in big ideas and work from large principals, versus Foxes who see the world as messy, often inconsistent and unpredictable, but who nevertheless tend to present better though less definitive predictions. The philosophy is less of modeling, and more of testing, and adjusting along the way to get closer to the truth.

See also Sales sucks, but a bear market offers hard lessons

For engineers & startups

Nate interviewed John Sanders of a scout for the LA Dodgers. He identified five abilities and characteristics that predict success in baseball. Looking at them together, I think they can well predict success in Startup land too.

1. Preparedness & work ethic
2. Concentration & focus
3. Competitiveness & self-confidence
4. Stress management & humility
5. Adaptiveness & learning ability

The book is a bit technical and sometimes long winded. But it is choc full of real insight, and wisdom that we can all put to use in our careers and businesses.

Also: AirBNB didn’t have to fail – AWS outages be damned!

Get a whole lot more! Scalable Startups. Exclusive. Monthly.. We share special content. Here’s a sample

Sales sucks, but then I learned

Are you a developer or startup entrepreneur? Have you ever been frustrated with some of the claims made by the sales team or lacked the patience or ability to communicate across departments?

Join 4000 others and follow Sean Hull on twitter @hullsean.

Just out of college

Just out of college I got a job as a Macintosh Software Developer for a small firm outside of University at Buffalo. It was a ten person company, and half of us were on the technology side of the house. I was doing C++ & Graphical Interface design & coding.

Why is it so hard to find operations & devops talent? Enter the Mythical MySQL DBA!.

Sales is “ahead” of engineering

Besides coding, I also fielded support calls from customers which brought me perspective on both what they wanted, and where they struggled with the software. Our app helped consumers and nutritionists track diet & exercise.

[quote]
The sales team made promises of technology the company wasn’t capable of delivering. Meanwhile the engineering team was sent scrambling to answer to those promises.
[/quote]

Soon I was fielding questions from customers asking when the new heart rate monitoring would be available. I followed up by talking with the team lead & chief architect. He had no plans of building such a feature, nor did we even know how it would be possible!

[mytweetlinks]

Searching for a database expert? MySQL DBA Interview Guide.

We checked in at our weekly meetings, and the CEO explained that the sales team was simply “ahead” of engineering. Years ahead apparently even of the technology that was possible at the time!

Fast forward 5 years to professional services

A half decade later I’m doing independent consultanting for dot-coms. Much of my business came from word of mouth. Helping a firm out of a pinch, speeding their site so they can handle 10x customers on the same servers and suddenly everyone is your friend!

Too many customers is a good problem to have right? For hyper growth companies there are 5 Things Toxic To Scalability .

But all is not smooth sailing in the freelance consulting world. The dot-com crash comes along and budgets are squeezed tighter. Business spend is reduced and every dollar is scrutinized. I learned to speak to prospects about savings and personalized service, advantages of lower overhead, and real return I could provide. At the end of the day if they’re not buying, your services aren’t worth their cost!

[quote]
The sales process should inform the business about what customers really want. In a successful startup there is communication back and forth with engineering and business units so all are working in harmony.
[/quote]

Full Circle

Now coming full circle I have a wide perspective on business. I understand the engineering fundamentals, and the limitations of technology. I also have a grasp of product, and how business units must manage the bottom line, and deliver to customers or else perish in the marketplace.

Looking for a top flight cloud engineer? Grab our Amazon EC2 Interview Guide.

For the two to achieve a happy marriage, you must bring a balance of execution & technical debt, with satisfying a real customer need in the marketplace. And therein lies the innovation & startup sweet spot!

You might also like our piece Why generalists are better at scaling the web

Like our content? We publish an exclusive monthly Scalable Startups where we share more tips and special content. Here’s a sample

Dinner, dollars & devlishly creative thinking

Efficiency at Dinner?

I just finished reading Tyler Cowen’s opus, An Economist Gets Lunch. I have to admit I’m already a fan of his writing, getting a daily dose on from his blog Marginal Revolution.

Join 4000 others and follow Sean Hull on twitter @hullsean.

What I like about this book is that it is unconventional by definition. Further economists like scalability engineers like to think about efficiency. How can I squeeze out more from less? Like the business question how do I get better ROI or more bang for my buck? Questions spring to mind like – What does an economist know about food? Or – What does food eating have to do with economics? Well on both points you’ll get some surprising answers.

Hiring or job seeking? Check out our MySQL DBA Interview questions which is useful to managers, candidates and human resources alike.

To the former question, Cowen has some really good insight because he brings the fresh perspective of an economist. His sort of mantra throughout the book is:

[quote]
Food is a product of economic supply and demand, so try to figure out where the supplies are fresh, the suppliers are creative and the demanders are informed.
[/quote]

Economists & engineers talk shop

What about the second question, how is the food we eat related to economics? Further does it have an impact on environmental and energy consumption questions? As it turns out in a rather big way yes it does. Let Tyler say it in his own words…

[quote]
When it comes to relieving climate change problems, there are two approaches. The first to put it squarely is to have everyone memorize facts about boats & bananas, and update that analysis as often as is necessary. The second approach is to rely on the price system, specifically to modify prices so that they reflect more information about the value of the environment. That’s the economically smart way to address climate change. The first method is wielding a pea shooter and the second is more like a bazooka.
[/quote]

Interested in web speed? Why generalists are better at scaling the web.

What he advocates more specifically is taxing the things we want to reduce. Biggest on the list are fossil fuels he says and next up meat production which through methane emissions contribute to climate change problems. These taxes will naturally curb our use, cause us to take fewer trips, be more efficient with our use, and tighten the wallet naturally.

Applying an economists eye to food & environment yields some excellent insights. For those of us in the startups & internet these fresh takes may well give us some insight in business too. If nothing else it’ll help us find the best meal for dinner!

Want more? Grab our Scalable Startups monthly for more tips and special content. Here’s a sample

When You Have to Take the Fall

Also find Sean Hull’s ramblings on twitter @hullsean.

One of the biggest jobs in operations is monitoring. There are so many servers, databases, webservers, search servers, backup servers. Each has lots of moving parts, lots that can go wrong. Typically if you have monitoring, and react to that monitoring, you’ll head off bigger problems later.

A problem is brewing

We, myself & the operations team started receiving alerts for one server. Space was filling up. Anyone can relate to this problem. You fill up your dropbox, or the drive on your laptop and all sorts of problems will quickly bubble to the surface.

Also check out – Why generalists are better at scaling the web.

As we investigated over the coming days, a complicated chain of processes and backups were using space on this server. Space that didn’t belong to them.

Dinner boils over

What happened next was inevitable. The weekly batch jobs kicked off and failed for lack of space. Those processes were not being monitored. Business units then discovered missing data in their reports and a firestorm of emails ensued.

Hiring? Get our MySQL DBA Interview Guide for managers, recruiters and candidates alike.

Why weren’t these services being monitored, they wanted to know.

Time to shoot the messenger

Having recently seen a changing of the guard, and a couple of key positions left vacant, it was clear that the root problem was communication.

Looking for talent? Why is it so hard to find a mythical MySQL DBA or devops expert these days?

I followed up the group emails, explaining in polite tone that we do in fact have monitoring in place, but that it seemed a clear chain of command was missing, and this process fell through the cracks.

I quickly received a response from the CTO requesting that I not send “these types of emails” to the team and to direct issues directly to him.

You might also like: A CTO Must Never Do This

A consultants job

As the sands continued to shift, a lead architect did emerge, one who took ownership of the products overall. Acting as a sort of life guard with a higher perch from which to watch, we were able to escalate important issues & he would then prioritize the team accordingly.

Are you a startup grappling with scalability? Keep in mind these 5 things toxic to scalability

Sometimes things have to break a little first.

What’s more a consultants job isn’t necessarily to lead the pack, nor to force management to act. A consultant’s job is to provide the best advice possible & to raise issues to the decision makers. And yes sometimes it means being a bit of a fall guy.

Those are the breaks of the game.

Want more? Grab our Scalable Startups monthly for more tips and special content. Here’s a sample

Ken Auletta Gets Us Googled

Also find Sean Hull’s ramblings on twitter @hullsean.

The title sounds vaguely fatalistic, the end of the world is nigh, that kind of thing. It turns out though that Auletta is a journalist having reported over the years a lot on old media. So when he says “as we know it” he’s speaking as much to old media as he is to the tech vanguard.

You’ll also want to check out The Big Switch – Rewiring the World From Edison to Google by Nicholas Carr.

But what makes his book superb isn’t just his phenomenal journalistic skills, in digging up all the facts and serving up a fair and accurate presentation of things. I think it’s important that he’s not a cheerleader at all, and approaches the topic with a critical eye as much to old media who ignored many of the warning signs in early 2000′s as to google who he emphasizes has been hubristic, at times arrogant, and has struggled with issues of privacy and copyright as they’ve built their technology.

Also check Jeff Jarvis’ What Would Google Do.

What makes this book even more important though is to step back and think of it as a case study in how the internet has become such a disruptive force. And in that light, google is a business which has rode that wave as much as it has defined it. Interestingly Google was not afraid to bring him to Mountain View to speak in their AtGoogleTalks series, and that video is now up on YouTube.

Ken Auletta on At Google Talks

Want more? Grab our Scalable Startups monthly for more tips and special content. Here’s a sample

When You're Hired to Solve a People Problem

Also find Sean Hull’s ramblings on twitter @hullsean.

A good five years ago I worked for a firm in online education. Among various products they provided through their website, they were struggling with a process to get content churned out more quickly. The bottleneck was slowing down their business, and limiting the new products they could offer.

Help Us Publish, Please…

Among a number of things I was asked to look at, one particularly vexing problem surrounded publishing. Adding new products had become a cumbersome & difficult process. It took days sometimes weeks. For obvious reasons the stake holders wanted to wrestle this process out of the hands of engineering, and place it were it arguably belonged in the hands of the business units.

[quote]
When you’re hired to solve specific technical problems it only figures that you go looking for software solutions. But sometimes the problems turn out to involve the people and processes of an organization. Getting them unstuck is one of the biggest challenges an professional services consultant can face. But it is also one of the most rewarding to solve.
[/quote]

Bumping into Fear, Uncertainty & Doubt

As I dug into the meat of the problem I began to work closely with the database administrator. He was a very smart gentleman & friendly in his own way. But he also spoke with a very thick accent and brusqueness about his manner that proved difficult at times. After working together for some awhile, however I began to win him over, and he started to trust me.

Looking for a top-flight database administrator? Here’s our interview guide for recruiters, managers and candidates alike

It became apparent that he was rather resistant to handing over the keys to the publish process to non-technical folks in other departments. Having handled his share of outages, and bungling screw ups, which sometimes fall on operations during some of the least hospitable hours on the dial, I could understand his concern. What’s more he knew the code which had grown unwieldy.

If I were to use a polite euphemism I would call it spaghetti code.

Management, Managers & Trouble Brewing

Around then the CTO decided to send a manager to sniff around. Unfortunately the manager in question was a very hands off type. His edict was simply to get this done in two weeks, and proceeded to go on vacation. Upon his return when things were still hitting snags, things started to go south.

Despite AWS failures firms like AirBNB and Reddit didn’t have to go down.

Though some of the process had been automated, I refused to move the changes into full push-button automation without first testing on dev environments. Of course those requests had fallen on deaf ears.

Problem comes to a head

Next the hands off manager escalated things upstream, of course adding his own spin on the situation. Shortly thereafter I’m called into the CTOs office only to get royally chewed out. A serious smack down which I’ll admit came almost out of nowhere.

A related article which readers also found quite popular: A CTO Must Never Do This

Oh, honestly I’m not complaining. On some level this is the job of the consultant. To act as the third party, wise or unbiased second opinion, and even punching bag at times.

Once things calmed down, I explained the situation from top to bottom. Yes there was messy code, and yes the process was complex, but it could of course be automated. What really stood in the way was a very resistant engineer who currently owned the process.

As much of the Sandy recovery continues, Devops can learn real lessons from the hurricane.

The CTO for his part concurred, having had trouble communicating with the engineer himself, and really not liking him much. He then appointed a proper project manager to oversee redoing the publish process from scratch.

A Plea for Cooperation

If I were to do it all again, for my part I’d sniff out the people dynamics more carefully. It’s often the case the companies have the engineering talent in house to solve a particular problem, but not the will or knowledge to put it into play.

Is your business growing? Having trouble scaling? Here’s how we do a performance review. It’s the first step on your way to hyper growth.

To managers & CTOs I’d encourage where possible to look for people, process and communication issues. Try to ferret out when something is an engineering problem, or whether it is one of people, silos and territory.

Want more? Grab our Scalable Startups monthly for more tips and special content. Here’s a sample

Switch to the cloud – shift of a century

The switch to cloud is way bigger than you think

A Review of Nicholas Carr’s book “The Big Switch”, available on Amazon here.

Also find Sean Hull’s ramblings on twitter @hullsean.

Do you work in devops or as a performance consultant? Do you manage web applications backed by databases? If so you probably love high performance beefy iron, big servers with equally fast RAID arrays that deliver lightening fast response to an entire application and ultimately your customers.

A related article, Devops can learn from Sandy, serious and very real disaster recovery lessons.

So if you’re like me you may feel a little leery about the cloud. On AWS for example, server commoditization has taken infrastructure for a sharp turn south. We struggle with unreliable disk performance & shared network bandwidth, while our applications compete with other customers in the so-called multi-tenant environment. Even the servers themselves drop like flies. Something’s got to give!

[quote]
All of them would be wise to study the examples of General Electric and Westinghouse. A hundred years ago, both these companies were making a lot of money selling electricity-production components and systems to individual companies. That business disappeared as big utilities took over electricity supply. But GE and Westinghouse were able to reinvent themselves. They became leading suppliers of generators and other equipment to the new utilities, and they also operated or invested in utilities themselves. Most important of all, they built vast new businesses supplying electric appliances to consumers — businesses that only became possible after the arrival of large-scale electric utilities. Sometimes a company can discover an even better business if it’s willing to abandon an old one.
[/quote]

Shopping for a smartphone? Find out why the Android platform is broken.

That’s why Carr’s book offers an eerie and uncanny read. What we’re seeing today in infrastructure very closely parallels what happened to electricity before it. Turns out at the turn of the century electricity production was not centralized and no electric grid was yet criss-crossing the country. Big companies actually built and managed their own power plants.

What happened?

Through the efforts of great entrepreneurs like Thomas Edison & Samuel Insull, the electricity production machinery were centralized and eventually incorporated and run by government managed utilities. All of this drove costs of electricity precipitously downward.

Looking for a database expert? Take a look at our MySQL Interview guide for candidates, hiring managers and recruiters alike.

How did we benefit? Oh can you count the ways?

Now households could afford electricity too. Next we saw consumer appliances and automation begin. Vacuum cleaners to washing machines flourished, bringing social change with it.

[quote]
What the fiber-optic internet does for computing is exactly what the alternating-current network did for electricity: it makes the location of the equipment unimportant to the user. But it does more than that. Because the internet has been designed to accomodate any type of computer and any form of digital information, it also plays the role of Insull’s rotary converter: it allows disparate and formerly incompatible machines to operate together as a single system. It creates harmony out of cacophony.
[/quote]

Cloud providers fail, components fail, datacenters fail. Find out why AirBNB & Reddit didn’t have to fail even while it’s AWS cloud went down.

The takeaway

The shift to cloud computing is way bigger than one application or one business. And the gains and momentum are way larger than we in devops may realize. With that it’s an inexorable shift, and one we would do well to embrace. Like all shifts we need to learn to adopt our technologies, as the benefits to business are incalculable.

Want more? Grab our Scalable Startups monthly for more tips and special content. Here’s a sample

When Clients Don't Pay – Consulting War Stories

Also find Sean Hull’s ramblings on twitter @hullsean.

It’s a cold cold winter out there, so as they say you need money to keep the lights on? Yep, that’s true whether you’re a small business or a consultant. Everyone has to pay their bills. Or do they?

It’s an unfortunate fact of life in business, but sometimes there are differences. Disagreements about deliverables, timelines, milestones, and deadlines. But when all the work is done, there are still sometimes differences over dollars.

Over the years most of those I’ve managed to work out with clients, but there were a few that went sour. One case was with a large entertainment firm. The music business is one that I hadn’t interacted with much before. For these guys I’d done a few days work in the past, and was paid promptly. Now they were in a bit of a jam. They called me up and asked if I could help.

Related article: A CTO Must Never Do This.

These types of emergencies often come at the worst times, and I explained that I was already juggling a few other things. They pleaded for help, and I relented. I carved out a full day of time for them, explaining the day rate and so forth. While on the phone though, I expressed caution.

[quote]I understand that your issue is urgent now, what happens if your needs change in the next 48 hours, I asked? Not a problem they replied, we can use your help anyway, so we’ll book you either way. Fast forward 48 hours, work canceled & client won’t take my call.[/quote]

Great I thought, verbal assurances. That works for me, I thought.

Fast forward 32 hours, and I receive an email saying the problem is resolved, and offering a “kill fee” of which I knew nothing, and which was never negotiated or discussed.

Along comes the day of reconning, and I call the client. They don’t take my call. Shortly there after I receive additional emails. I reply and explain we should talk on the phone. Still the client can’t find the time to pickup the call.

So I judiciously put together an invoice for the work. I email it directly to finance, and CC all parties. From there I get responses ranging from disinterest to denial. Over the coming months I periodically resend the invoice, but to no avail.

Or was it? I actually feel that this experience is to great avail.

1. Small disagreements foreshadow larger ones down the road
2. A relationship between client and vendor is a mutual one. If parties can only pickup the phone when they need something, then things are out of balance already.

Sure I lost a day & the fee associated. But I gained a lesson.

Patience and polite persistence

I firmly believe that being patient and persistent wins in the end. Sometimes clients have hiccups in payroll or budgets. Keep communication lines open.

Freelancers & consultants: Grab my Consulting 101 Guide.

Appealing to fairness

If you maintain a healthy relationship with your client, then appeals to fairness are normally heeded.

Appealing to promises

Emails are important to keep a paper trail of agreements. Communicate clearly and often so you know when you get derailed, and can refer back to what was agreed previously.

Popular: AirBNB didn’t have to failed – AWS hosting outages.

Setting precedents & expectations up front

This is an important one, that freelancers and consultants alike sometimes forget. Setting and agreeing on expectations is key. Often details are in the fine print or left out completely. So ongoing communication can iron out those differences or bring them to light.

Want to hire the best? Read our DBA Interview Guide.

Getting a protective deposit

If you haven’t worked together before, a deposit makes a lot of sense. Executing on this is more than a show of faith. It underlines that accounts payable is on board with your hiring, and you are now in the payments system.

Related: Hiring is a numbers game

Sizing each other up

Websites provide the first representation of you and your client to each other. How you carry yourself and how they feel meeting them face to face is important as well. Ideally you’ll meet each other at the client’s offices, where you may meet others on the team, and put names to faces. At the very least a skype call will go a long way as well.

Managing spend – communication along the way

Keep a close eye on invoices. If a client is getting behind, resolve it before continuing to work. Deadlines are mutual in a business relationship. Yours to complete work by a certain date, and theirs to pay by an equally agreed upon time.

Read also: Real Disaster Recovery Lessons from hurricane Sandy.

Never go to court – defer to the handshake & gentleman’s agreement

I know the lawyers out there may think I’m naive. But I’ve been in business a long time, and I believe a handshake means and says a lot. Also common sense language & contracts, in the form of emails and so forth are better than heavily legalese ones that no one but a lawyer can understand.

What’s more going to court has a huge cost in time & stress. Don’t go that route. I also think it appeals to clients knowing that you’re not the litigious sort.

Want more? Grab our Scalable Startups monthly for more tips and special content. Here’s a sample