I like a book like Gerry Czarnecki’s even if I can’t pronounce his name. He’s laid out things for readers, a very busy bunch who are going to have trouble finding time in their day to read, but can sorely use the advice. Organized into 30 daily bites of probably 15-20 minutes, you can dig through on your commute to work, or on your lunch break.
Czarnecki should know. As a 2nd Army Lieutenant then later heading up board of directors at organizations large and small he’s seen a lot. He digs up the best stories, and offers up lessons straight out of his experiences.
On interviews – Be careful that you do not let the resume dominate your conversation. You will gain better insight if you listen to what the candidates want to talk about or what they think you want to talk about. – Gerry Czarnecki
Some of the topics you’ll touch on…
o dealing with mistaken hires that don’t work out
o finding a mentor
o career moves & pivots
o setting expectations with new hires
o hitting peak performance
o filling your bus with rock stars
For startups an all-time favorite of mine is REWORK – 37 Signals guys. These guys have taken the lean methodology and built a real business by being efficient. The pages resonate for me as a small business owner. I’ve found many of the same lessons work in the real world for me.
The covers are controversial and grab your attention, and the copy is top notch. It’s like reading the economist but with more tech & business heavy content. Work lifestyle articles, back-to-back with business book reviews as haiku? A solid dose of business & tech trends, startups, entrepreneurs and innovation. It’s all here.
When I say debutantes, it’s a nod to beginners, for this book forges a very solid and complete introduction to the topic of MySQL. Start with installing the software & setting up your environment, and then move on to really understanding the SQL language, from commands to create objects, to ones for adding & modifying data, and then writing code around it.
There’s a thorough discussion of datatypes, stored procedures, functions and views.
Paul Dubois’ definitive reference makes a excellent compliment to High Performance MySQL. They should sit alongside eachother on your database bookshelf.
For developers there are chapters on writing applications in C, another for Perl and a third for PHP.
For DBAs there are chapters on security, backups, replication, understanding the data directory and general server administration. There is also good coverage of both 5.5 and the newly released 5.6 of MySQL.
What I like about this book
You can think of this book as a definitive reference to MySQL. It includes much of the online documentation that you would find at Oracle’s site, such as command & variable reference, and detailed explanation of how to use the client tools.
Dubois also goes beyond the online documentation though, giving you a bit of a background around concepts, a broader more complete discussion.
Want to find out how far your slaves *really* are behind? pt-heartbeat is your friend.
Want to analyze your slow query log to produce a useful summary report? pt-query-digest to the rescue.
I also see no mention of innotop, which I would also say is an essential tool. These aren’t really advanced topics, so It’s unclear why they are missing. In the real world you need these tools to do your job.
My more general criticism is where the book lacks real-world advice from a seasoned DBA. At times the writing feels a bit more of the official line on how things work. But in day-to-day devops and operations, things can be quite different.
For example, stored procedures. In MySQL they are there, however using them brings real performance challenges. They’re not always compatible with replication. Given all of that, why include a whole chapter with endless discussion of them without strong reservations. It would lead a novice user or developer to incorporate them into an application only to be shocked and surprised at the problems they bring.
Another example, looking through the system variables reference, I see the sync_binlog option. There is a short caution “…lower values provide greater safety in the event of a crash, but also affect performance more adversely”. Now reading this as a novice DBA I might think great, crash protection. But having tried this parameter in production, I found a huge impact on performance and had to disable it. What’s the advice here? It’s a bit confusing.
This is a really great book as an introduction to MySQL, and delving into intermediate topics. I would sit it on your bookshelf along side High Performance MySQL. What this book lacks in advice, you can turn to the latter book, and what High Performance MySQL lacks in terms of introductory material this book covers in spades. They make a great compliment to each other.
Join 19k others and follow Sean Hull on twitter @hullsean.
1. Slow Disk I/O – RAID 5 – Multi-tenant EBS
Disk is the grounding of all your servers, and the base of their performance. True with larger and larger main memory, much is available in cache, a server still needs to constantly read from disk and flush things from memory. So it’s a very very important component to performance and scalability.
What’s wrong with Raid 5?
Raid 5 was designed to give you more space, using fewer disks. It’s often used in a server with few slots or because ops misunderstand how bad it will impact performance. On a database server it can be particularly bad.
All writes see a performance hit. What’s worse is if you lose a disk, the RAID though technically still on line, will perform SO SLOWLY as to be offline. And a rebuild takes many hours. Worse still is the risk to lose another drive during that rebuild. What if you have order a drive and it takes a couple of days?
RAID 10 is the solution. Mirror each set of two drives, then stripe over those. Even with only four slots available, it’s worth it. Good read performance, good write performance, and protection.
What the heck is multi-tentant?
In the cloud, you share servers, network & disk just like you do apartments in a building. Hence the name. Amazon’s EBS or elastic block storage, extends this metaphor, offering you the welcome flexibility of a storage network. But your bottleneck can be fighting with other tenants on that same network.
Default servers do have this problem, however AWS has addressed this serious problem with a little known but VERY VERY useful feature called Provisioned IOPS. It’s a technical name, but means you can lock in reliable disk I/O. Just what the scalability doctor ordered.
MySQL is good at a lot of things, but it’s not ideal for managing application queues. Do you have a table like JOBS in your database, with a status column including values like “queued”, “working”, and “completed”? If so you’re probably using the database to queue work in your application.
It’s not a great use of MySQL because of locking problems that come up, as well as the search and scan to find the next task.
Luckily their are great solutions for developers. RabbitMQ is a great queuing solution, as is Amazon’s SQS solution. What’s more as external services they’re easier to scale.
Scalability becomes key to your business, as you customer base grows. But it doesn’t have to be impossible. Disk I/O, caching, queuing and searching are all key areas where you can make a big dent, in a manageable way. Juggle your technical debt too, and you’re golden!
Oracle has full text search support, why shouldn’t we assume the same in MySQL? Well MySQL *does* have this, but in many versions only with the old MyISAM storage engine. It has it’s set of corruption problems, and isn’t really very performant.
Better to use a proven search solution like Apache Solr. It is built specifically for search, includes excellent library support for developers of most modern web languages and best of all is easy to SCALE! Just add more servers on your network, or distributed globally.
For folks interested in the bleeding edge, Fulltext is coming to Innodb crash safe & transactional storage engine in 5.6. That said you’re still probably better off going with an external solution like Solr or Sphinx and the MySQL Sphinx SE plugin.
Cache, cache, and cache some more. Your webservers should use a solid memcache or other object cache between them & the database. All those little result sets will sit in resident memory, waiting for future web pages that need them.
Next use a page cache such as varnish. This sits in front of the webserver, think of it as a mini-webserver that handles very simple pages, but in a very high speed way. Like a pack of motorbikes riding down an otherwise packed freeway, they speed up your webserver to do more complex work.
Browser caching is also important. But you can’t get at your customers browsers, or can you? Well not directly, but you can instruct them what things to cache. Do that with proper expires headers. Have your system administrator configure apache to support this.
Technical debt can bite. What is it? As you’re developing an new idea, you’ll build prototypes. As those get deployed to customers, change gets harder, and past things you glossed over because problems. One team leaves, and another inherits the application, and the problems multiple. Overtime you’re building your technical debt as your team spends more time supporting old code and fixing bugs, and less time building new features. At some point a rewrite of problem code becomes necessary.
Thomas Bayes was a scientist & thinker, Fellow of the Royal Society, and back in 1763 author of “An Essay toward Solving a Problem in the Doctrine of Chances”. His method advocated learning by approximation, to get closer and closer to the truth by gathering more information, and factoring that into probabilities & predictions.
What isn’t acceptable under Bayes’s theorem is to pretend that you don’t have any prior beliefs. You should work to reduce your biases, but to say you have none is a sign that you have many.
Why should you care?
From hurricane & earthquake warnings, to financial storms or terrorism, prediction is more important than ever. Epidemiologists can make use of Bayesian techniques to protect populations, gamblers can use it in sport, and investors for markets.
Nate is famous for predicting the 2012 presidential election with uncanny accuracy. So the book is an in depth look at how he thinks, and how he works with data. He talks of Hedgehogs – those who believe in big ideas and work from large principals, versus Foxes who see the world as messy, often inconsistent and unpredictable, but who nevertheless tend to present better though less definitive predictions. The philosophy is less of modeling, and more of testing, and adjusting along the way to get closer to the truth.
Nate interviewed John Sanders of a scout for the LA Dodgers. He identified five abilities and characteristics that predict success in baseball. Looking at them together, I think they can well predict success in Startup land too.
Are you a developer or startup entrepreneur? Have you ever been frustrated with some of the claims made by the sales team or lacked the patience or ability to communicate across departments?
Join 4000 others and follow Sean Hull on twitter @hullsean.
Just out of college
Just out of college I got a job as a Macintosh Software Developer for a small firm outside of University at Buffalo. It was a ten person company, and half of us were on the technology side of the house. I was doing C++ & Graphical Interface design & coding.
Besides coding, I also fielded support calls from customers which brought me perspective on both what they wanted, and where they struggled with the software. Our app helped consumers and nutritionists track diet & exercise.
The sales team made promises of technology the company wasn’t capable of delivering. Meanwhile the engineering team was sent scrambling to answer to those promises.
Soon I was fielding questions from customers asking when the new heart rate monitoring would be available. I followed up by talking with the team lead & chief architect. He had no plans of building such a feature, nor did we even know how it would be possible!
We checked in at our weekly meetings, and the CEO explained that the sales team was simply “ahead” of engineering. Years ahead apparently even of the technology that was possible at the time!
Fast forward 5 years to professional services
A half decade later I’m doing independent consultanting for dot-coms. Much of my business came from word of mouth. Helping a firm out of a pinch, speeding their site so they can handle 10x customers on the same servers and suddenly everyone is your friend!
But all is not smooth sailing in the freelance consulting world. The dot-com crash comes along and budgets are squeezed tighter. Business spend is reduced and every dollar is scrutinized. I learned to speak to prospects about savings and personalized service, advantages of lower overhead, and real return I could provide. At the end of the day if they’re not buying, your services aren’t worth their cost!
The sales process should inform the business about what customers really want. In a successful startup there is communication back and forth with engineering and business units so all are working in harmony.
Now coming full circle I have a wide perspective on business. I understand the engineering fundamentals, and the limitations of technology. I also have a grasp of product, and how business units must manage the bottom line, and deliver to customers or else perish in the marketplace.
For the two to achieve a happy marriage, you must bring a balance of execution & technical debt, with satisfying a real customer need in the marketplace. And therein lies the innovation & startup sweet spot!
I just finished reading Tyler Cowen’s opus, An Economist Gets Lunch. I have to admit I’m already a fan of his writing, getting a daily dose on from his blog Marginal Revolution.
Join 4000 others and follow Sean Hull on twitter @hullsean.
What I like about this book is that it is unconventional by definition. Further economists like scalability engineers like to think about efficiency. How can I squeeze out more from less? Like the business question how do I get better ROI or more bang for my buck? Questions spring to mind like – What does an economist know about food? Or – What does food eating have to do with economics? Well on both points you’ll get some surprising answers.
To the former question, Cowen has some really good insight because he brings the fresh perspective of an economist. His sort of mantra throughout the book is:
Food is a product of economic supply and demand, so try to figure out where the supplies are fresh, the suppliers are creative and the demanders are informed.
Economists & engineers talk shop
What about the second question, how is the food we eat related to economics? Further does it have an impact on environmental and energy consumption questions? As it turns out in a rather big way yes it does. Let Tyler say it in his own words…
When it comes to relieving climate change problems, there are two approaches. The first to put it squarely is to have everyone memorize facts about boats & bananas, and update that analysis as often as is necessary. The second approach is to rely on the price system, specifically to modify prices so that they reflect more information about the value of the environment. That’s the economically smart way to address climate change. The first method is wielding a pea shooter and the second is more like a bazooka.
What he advocates more specifically is taxing the things we want to reduce. Biggest on the list are fossil fuels he says and next up meat production which through methane emissions contribute to climate change problems. These taxes will naturally curb our use, cause us to take fewer trips, be more efficient with our use, and tighten the wallet naturally.
Applying an economists eye to food & environment yields some excellent insights. For those of us in the startups & internet these fresh takes may well give us some insight in business too. If nothing else it’ll help us find the best meal for dinner!
Also find Sean Hull’s ramblings on twitter @hullsean.
One of the biggest jobs in operations is monitoring. There are so many servers, databases, webservers, search servers, backup servers. Each has lots of moving parts, lots that can go wrong. Typically if you have monitoring, and react to that monitoring, you’ll head off bigger problems later.
A problem is brewing
We, myself & the operations team started receiving alerts for one server. Space was filling up. Anyone can relate to this problem. You fill up your dropbox, or the drive on your laptop and all sorts of problems will quickly bubble to the surface.
As we investigated over the coming days, a complicated chain of processes and backups were using space on this server. Space that didn’t belong to them.
Dinner boils over
What happened next was inevitable. The weekly batch jobs kicked off and failed for lack of space. Those processes were not being monitored. Business units then discovered missing data in their reports and a firestorm of emails ensued.
I followed up the group emails, explaining in polite tone that we do in fact have monitoring in place, but that it seemed a clear chain of command was missing, and this process fell through the cracks.
I quickly received a response from the CTO requesting that I not send “these types of emails” to the team and to direct issues directly to him.
As the sands continued to shift, a lead architect did emerge, one who took ownership of the products overall. Acting as a sort of life guard with a higher perch from which to watch, we were able to escalate important issues & he would then prioritize the team accordingly.
What’s more a consultants job isn’t necessarily to lead the pack, nor to force management to act. A consultant’s job is to provide the best advice possible & to raise issues to the decision makers. And yes sometimes it means being a bit of a fall guy.
Also find Sean Hull’s ramblings on twitter @hullsean.
The title sounds vaguely fatalistic, the end of the world is nigh, that kind of thing. It turns out though that Auletta is a journalist having reported over the years a lot on old media. So when he says “as we know it” he’s speaking as much to old media as he is to the tech vanguard.
But what makes his book superb isn’t just his phenomenal journalistic skills, in digging up all the facts and serving up a fair and accurate presentation of things. I think it’s important that he’s not a cheerleader at all, and approaches the topic with a critical eye as much to old media who ignored many of the warning signs in early 2000′s as to google who he emphasizes has been hubristic, at times arrogant, and has struggled with issues of privacy and copyright as they’ve built their technology.
What makes this book even more important though is to step back and think of it as a case study in how the internet has become such a disruptive force. And in that light, google is a business which has rode that wave as much as it has defined it. Interestingly Google was not afraid to bring him to Mountain View to speak in their AtGoogleTalks series, and that video is now up on YouTube.
Also find Sean Hull’s ramblings on twitter @hullsean.
A good five years ago I worked for a firm in online education. Among various products they provided through their website, they were struggling with a process to get content churned out more quickly. The bottleneck was slowing down their business, and limiting the new products they could offer.
Help Us Publish, Please…
Among a number of things I was asked to look at, one particularly vexing problem surrounded publishing. Adding new products had become a cumbersome & difficult process. It took days sometimes weeks. For obvious reasons the stake holders wanted to wrestle this process out of the hands of engineering, and place it were it arguably belonged in the hands of the business units.
When you’re hired to solve specific technical problems it only figures that you go looking for software solutions. But sometimes the problems turn out to involve the people and processes of an organization. Getting them unstuck is one of the biggest challenges an professional services consultant can face. But it is also one of the most rewarding to solve.
Bumping into Fear, Uncertainty & Doubt
As I dug into the meat of the problem I began to work closely with the database administrator. He was a very smart gentleman & friendly in his own way. But he also spoke with a very thick accent and brusqueness about his manner that proved difficult at times. After working together for some awhile, however I began to win him over, and he started to trust me.
It became apparent that he was rather resistant to handing over the keys to the publish process to non-technical folks in other departments. Having handled his share of outages, and bungling screw ups, which sometimes fall on operations during some of the least hospitable hours on the dial, I could understand his concern. What’s more he knew the code which had grown unwieldy.
If I were to use a polite euphemism I would call it spaghetti code.
Management, Managers & Trouble Brewing
Around then the CTO decided to send a manager to sniff around. Unfortunately the manager in question was a very hands off type. His edict was simply to get this done in two weeks, and proceeded to go on vacation. Upon his return when things were still hitting snags, things started to go south.
Though some of the process had been automated, I refused to move the changes into full push-button automation without first testing on dev environments. Of course those requests had fallen on deaf ears.
Problem comes to a head
Next the hands off manager escalated things upstream, of course adding his own spin on the situation. Shortly thereafter I’m called into the CTOs office only to get royally chewed out. A serious smack down which I’ll admit came almost out of nowhere.
Oh, honestly I’m not complaining. On some level this is the job of the consultant. To act as the third party, wise or unbiased second opinion, and even punching bag at times.
Once things calmed down, I explained the situation from top to bottom. Yes there was messy code, and yes the process was complex, but it could of course be automated. What really stood in the way was a very resistant engineer who currently owned the process.
The CTO for his part concurred, having had trouble communicating with the engineer himself, and really not liking him much. He then appointed a proper project manager to oversee redoing the publish process from scratch.
A Plea for Cooperation
If I were to do it all again, for my part I’d sniff out the people dynamics more carefully. It’s often the case the companies have the engineering talent in house to solve a particular problem, but not the will or knowledge to put it into play.
To managers & CTOs I’d encourage where possible to look for people, process and communication issues. Try to ferret out when something is an engineering problem, or whether it is one of people, silos and territory.