How crazy can Kubernetes get?

I was flipping through Reddit and found this hilarious post referencing a Scott Adams Dilbert strip on Kubernetes.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

What I found even funnier were the comments on the Reddit thread. Read on for fun!

1. Watch your memory

One engineer said he had a dockerized app running with 3GB of memory serving just 7 customers.

Not to be outdone another ops guy pipes in that he has one using 180Gb of memory serving just a few hundred customers.

Of course this is the internet, and along comes a guy who has an app using 1TB of memory, with only one user!

Optimization be damned!

Read: Infrastructure provisioning – what is it and why is it important?

2. Beware growing application complexity

As you dockerize your application, you can support multiple versions of software and packages. This can keep you flexible but also enable engineers to kick the can down the road. Technical debt is real!

What’s more each microservice *can* be on a different stack, using a different language and framework. But just because you *can* do something doesn’t mean you should.

Though docker & kubernetes will enable the above, keep in mind your team has to support it. Using some cool new language that hasn’t really achieved critical mass? Remember the engineer who championed that, and built your business crown jewels on top of that, will eventually leave. And when he or she does, you will be faced with the challenge of finding someone who knows the stuff!

Related: 6 Devops interview questions

3. What is a microserved monolith?

Well it’s not really a thing, except it sounds fun. And a bit absurd. If all those docker containers never get optimized, they probably have layer upon layer of useless stuff. Start with a smaller base image, dont include debug stuff, and extra layers. And cleanup after installing packages.

Here’s a more detailed howto optimizing Docker image sizes.

Read: Is zero downtime even possible on RDS?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

What Matt Ranney learned scaling Uber to 1000 services

via GIPHY

I was recently watching Matt Ranney’s 2016 talk from GOTO conference. He’s an excellent speaker, not least because of his formidable experience at the helm of Uber.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

Jump to 11:45 to start out…

1. What’s the cost of many languages

As they embraced microservices, they discovered surprising problems around having so many languages.

For example it became hard to share code and furthermore it became hard to move between teems.

The upshot is that the fragmentation of languages siloed teams, and built up tribes around those languages, and expertise or interest in them.

Related: How can we keep cloud architectures simple

2. What’s the cost of using RPC for everything

In the microservices world, everything becomes a remote procedure call. For starters there are problems with HTTP and related semantics.

Also there are challenges with JSON. Since there aren’t any types, you don’t have a schema to lock you into formats. So you get weird behaviors down the line.

The upshot: Servers are not browsers. Indeed. Function calls are wayyyy better *within* a datacenter. That’s because you control both servers, so you can tune accordingingly.

Also: What hidden things does a deposit reveal?

3. Tuning across languages can be very challenging

Different languages have different tools available. Can your favorite tool work across all the languages you’ve decided to support in your microservices?

Even though you’re going to go with microservices, think about standardizing across those services. Use tools that work on all of them. And when building your services, make sure it outputs the same dashboard metrics.

Read: Can communication mixups sour an engagement?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Are shared databases back in vogue?

via GIPHY

I just stumbled upon this article by Roman Krivtsov on YC News Is shared database in microservices actually anti-pattern?. As a seasoned DBA in another life, I was intrigued by the title.

I devoured the piece.

Join 38,000 others and follow Sean Hull on twitter @hullsean.

One quote at the end really sums things up…

In the very beginning you probably don’t need microservices. Start with monolith and see, if you really need them in future.

Here’s my takeaway from it.

1. DB access by proxy

As the database itself is a service, does it make sense to use a microservice to essentially front the database? By doing that we simply add a layer of abstraction, need to keep the API up to date, and eat the network and compute expense of interacting with that data by proxy.

Related: When you have to take the fall

2. Changing db schema or service API

In a traditional database, when we update a schema, we can keep the old columns around, and simply add new. Thus we are backward compatible.

With the one db per microservice model, we must update the API everytime we add and change schema. This requires a lot more coding, and maintenance. It also means more nuance to remain backward compatible.

Related: Why generalists are better at scaling the web

3. Consistency of data on restore

This is one factor that is often forgotten. Suppose you have an orders service and users service. The Users db behind the users service fails, and must be restored. When the db is restored, a new user that happened just before failure, is lost. However, the Orders service still has a record which references that lost user. What then?

From there we would need a cleanup routine that would go around and remove inconsistent child records after failure. Alternatively we would need a way to backup all dbs from all microservices in a consistent point in time manner. *NOT* an easy task.

Shared database solves these problems in an elegant way.

Related: Why i ask for a deposit

4. Improving performance

Allowing access to orders & users tables in the same db call means eliminating all those slow API calls, associated network congestion and more. It centralizes that, allowing you to do SQL joins. Here the database does the heavy lifting of slicing and dicing, and returning only the packets of data you need.

Related: How progress reports can help engagements succeed

5. Should we bring back db admin job role?

When we centralize our database, we also centralize responsibility. There too, we return to the old debate of ops versus devs. I wrote an article years back titled the four-letter word dividing dev and ops.

When we have a job role for the database management, they have a mandate. Ensure backup & recovery, consistency & performance. Watch things. Monitor. Provide care & feeding. Put fences around applications, and constraints on data going in and out.

All these things are good things. And just like you want a building inspector to be different from the building developer, so too you want those separation of job roles in the software arena.

Related: How to hire a developer that doesn’t suck

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

How organizations can move faster with devops – a16z Sonal Chokshi interviews Nicole Forsgren & Jez Humble

via GIPHY

We hear a lot about devops these days, and the promise is temendous. It originally evolved out of Agile operations. But how to get those benefits at *my* organization?

Join 38,000 others and follow Sean Hull on twitter @hullsean.

How do we become a high performing organization, to move faster and build more secure and resilient systems? That’s the $64,000 question!

A16Z strikes again! Andreeson Horowitz’s epic podcast hosts world class guests around all sorts of startup & new technology topics. This week they interview Jez Humble and Nicole Forsgren. They run Dora which is DevOps Research and Assessment, which shows organizations just how to get the advantages of devops in the real world.

Technology does not drive organizational performance

Check out section 16:04 in the podcast…


“the point of distinction comes from how you tie process and culture together technology through devops”

It’s the classic Amazon model. They’re running hundreds of experiments in production at any one time!

Related: The 4 letter word dividing dev and ops

Day one is short, day two is long

The first interesting quote that caught my attention was at 4:40…


“Day one is when we create all of these systems. Day two is when we deploy to production. We have to deploy and maintain forever and ever and ever. We hope that day two is really long.”

As a long time op, this really really resonates for me. Brownfield deployments, which have already seen a wave of developers finish, and leave, and trying to manage that. Not easy!

Related: Why generalists are better at scaling the web

Mainframes of Kubernetes?

What about tooling? Is that important? Here’s what Jez has to say. Jump to 29:30…


“Implementing those technologies does *not* give you those outcomes. You can achieve those results with Mainframes. Equally you can use Kubernetes, Docker and microservices and not achieve those outcomes.”

Related: Is Amazon too big to fail?

Reducing Friction

Fast forward to timecode 28:45…


“Conways Law: Organizations which design systems are constrained to produce designs that are copies of the communication structures of these organizations.”

ie your software code looks like the shape of organization itself, and how we communicate. Super interesting. 🙂

Related: 6 devops interview questions

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Key lessons from the Devops Handbook

I picked up a copy of the DevOps Handbook.

This is not a book about how to setup Amazon servers, how to use git, codePipeline or Jenkins. It’s not about Chef or Ansible or other tools.

Join 33,000 others and follow Sean Hull on twitter @hullsean.

This is a book about processes & people. It’s about how & why automation & world-class infrastructure will make your business more agile, raise quality & increase productivity.

1. Infrastructure in version control

With technologies like Terraform and CloudFormation, the entire state of your infrastructure can be captured. That means you can manage it just like any other code.

Also: Myth of five nines – Why high availability is overrated

2. Pushbutton builds

You’ve heard it before. Automate your builds. That means putting everything in version control, from environment building scripts, to configs, artifacts & reference data. Once you can do that, you’re on your way to automating production deploys completely.

Related: 5 ways to move data to amazon redshift

3. Devs & Ops comingled

In the devops world, devs should learn about operations, infrastructure, performance & more. What’s more operations teams should work closely with devs.

Read: Why were dev & ops siloed job roles?

4. Servers as cattle not pets

In the old days, we logged into servers & provided personal care & feeding. We treated them like pets.

In the new world of devops, we should treat servers like cattle. When it begins to fail, take it out back and shoot it. (tbh i don’t love the analogy, but it carries some meaning…)

Also: Are SQL databases dead?

5. Open to learnings & failures

Organizations that are open to failures, without playing the blame game, learn quicker & recover from problems faster.

Also: Is Amazon too big to fail?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Some thoughts on 12 factor apps

12 factor app

I was talking with a colleague recently about an upcoming project.

In the summary of technologies, he listed 12 factor, microservices, containers, orchestration, CI and nodejs. All familiar to everyone out there, right?

Join 28,000 others and follow Sean Hull on twitter @hullsean.

Actually it was the first I had heard of 12 factor, so I did a bit of reading.

1. How to treat your data resources

12 factor recommends that backing services be treated like attached resources. Databases are loosely coupled to the applications, making it easier to replace a badly behaving database, or connect multiple ones.

Also: Is the difference between dev & ops a four-letter word?

2. Stay loosely coupled

In 12 Fractured Apps Kelsey Hightower adds that this loose coupling can be taken a step further. Applications shouldn’t even assume the database is available. Why not fall back to some useful state, even when the database isn’t online. Great idea!

Related: Is Amazon too big to fail?

3. Degrade gracefully

A read-only or browse-only mode is another example of this. Allow your application to have multiple decoupled database resources, some that are read-only. The application behaves intelligently based on what’s available. I’ve advocated those before in Why Dropbox didn’t have to fail.

Read: When hosting data on Amazon turns bloodsport

Conclusion

The twelve-factor app appears to be an excellent guideline on building cleaning applications that are easier to deploy and manage. I’ll be delving into it more in future posts, so check back!

Read: Are we fast approaching cloud-mageddon?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Should we be muddying the relational waters? Use cases for MySQL & Mongodb

muddy sewer tunnels

Many of you know I publish a newsletter monthly. One thing I love about it is that after almost a decade of writing it regularly, the list has grown considerably. And I’m always surprised at how many former colleagues are actually reading it.

So that is a really gratifying thing. Thanks to those who are, and if you’re not already on there, signup here.

Join 28,000 others and follow Sean Hull on twitter @hullsean.

Recently a CTO & former customer of mine reached out. He asked:

“I’m interested to hear your thoughts on the pros and cons of using a json column to embed data (almost like a poor-man’s Mongo) vs having a separate table for the bill of materials.”

Interesting question. Here are my thoughts.

1. Be clean or muddy?

In my view, these type of design decisions are always about tradeoffs.  

The old advice was normalize everything to start off with.  Then as you’re performance tuning, denormalize in special cases where it’ll eliminate messy joins.  The special cases would then also need to be handled at insert & update time, as you’d have duplication of data.

NoSQL & mongo introduce all sorts of new choices.  So too Postgres with json column data.  

We know that putting everything in one table will be blazingly fast, as you don’t need to join.  So reads will be cached cleanly, and hopefully based on single ID or a small set of ID lookups.  

Also: Is the difference between dev & ops a four-letter word?

2. Go relational

For example you might choose MySQL or Postgres as your datastore, use it for what it’s good at.  Keep your data in rows & columns, so you can later query it in arbitrary ways.  That’s the discipline up front, and the benefit & beauty down the line.

I would shy away from the NoSQL add-ons that some relational vendors have added, to compete with their newer database cousins. This starts to feel like a fashion contest after a while.

Related: Is automation killing old-school operations?

3. Go distributed

If you’d like to go the NoSQL route, for example you could choose Mongodb. You’ll gain advantages like distributed-out-of-the-box, eventually consistent, and easy coding & integration with applications.

Downside being you’ll have to rearrange and/or pipeline to a relational or warehouse (redshift?) if & when you need arbitrary reports from that data.  For example there may be new reports & ways of slicing & dicing the data that you can’t forsee right now.

Read: Do managers underestimate operational cost?

4. Hazards of muddy models

Given those two options, I’m erring against the model of muddying the waters.  My feeling is that features like JSON blobs in Postgres, and the memcache plugin in MySQL are features that the db designers are adding to compete in the fashion show with the NoSQL offerings, and still keep you in their ecosystem.  But those interfaces within the relational (legacy?) databases are often cumbersome and clunky compared to their NoSQL cousins like Mongo.

Also: Is the difference between dev & ops a four-letter word?

5. Tradeoffs of isolation

Daniel Abadi and Jose Faleiro published an interesting article on a very related topic Why MongoDB, Cassandra, HBase, DynamoDB, and Riak will only let you perform transactions on a single data item.

The upshot is that in databases you can choose *TWO* of these three characteristics. Fairness, Isolation & Throughput.

Relational databases sacrifice throughput for fairness & isolation. Distributed databases sacrifice isolation to bring you gains in throughput & horizontal scalability of writes.

That’s a lot of big words to say one simple thing.

You can’t have it both ways.

Also: Is the difference between dev & ops a four-letter word?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Are frameworks a shortcut to slow bloated code?

elephant traffic jam

I was reading one of my favorite blogs again, Todd Hoff’s High Scalability. He has an interesting weekly post format called “Quotable Quotes”. I like them because they’re often choice quotes that highlight some larger truth.

Join 32,000 others and follow Sean Hull on twitter @hullsean.

1. Chasing bloody frameworks

One that caught my eye was this one by Sandro Mancuso:

Frameworks are bad? Aren’t these the foundation of reusable code? Can we build for the web without Ruby’s Rails, PHP’s Laravel or Java’s Spring?

Also: Are we fast approaching cloud-mageddon??

2. Architecture purity or bloat?

Luckily the conversation remained civil. The other side of the isle piped up with practicality. I think Jeff is saying “Hey we gotta get our jobs done.”

Related: Did MySQL & Mongo have a beautiful baby called Aurora?

3. Look Ma No Frameworks

Then Pablo Chacin jumps in with his slideshare deck, rejecting Frameworks as hugely wastful.

Read: When hosting data on Amazon turns bloodsport

4. Beware the ORM

Anyone who’s a regular reader of this blog knows that I’ve railed on against ORM’s for a long time. These are object relational mappers, they are a library on to of a relational database. They allow developers to build software, without mucking around in SQL.

Hibernate is a famous one. I remember back in the late 90’s I was brought on to fix some terrible performance problems. At root the problem was the SQL code. It wasn’t being written by their team & tuned carefully. It was written en mass & horribly by this Hibernate library.

Also: Is the difference between dev & ops a four-letter word?

5. Where’s the oversight?

Perhaps the biggest risk to many startups is that the decision isn’t being made at all. What do I mean by this?

I think this tweet gets to the heart of it. Often the decision to use a framework or not is simply hidden in plain sight, an assumption albeit a large one, that this is how you build software.

But these decisions are so fundamental because once your scaffolding is built, it becomes very hard to disentangle. Rip & replace becomes terribly expensive, and scalability becomes a painful unattainable dream.

Also: 5 things toxic to scalability

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Why you should build a portable cloud

486 linux server

I was recently browsing Ycombinator News. It is always an endless trove of the curious & interesting.

I stumbled onto Karanbir Singh’s post The Portable Cloud. I was curious, what is that?

Join 28,000 others and follow Sean Hull on twitter @hullsean.

This story gave me warm fuzzies… I was excited in a similar way when Linux was first released. This was many years back through the mists of time in 1992. I had recently graduated computer science, and one of my favorite classes was Operating Systems. We worked to build an OS following Andrew Tennenbaum’s book Minix.

After graduation, I heard about the Linux project & got excited. I was hearing whispering online that Linux could really completely replace windows. So I bought all the parts to build a 486 tower, graphics card, motherboard, memory cards & IDE drives. This ran into the thousands of dollars. Hardware wasn’t cheap then! Keyboard & monitor. I even ordered an optical mouse because it felt like you were sitting at a sun workstation, at home!

I remember putting all this together, and loading the first floppy disk into the thing. Did I image the disks properly? Will it really load something?

Up comes the bios and sure enough it’s booting off of the floppy drive. I thought “Wow, mother of god! This is amazing!”

From there I had init running, and soon the very seat of the soul, the Unix OS itself. That felt so darn cool.

After that I’d spend weeks configuring x-windows, but to have a GUI seemed like the mission impossible. And then you’d go about tweaking and rebuilding your kernel for this or that.

Thank you to Karanbir for rekindling this memory. It’s a great one!

For those starting out now as a developer, operations, or cloud site reliability engineer, I would totally recommend following Karanbir’s instructions. Here’s why!

1. Learning by building

My favorite thing about building a server myself, is that there’s something physical going on. You’re plugging in a cable for the disk bus. Bus is no longer just a concept, but a thing you can hold. You’re plugging in memory, you can look at it & say oh this is a chip, it’s different than a disk drive. You can hold the drive and say, oh there’s a miniature little record player in there, with magnetic arm. Cool!

Also: Is Amazon too big to fail?

2. Linux early beginnings

Another thing I remember about those days, was feeling like I was part of something big. I knew operating systems were crucial. And I knew that Windows wasn’t working. I knew it wouldn’t scale to the datacenter anyway.

I realized I wasn’t the only one to think this way. There were many others as excited as me, who were contributing code, and debug reports.

Related: Did Airbnb have to fail?

3. Debugging & problem solving

Building your own server involves a ton of debugging. In those days you had to compile all those support programs, using the GNU C compiler. You’d run make and get a whole slew of errors, and fire up your editor and get to work.

Configuring your windowing system meant figuring out where the right driver was, and also buying a graphics card that was *supported*. You would then tweak the refresh frequencies, resolution, and so forth. There was no auto detection. You could actually fry your monitor, if you set those numbers wrong!

Read: When hosting data on Amazon turns bloodsport

4. Ownership of the stack

These days you hear a lot about “fullstack engineers”. There is no doubt in my mind, that this is the way to become one. Basic systems administration requires you to compile other peoples software & troubleshoot it. All those developer skills that will come in very handy.

They force you to see all the hardware, and how it fits together into a greater performant whole. It also gives you an appreciation for speed. Use one bus such as IDE or another such as SCSI and experience a different performance profile. Because all that software that Unix is paging in and out of memory, it’s doing by reading & writing to disk!

Also: Are SQL Databases Dead?

5. Learn to be a generalist

I’ve argued before that being a generalist allows you to better scale the web. I still think this, and I believe it is this formative experience building servers from individual parts, that taught me whole the larger whole fit together.

This allows me to look at problems today, and jump to causes of performance problems quickly.

Also: Does Volkswagen highlight the bloodsport in benchmarks?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Are we fast approaching cloud-mageddon ?

storm coming

One look at StackShare’s trending technologies, and you’ll discover the exploding growth of languages, webservers, load balancers, databases, caching servers, automation & monitoring tools, continuous integration suites & a broad spectrum of Software as a service solutions.

The choices today boggles the mind. Choice is good, but too much choice can mean trouble too.

Join 30,000 others and follow Sean Hull on twitter @hullsean.

1. What am I actually using?

Erich Schubert wrote a superb piece about the sad state of the sysadmin in the age of containers. Here’s what caught my eye:

Stack is the new term for “I have no idea what I’m actually using”.

That definitely rings true for me. The customers I’m seeing these days have such complicated stacks, that nobody really knows what’s installed. That’s dangerous!

Also: Do today’s startups assemble at their own risk?

2. Embrace failure more broadly

Recently I wrote a blog post asking Is AWS the patient that needs constant medication?. It got a lot of traction, and here’s why I think that happened.

AWS uses very commodity, cheapo components. The assumption is, with an infinitely redundant datacenter, component failure is ok. It’s ordinary & everyday.

Unfortunately most startups, even ones that employ some Ansible & devops, still don’t have Netflix grade automation.

Those regularly everyday failures are still getting detected by old-school manual monitoring. And that’s a recipe for trouble

Also: 5 Things toxic to scalability

3. What are complex systems?

In this excellent deck, James Urquhart talks about emergent behavior in complex systems. It’s worth a quick read.

***

Read: How I find entrepreneurial focus

4. What to do? Do you like boring?

Dan McKinley formerly principal engineer at Etsy & now with Stripe wrote a brilliant essay arguing for boring technology.

This comes as a shock to many in the startup world. It sort of smacks in the face of open source, or does it?

I worked in the enterprise space as an Oracle DBA for a decade starting in the mid-nineties. Among DBAs there was always a chuckle when a new version of Oracle came out. No one wanted to touch it with a ten foot pole. Sure we’d install it on test boxes, start learning the new features and so forth. But deploy it? No way, wait a good 2 or even 3 years before upgrading.

Meanwhile management was eager for the latest software. Don’t we want the newest? The Oracle sales guys would be selling the virtues of all sorts of features that nobody needed right away anyway.

Choosing boring components takes discipline to fight sexy new technologies & bleeding edge versions. But staid & stodgy wins you everyday in operations uptime.

Related: Is automation killing old-school operations?

5. Use tried & tested components

Do you find your application or stack contains java, ruby, python & PHP? Choose one.

One webserver like nginx, one caching server like memcache or redis, one search server like solr or elasticsearch, one database like MySQL or postgres. Standardize all your components on one image, so you can use that for all your servers, regardless of which you use.

Fewer components will mean fewer interdependencies, less maintenance, & less chaos.

Also: What’s the luckiest thing that’s happened in your career?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters