Categories
All CTO/CIO Devops

What tools & tech are devops engineers using today

via GIPHY

I just stumbled upon Graham King’s blog, and I’m liking his writing. He wrote an excellent piece a developer goes to a DevOps conference.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

I’ve been to plenty of Unix & operations type conferences over the years, so topics don’t surprise me. But hearing about a developer’s experience brings a new perspective and some great insights.

1. Tools change but mindset stays the same

Some talk about Devops as doing away with operations. Those job roles just aren’t necessary anymore. Well maybe for a small firm, or maybe shops that have pushed 2-pizza agile to the max. But handing the operations duties to developers has limitations. As I mentioned here (the difference between dev and ops is a four letter word…) these different job roles have different mandates.

It’s like an architect can design a building, and it can be a very beautiful house. But a super or building manager keeps it running over the years. He or she knows what to look for in cracked roofs, knows how to keep rodents & pests at bay, knows how to repair and maintain & stay ahead of the game.

In that analogy, the architect is the developer, while the super or building manager is the operations team. They’re two different mindsets, rarely shared in one person.

Read: What did Matt Ranney discover scaling Uber to 1000 microservices?

2. Being on-call is a b*tch

I could write volumes about being on-call. Getting woken up in the middle of the night, because someone pushed broken code is no fun. What’s more broken can have different meanings.

Broken can be something QA should catch, like a button doesn’t work or there’s an issue with some browser. It could also be that some new product feature doesn’t work properly.

But from the ops perspective, broken could also be some new feature doesn’t scale. It makes a million API calls, or makes a servless call that times out. These types of broken are much harder to test for.

This is also why traditionally operations and development were two different teams. Because from the vantage of the business, they had different mandates.

Ops was mandated with stability. So they don’t want change. Change breaks things, and wakes you up at 3am.

Devs are mandated with features changes, and product improvement. So they naturally bring change to the table.

And between the two we search for balance. I wrote a piece that hit on exactly these points the difference between dev and ops is a four letter word…

Related: Can humility help you in your career?

3. The kingmaker tools

Kubernetes – you’ve heard of it, you’re probably using it. Devs package their app as a docker container, and ops push that container through CI/CD pipeline, and finally orchestrate & deploy with kubernetes. Seems like the *only* way to do things these days, right?

But some argue Docker may not be right for everyone and certainly this stack brings a *lot* of complexity for small organizations.

Related: Is AWS too complex for small dev teams?

Terraform I’m a big fan of this technology. Once you’ve captured your entire stack in code, you can version it, check it into git, and manage it like any other asset. That’s great, but there are so many other benefits. You can easily deploy that same stack in another region, or tweak it to create dev, stage and production. Cool stuff!

Related: I tried to build infrastructure as with Terraform and AWS. It didn’t go as I expected

Ansible All those BASH scripts you have sitting around? Check them into version control before it’s too late! One great thing about Ansible is with slight tweaks and can run those bash scripts almost as-is.

And for ops who already have experience with managing things by hand, you can get up to speed with Ansible, in a few days. The learning curve isn’t as tough as Puppet or Chef, and brings many or most of the benefits.

Packer Here’s another cool tool. Chances are all those AMI’s that Amazon has pre-baked, need tweaks for your setup. Now you could do all that work post spinup with Ansible. And that’s fine. But it’ll be slower, and possibly prone to breaking if the base AMI changes.

Enter Packer, another great tool from the folks who brought us Terraform, Hashicorp. This tool allows you to write yaml files that then build AMI’s. You can then use your pipeline and other automation tools to automate those as well. Cool !

Read: What happened when I offered advice outside my pay grade?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Categories
All Consulting CTO/CIO

How do we make domain decisions when coding software?

via GIPHY

I stumbled on an interesting discussion over at hacker news about when you have to make a decision in code outside of your domain of expertise.

This is a super interesting topic to me. As the world is more and more powered by computers and algorithms, this seemingly obscure philisophical corner of computing begins to affect us all.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

While many programmers are familiar with the experience of stumbling upon an edge case which wasn’t covered in the spec. The truth is you can’t go back and block on every single tiny detail.

Some of the bigger ones can trigger further discussion, but were it not for Falsehoods programmers believe about X the software would never go out the door.

Here’s some more food for thought…

1. Falsehoods programmers believe about names

o They won’t be a *bad* word. Whose bad? i’m not sure.
o All people have names.
o We only have to work with names from English.
o Names are ordered in a sensible manner.
o A name won’t have to change later.

You get the picture. As you start working with the minutae around names, you realize there is a lot of assumptions. Even when you weed out the big ones, there are still small assumptions.

Ultimately, you can’t get to the bottom. There will always be some assumptions, and as software evolves, changes will be required to match the real world. Living systems grow. Software systems must receive *updates*.

Read: What happened when I offered advice outside my pay grade?

2. Falsehoods programmers believe about selling products

o each product has a price
o each product has one price
o each product is one unit
o two decimal places is enough for all currencies
o all currencies have a unicode symbol

There are many more. The point is once you start digging, everything gets fuzzy. Products can be in stock and not in the warehouse. Single product can have multiple prices. It goes on and on!

Read: What did Matt Ranney discover scaling Uber to 1000 microservices?

3. Falsehoods programmers believe about addresses

o two districts will not have the same number
o postal codes will agree across jurisdictions
o a fixed number of characters will work for every address
o roads have names
o buildings are not numbered zero

Addresses are *really* messy. I remember being surprised when I moved to brooklyn. Streeteasy & Google put me in different neighborhoods. I asked my lawyer why this was. He said DOB and the post office use different databases. And google further samples from one or the other.

Never the twain shall meet!

Related: Can humility help you in your career?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Categories
All Consulting CTO/CIO Devops Hiring Serverless Trends

Is there a serious skills shortage around devops space?

via GIPHY

As devops adoption picks up pace, the signs are everywhere. Infrastructure as code once a backwater concept, and a hoped for ideal, has become an essential to many startups.

Why might that be?

Join 37,000 others and follow Sean Hull on twitter @hullsean.

My theory is that devops enables the business in a lot of profound ways. Sure it means one sysadmin can do much more, manage a fleet of servers, and support a large user base. But it goes much deeper than that.





Being able to standup your entire dev, qa, or production environment at the click of the button transforms software delivery dramatically. It means it can happen more often, more easily, and with less risk to the business. It means you can do things like blue/green deployments, rolling out featues without any risk to the production environment running in parallel.

What kind of chops does it take?

Strong generalist skills

For starters you’ll need a pragmatist mindset. Not fanatical about one technology, but open to the many choices available. And as a generalist, you start with a familiarity with a broad spectrum of skills, from coding, troubleshooting & debugging, to performance tuning & integration testing.

Stir into the mix good operating system fundamentals, top to bottom knowledge of Unix & Linux, networking, configuration and more. Maybe you’ve built kernels, compiled packages by hand, or better yet contributed to a few open source projects yourself.

You’ll be comfortable with databases, frontend frameworks, backend technologies & APIs. But that’s not all. You’ll need a broad understanding of cloud technologies, from GCP to AWS. S3, EC2, VPCs, EBS, webservers, caching servers, load balancing, Route53 DNS, serverless lambda. Add to all of that programmable infrastructure through CloudFormation or Terraform.

Related: 30 questions to ask a serverless fanboy

Competent programmer

Although as a devop you probably won’t be doing frontend dev, you’ll need some cursory understanding of those. You should be competent at Python and perhaps Nodejs. Maybe Ruby & bash scripts. You’ll need to understand JSON & Yaml, CloudFormation & Terraform if you want to deliver IAC.

Related: Does a 4-letter-word divide dev & ops?

Strong sysadmin with ops mindset

These are fundamental. But what does that mean? Ops mindset is born out of necessity. Having seen failures & outages, you prioritize around uptime. A simpler stack means fewer moving parts & less to manage. Do as Martin Weiner would suggest & use boring tech.

But you’ll also need to reason about all these components. That’ll come from dozens of debug & troubleshooting sessions you’ll do through years of practice.

Related: How to hire a developer that doesn’t suck

Understand build systems & deployment models

Build systems like CircleCI, Jenkins or Gitlab offer a way to automate code delivery. And as their use becomes more widespread knowing them becomes de rigueur. But it doesn’t end there.

With deployments you’ll have a lot to choose from. At the very simplest a single target deploy, to all-at-once, minimum in service and rolling upgrades. But if you have completely automated your dev, qa & prod infra buildout, you can dive into blue/green deployments, where you make a completely knew infra for each deploy, test, then tear down the old.

Related: Is AWS too complex for small dev teams?

Personality to communicate across organization

I think if you’ve made it this far you will agree that the technical know-how is a broad spectrum of modern computing expertise. But you’ll also need excellent people skills to put all this into practice.

That’s because devops is also about organizational transformation. Yes devs & ops have to get up to speed on the tech, but the organization has to get on board too. Many entrenched orgs pay lip service to devops, but still do a lot of things manually. This is out of fear as much as it stands as technical debt.

But getting past that requires evangelizing, and advocating. For that a leader in the devops department will need superb people skills. They’ll communicate concepts broadly across the organization to win hearts and minds.

Related: Will Microservices just die already?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Categories
All Cloud Computing Cloud Migrations CTO/CIO Database Management Devops NoSQL

30 questions to ask a serverless fanboy

Everyone is hot under the collar again. So-called serverless or no-ops services are popping up everywhere allowing you to deploy “just code” into the cloud. Not only won’t you have to login to a server, you won’t even have to know they’re there.

As your code is called, but cloud events such a file upload, or hitting an http endpoint, your code runs. Behind the scene through the magic of containers & autoscaling, Amazon & others are able to provision in milliseconds.

Join 32,000 others and follow Sean Hull on twitter @hullsean.

Pretty cool. Yes even as it outsources the operations role to invisible teams behind Amazon Lambda, Google Cloud Functions or Webtask it’s also making companies more agile, and allowing startup innovation to happen even faster.

Believe it or not I’m a fan too.

That said I thought it would be fun to poke a hole in the bubble, and throw some criticisms at the technology. I mean going serverless today is still bleeding edge, and everyone isn’t cut out to be a pioneer!

With that, here’s 30 questions to throw on the serverless fanboys (and ladies!)…

1. Security

o Are you comfortable removing the barrier around your database?
o With more services, there is more surface area. How do you prevent malicious code?
o How do you know your vendor is doing security right?
o How transparent is your vendor about vulnerabilities?

Also: Myth of five nines – Why high availability is overrated

2. Testing

o How do you do integration testing with multiple vendor service components?
o How do you test your API Gateway configurations?
o Is there a way to version control changes to API Gateway configs?
o Can Terraform or CloudFormation help with this?
o How do you do load testing with a third party db backend?
o Are your QA tests hitting the prod backend db?
o Can you easily create & destroy test dbs?

Related: 5 ways to move data to amazon redshift

3. Management

o How do you do zero downtime deployments with Lambda?
o Is there a way to deploy functions in groups, all at once?
o How do you manage vendor lock-in at the monitoring & tools level but also code & services?
o How do you mitigate your vendors maintenance? Downtime? Upgrades?
o How do you plan for move to alternate vendor? Database import & export may not be ideal, plus code & infrastructure would need to be duplicated.
o How do you manage a third party service for authentication? What are the pros & cons there?
o What are the pros & cons of using a service-based backend database?
o How do you manage redundancy of code when every client needs to talk to backend db?

Read: Why were dev & ops siloed job roles?

4. Monitoring & debugging

o How do you build a third-party monitoring tool? Where are the APIs?
o When you’re down, is it your app or a system-wide problem?
o Where is the New Relic for Lambda?
o How do you degrade gracefully when using multiple vendors?
o How do you monitor execution duration so your function doesn’t fail unexpectedly?
o How do you monitor your account wide limits so dev deploy doesn’t take down production?

Also: Are SQL databases dead?

5. Performance

o How do you handle startup latency?
o How do you optimize code for mobile?
o Does battery life preclude a large codebase on client?
o How do you do caching on server when each invocation resets everything?
o How do you do database connection pooling?

Also: Is Amazon too big to fail?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Categories
All iHeavy Newsletter Software Development

Why the Android ecosystem is broken

Six months ago I got this crazy idea. Why not leave the mothership? Give up on iPhone and try Android. This is what being tech-agnostic is about, I thought––not being wedded to a single platform. Besides, the iPhone’s digital keypad just wasn’t working for me.

I got a monthly Boost Mobile plan, which uses Sprint. Service was stellar and I mean really good. I could call anywhere and always had a signal, even inside all these pre-war buildings you find in downtown Manhattan. How is this possible I thought? Service is one thing, but thats where the fun ends. A few months into Android hyperspace and I find myself grappling with a system that just doesn’t seem to understand what users want.

Shock and Awe

On Android – first Samsung Transform Ultra then Sidekick 4G I found the app store was like wild, wild west. Buggy apps sat along well tested ones, and only a very discerning eye and mobile guru might know the difference. Syncing was absolutely horrible. The whole platform assumes you want to sync up with Google accounts. I went with Missing Sync from Mark/Space. My addressbook started getting corrupted, duplicate records appeared, syncs would fail halfway through.

What’s more there were tons of started services I didn’t even use like Smart Navigation, Group Texting, and so on. These services seemed to run in the background, hog & bleed memory, and slow down my phone til it started crashing. I actually had to download an app called Easy Task Killer. Apparently a very popular app on the Android phones, I wonder why?

Later on I found out that T-mobile was no longer supporting the Sidekick. No wonder it was so buggy. I can’t believe they’d ship something like this.

My full list of beefs:

  • corrupted data
  • slow to non-functional syncing
  • dangerous apps
  • sharing of private data
  • ineffective calendaring
  • no support for addressbook groups
  • apps not remembering context & position
  • no good email app
  • weaker less feature rich apps
  • kludgy interface

I’ve long since quelled my desire for a physical keyboard. I was struggling with every other thing I would do with the device. Sometimes I’d just give up.

I really wanted to get along with my Android phone but my experience with it only gave me an insight into three crucial areas where it falls short.

  1. An Iron Fist
  2. People complain about Apple’s iron fist in app store approval. You can’t have it both ways. Android completely lacks discipline and users suffer hugely because of it. That weakens the platform.

  3. A manageable set of devices
  4. Developers building for Android must test on a huge spectrum of hardware. Smaller shops are likely to choose a few of the biggest ones only. Phones with a smaller user base likely have a lot of bugs just in the Android version they run. All this bodes badly as users just see buggy software, they don’t know why or how. This perception further weakens the platform.

  5. Affordable development

Building apps costs businesses money. Businesses must balance the costs of building features, test, debug, troubleshoot & release. That’s cheaper on the iphone because you have one device that is much more mature. This has a pile on affect as it strengthens the platform, users spend more on their devices, more users pile onto the iphone platform, so more money can be made building an iphone apps.  On Android higher costs, lower margins further weakens the platform.

A new love for Apple

This whole experience has brought me back to the iPhone 4S, and I have a whole new love and appreciation for the platform and the device.

  • calendar reminders make me more productive
  • data is always right, and where I need it
  • I don’t fumble with menus – I’m faster & less frustrated
  • cross-platform apps are more feature rich on the iphone
  • complex features are hidden in plain view – superb interface
  • pulse, hootsuite, yelp, notes & calendar are all integrated & productive

I also learned that sometimes less is more… much more!