Can misfits teach us a thing or two about innovation?

I just finished reading Alexa Clay & Kyra Maya Phillips tour de force, The Misfit Economy.

(Yes that’s an affiliate link. The first one I’ve ever posted on this blog. If you like the book, please 🙂 buy through my link. )
I have to admit I was surprised & delighted by the book.

Join 38,000 others and follow Sean Hull on twitter @hullsean.

Alexa & Kyra offer us a tantalizing question. Could it be that we could learn a lot from oddball innovators at the edge of the economy? When I say edge, I really mean it. She interviews Sam Hostetler who is building a business around milking camels, and then there’s Abdi Hasan a pirate from Galkayo northern Somalia. Yeah really! Or what about the German copycats Wimdu who built a complete replica of Airbnb by reverse engineering it.

1. Hack the cold call

Take the example of Lance Weiler. Early on the industry was very against digital. They didn’t see it as really making films.

“Part of [Lance] Weiler’s success was due to his ability to work the system. He wrote letters to major production companies telling them he wanted to make the first digital motion picture. After he didn’t hear back, he took a page from the con man’s handbook and wrote the same letters but intentionally misaddressed them so they were sent to the wrong companies. Sony for example would get a letter intended for Barco.”

He was later able to bring digital projection to Cannes & Sundance!

“For Weiler his big epiphany was when he realized he could be creative across all of it [the business]. Not just in the art product, but in financing, distribution, and business aspects of artistic production.”

Related: The art of resistence or when you have to be the bad guy

2. Copy the product

The german brothers Oliver, Marc & Alexander Samwer make a superb example of how copying can bring building prowess to compete against innovators that were first to market.

“in 1998 Marc Samwer had an instinct that eBay would thrive in the German market… his brothers agreed… they contacted eBay via email numerous times, recommending that the company replicate it’s platformin Germany. Claiming that eBay failed to respond, the brother’s started their own German-language auction site, Alando, which was then purchased by eBay for 38 million euros (over $50 million) only 100 days after it’s debut. Had the Samwers not copied, eBay might have remained complacent, not realizing its potential within the german market.”

Although not mentioned in the book, Inditex the wildly successful firm behind fashion brand Zara did much the same thing to the fashion industry. By mastering the supply chain, they enabled their company to take designs from the runway & replicate them, turning designs into real clothing in stores, in just two weeks! And indeed they really do replicate, borrow & straight copy those designs from what they see at fashion week. Sad & brilliant at the same time.

Related: When you have to take the fall

3. Don’t forget to hustle


“In the lexicon of the Misfit Economy, we define “hustle” as making something out of nothing. To move fast, to trade one thing for another, and to proactively create your own opportunities rather than waiting for opportunity to come your way. To hustle means getting your hands dirty, being lean and facile, working hard, being resourceful and resilient, and showing or having gumption, chutzpah, or mojo.”

And after all, isn’t that everything the startup industry aspires to? Agile teams? Growth hackers? Scrappy startups & innovation?

Related: When clients don’t pay

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Can daily notes help you work better with clients?

via GIPHY

Years ago I was working at a customer for a few weeks. There was some confusion as to what was going on, in terms of progress. Things weren’t moving as quickly as they expected.

After a lot of back and forth, I suggested I could provide detailed notes of what I had done.

Join 38,000 others and follow Sean Hull on twitter @hullsean.

After I put together my in-depth notes the customer was really happy. It seems these notes had highlighted a few problems that they didn’t know about. What’s more they even highlighted some people issues, where communicate was blocked. Whats more the notes underlined what I was doing, and this really improved the customers confidence in the work product.

1. Visibility

Keeping daily notes is a habit I found useful over and over again. If your client or customer comes to you and says, why are we paying $X, you can provide the notes as a detailed explanation of what they have gotten for their money.

Related: Are generalists better at scaling the web?

2. Transparency

Transparency is a door that swings both ways. As I mentioned above it can be great when the customer is not sure how much work was done, or what the bill is for. But it can also highlight things they may not want done. For instance perhaps you were investigating a problem authenticating to a server. You determined that it was an important piece.

When the customer sees this in your notes they may say “Oh we don’t need to deal with that system. Please leave it alone” or they may say “We actually have Rakesh available to help us with that piece, so please communicate with him and he can resolve that”.

Related: When clients don’t pay

3. Trust

Most important of all, keeping detailed notes helps build trust. Many customers, hiring managers & CTOs are not command-line technical. And that’s perfectly normal. However looking over a long list of notes like these provides great insight to them as to what you do from day-to-day.

Do they need to know what every line means? No. But the visibility goes a long long way toward building trust in the consultant client relationship.

Related: 5 conversational ways to evaluate great consultants

Week 1 April 1 – April 10

Here’s a sample of the kind notes I keep. Actually they cover a ten day period, but that’s because the initial day was towards the end of a week.

Friday April 1st
o coord with Jake on getting started
o dropbox for password, creds & server docs
o reviewing system network diagram
o reviewing techlist excel doc
– techlist
– server list & access
– database access
– projects -old
o reviewing systems access.docx
o testing AWS login credentials
– issue with permissions
– coordinating with Jake on Admin access
o testing AWS creds again
– access to all AWS services
– IAM for seanhull user
– enabling MFA for user
o questions for outgoing op Roger

Sat April 2nd (no hours)

Sun April 3rd (no hours)

Mon April 4th
o coord with Jake to get onboarded
o sending W9 form to Acme Inc.
o setup slack
o plan for today
– review aws servers
– review dg servers
– questions for Roger
– review docs
o coord with Roger on VPN access
– reach out to Larry
– emailed Larry CC Jake
– Larry requests Acme access CC to mgmt
– turns out VPN access isn’t required
– can just whitelist IP inside the relevant security groups
– coord with J, going ahead to add whitelist 1.2.3.4/32
o updating Acmemedia-sandbox security group
– trying to reach host, coord with Roger
– asked to drop ssh key onto servers
– asked about .ssh/config file – Did you get from Jake?
– found the AWS PEM folder that I overlooked 🙂
o configuring .ssh/config file
– copying up to iheavy.com
– setting permissions 600 on pem files
– ssh to sandbox successful!!
o adding whitelist to Acmemedia-prod security group
o updating Jake – access is working

Tue April 5th
o coord with Jake on todo list for today
o verifying mysql access
– review security groups
– no whitelisted IPs
– can reach from webserver?
– test db1 MySQL access via webserver, OK
– test db2 MySQL access via webserver OK
o reviewing monitoring system
– testing nagios access
– locating configurations
– reviewing dashboard
– understanding tests
– down system db1 – 108 days – why?
– down system p1 – LB1 sailthru check down for 85 days why?
– down staging – 174 days why?
– emailed nagios questions to Roger
– request to add me to nagios notifications group
o coord with Roger on questions
– nagios setup & stopped checks
– add to admin group
o github access for sandbox details doc
o login to Acmemedia wp
– check list of 25 plugins
– review recent backups on abc (8)
o login to DDD wordpress
– check list of 33 plugins
– review recent backups in abc (8)
o login to EEE wordpress
– check list of 31 plugins
– review recent backups in abc (37)
o login to FFF wordpress
– check list of 35 plugins
– review recent backups in abc (8)
o login to DDD
o login to EEE
o login to FFF
o emailed Roger – request details about Glasgow server
o review various Acme github pages

Also: The art of resistance or when you have to be the bad guy

Wed April 6th
o coord with Jake on todos for today
o reviewing github pages docs on various system processes
– git deployment server page
– git deployment process
– new deploy process Nov 2015
– wiki pages are a bit sparse overall
o tested jenkins login
– found API cache clear
– found varnish cache clear
o understand separation of dev & production
o digging into Jenkins docs
o understanding build process
o tried login to EEEv2 wp login, don’t have pass
– coordinating with Jake on that login
o checking on nfs disk full nagios alert
– can’t reach box
– notified Jake & Roger via slack
– slack with Lester
– yes nfs01 space 90% is normal
– new launch of EEE tomorrow & old stuff will be deleted then
o updating nfs security group
– ssh login working now.
o getting diskspace error on prod04
– messaged Lester, related to EEE launch tonight
o email from Jake – local dev & test environment setups are slow
– very overengineered for simple wordpress site
– not using multisites, so have FOUR SEPARATE setups
– different plugins on each install
– four sets of logins
– four places to update
– four places to test/qa
– migration may be complex based on custom Acme plugins
– shortcodes compatability across four sites
– not using ithemes security plugin
o discuss with Lester on slack
– API is hosted on datagram
– single point of failure for the site currently
– outage there would take the site down
– migrate to AWS using internal loadbalancer & webservers in 2 AZs

Thu April 7th
o call with Jake on EEEv2 launch today
– general observations of Acme sites & architecture
o reviewing access.Acmemedia.com
o discuss with Jake
– hosting media files on S3 vs nfs
– using multisite
– using wordpress through API only
– javascript based static site builder
– moving API to amazon EC2
– create slave MySQL db of master MySQL currently in datadotnet
o discuss with Roger
– launch plan
– two vhosts new.EEE.com
– old.EEE.com
– simply restart apache to enable switch
– refresh maxCDN after launch
o review EEEv2 deploy steps
– pre-deploy steps
– DNS for old.EEE.com
– add vhosts EEEv2.conf
– restart apache
– restart varnish
– clear maxcdn
o verified login to access.Acmemedia.com
– API log is in /var/log/httpd/production-access.log
– login as sandy & root
o not able to login to dashboard.Acmemedia.com
– tried admin & pass in datagram docs

o meeting onsite with Jake & Roger
– discuss deployment process
– discuss legacy systems
– discuss NFS vs S3 for media files
– discuss plugins & management
– discuss wordpress version upgrade process
– discuss plugin version upgrade process
– discuss Jenkins access, configs, success & error logs
– discuss managing secrets file
– script that takes webserver out of load balancer while apache restarting
o met Rachel, Louis, Lester, Rick, Stuart, Jack

Fri April 8th
o testing Acme stage build
o emailed Roger further questions
– where is secrets file configuration & process
– composer is PHP dependency management
– what are the steps to upgrade plugin only
o summarizing & notes on Acme
o put together steps for complete firedrill
– questions for Roger, requesting help with process
– build webserver with varnish & apache
– should setup separate NFS server
– should use Acmemedia.com bc it uses API heavily
– setup copy of API server & db
– setup mysql instance for wordpress
– setup amazon cloudfront for content
o outline additional questions for Roger
– how to upgrade plugin only
– composer for php dependency management
– how are secrets files managed & deployed outside developer access
o secrets management
– asked Roger for clarificaiton
o plugin-only installs
– reviewed jenkins configs
– various questions to Roger
– composer:install seems to be the key change (not just deploy which does all?)
– why is STAGING PLUGIN DEPLOY for ORIM different?
o what happens when github account is disabled!!
– jenkins changes for new github deploy account
– THIS WOULD BREAK ALL DEPLOYS & CI/CD pipeline
– capistrano changes?
– any other changes on sandbox
– any other dependencies for Roger github?
o email step-by-step outline to add a plugin
– reviewing steps with Roger
– making sure no missing pieces

Sat April 9th (1 off-hour)
o receiving nagios alert for p1
o emailed Roger, Jake about issue
o slack messaged Jake
o raises question about off-hours coverage

Sun April 10th (2 off-hours)

o p1 still throwing errors
o coordinating with Lester & Ralph on Slack
– reiterated this is *not* an issue with NFS
– because of large number of nagios alerts, p1 lost in the shuffle
– p1 is new error, 97% so more dire than the NFS issue
– Lester attempting to login, fails because of AWS security group
– adding his *own* home IP as whitelist (devs have access to AWS console)
– first time logging in from home?
– Lester deleted old DDD logfiles to clear up 1.2G
– plan to touch base again tomorrow about issue
o emailing Jake about status
o questions for Jake
– how to manage on-call & alerts
– how to manage developer access
– Roger mentioned secrets files are not shared with devs
o Lester questions, comments on servers & diskspace

Related: When you have to take the fall

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

What have I learned in 10 years of blogging?

via GIPHY

I was just reading Andrew Chen’s latest posting, where he distills many of the things he’s learned from blogging over a decade.

Join 38,000 others and follow Sean Hull on twitter @hullsean.

This reminded me that I’ve been blogging that long as well. And to be sure it has brought great benefits. In the way that public speaking gives you visibility, but also forces you to communicate better, form your voice, and so on.

All the great things you gain by talking to other people, and getting into the conversation.

1. Understand your audience

I struggled with this when I first started blogging. As any engineer might approach things, I thought I should publish technical material. What better way to show what I know. And further how I can help a customer.

What I didn’t realize is that all of your readers aren’t technical. So it goes a long way if you can appeal to a broader audience.

I found that my readers fell into a few big categories.

1. Fellow engineers & peers
2. Hiring managers & startup CTOs
3. Recruiters & other publishers

This really helped me divide up the types of content I would write, some directed towards each of the different audiences.

Related: Why does Reddit CTO Martin Weiner advocate boring tech?

2. Tell your story

I’ve written often about why I wrote the book on Oracle. In it I outlined a long arc of datacenter evolution which started with the maturity of Linux, and today provides the bedrock of the cloud that is Amazon Web Services among others.

What this also allowed me to do is tell my own history.

Related: 5 reasons devops should blog

3. Form your voice

Forming your voice is different than speaking to specific audiences. It’s about having opinions & getting into the line of fire. Being passionate about a subject, you’re sure to care & sit on one side or the other of a particular argument.

For example I argued the Android ecosystem was broken. Although Google has fixed some of these problems, many remain as a symptom of the platform itself.

I also argued with Fred Wilson’s estimation of Apple being overvalued. At the time in May 2014 the price was at $85. Now it sits comfortably at $177.

Related: How to hire a developer that doesn’t suck

4. Put yourself out there

Putting yourself out there isn’t easy. You’ll be open to criticism. And sometimes you’ll be wrong. But by challenging yourself in this way you’ll grow too. And prospects will notice this. More than engineering might, and power at the keyboard, your perspective of what’s happening in computing generally, and what is on the horizon is invaluable to customers.

Related: 30 questions to ask a serverless fanboy

5. Learn & Share

Writing howtos is a great challenge too. By forcing yourself to teach something, you in turn learn the material better. You become better at executing, and formulating solutions.

As you share knowledge, you’ll also learn from others. As the disqus.com comments on my site can attest. Sure you get much of this same value from having an active account on Reddit.com, but your own real estate carries even more weight for your personal brand.

Related: Why you should always be publishing

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Is Alex Hudson right that software architecture is failing?

via GIPHY

I read Hacker News aka Ycombinator’s popular top 100. I never fail to find useful, surprising & stimulating reading there.

I recently stumbled on Alex Hudson’s software architecture is failing.

It’s very good, I recommend reading it.

Join 38,000 others and follow Sean Hull on twitter @hullsean.

But why did it grab my attention, you might ask? Perhaps I’m a naysayer. But I do find there is a lot of hype, and a lot of sex in software today. It’s as though the shiniest, newest, coolest toys are the ones getting the spotlight.

So when I find an alternative view, I sit up and take notice.

1. Are we making systems too complex?

Right out of the gates, Alex makes a great point:

“We’re not delivering quickly enough!”. “Our systems are too complex to maintain!”. “The application we delivered last year is completely legacy now but it’s too difficult to replace!”.

Our industry’s obsession with the newest & coolest toys, means we’re building things that don’t last very long. A real & ongoing problem.

Related: Why does Reddit CTO Martin Weiner advocate boring tech?

2. Smaller enterprises

One thing Alex pointed out that really struck a nerve was this:

For those in tech who are not working at Facebook/Google/Amazon, we’re simply not talking enough about what systems at smaller enterprises look like.

I couldn’t agree more. As a profession, we watch closely at what the big guys are doing. And that’s useful to a point. But for many smaller companies, to use such architectures would be over engineering in the extreme. Not to mention extremely costly!

Related: How I use terraform & composer to automate wordpress on AWS

3. Not bleeding & far from the edge

Another choice quote from Alex’s piece:


“It’s totally legacy, and no-one maintains it – it just sits there working, except for the occasions it doesn’t. The problem is replacing it is so hard, it’s got great performance, and the business doesn’t want to spend time replacing something working”. This is the problem being ahead of the curve – the definition of “success” (it works great, it’s reliable, it’s performant, we don’t need to think about it) looks a hell of a lot like the definition of “legacy”.

We know the term bleeding edge because it’s tough being out there trail blazing. Here I agree that sometimes legacy is also boring, yet eminently reliable.

Related: 30 questions to ask a serverless fanboy

4. Reduce, reuse, recycle

Should we build it or should we buy it? Here’s what Alex says:


I think we’re often getting the build/buy decision wrong. Software development should be the tool of last resort: “we’re building this because it doesn’t exist in the form we need it”.

Well said. Sure we should consider integration costs & testing. And using a service brings other things to balance. But it means we don’t have to own that code.

Better to focus on our business core competency.

Related: Is Amazon about to disrupt your data warehouse?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

How do I migrate my skills to the cloud?

via GIPHY


Hi, I’m currently an IT professional and I’m training for AWS Solutions Architect – Associate exam. My question is how to gain some valuable hands-on experience without quitting my well-paying consulting gig I currently have which is not cloud based. I was thinking, perhaps I could do some cloud work part time after I get certified.

Join 38,000 others and follow Sean Hull on twitter @hullsean.


I work in the public sector and the IT contract prohibits the agency from engaging any cloud solutions until the current contract expires in 2019. But I can’t just sit there without using these new skills – I’ll lose it. And if I jump ship I’ll loose $$$ because I don’t have the cloud experience.


Hi George,

Here’s what I’d suggest:

1. Setup your AWS account

A. open aws account, secure with 2FA & create IAM roles

First things first, if you don’t already have one, go signup. Takes 5 minutes & a credit card.

From there be sure to enable two factor authentication. Then stop using your root account! Create a new IAM user with permissions to command line & API. Then use that to authenticate. You’ll be using the awscli python package.

Also: Is Amazon too big to fail?

2. Automatic deployments

B. plugin a github project
C. setup CI & deployment
D. get comfy with Ansible

Got a pet project on github? If not it’s time to start one. 🙂

You can also alternatively use Amazon’s own CodeCommit which is a drop-in replacement for github and works fine too. Get your code in there.

Next setup codedeploy so that you can deploy that application to your EC2 instance with one command.

But you’re not done yet. Now automate the spinup of the EC2 instance itself with Ansible. If you’re comfortable with shell scripts, or other operational tools, the learning curve should be pretty easy for you.

Read: Is AWS too complex for small dev teams? The growing demand for Cloud SRE

3. Clusters

E. play around with kubernetes or docker swarm

Both of these technologies allow you to spinup & control a fleet of containers that are running on a fixed set of EC2 instances. You may also use Amazon ECS which is a similar type of offering.

Related: How to deploy on EC2 with Vagrant

4. Version your infrastructure

F. use terraform or cloudformation to manage your aws objects
G. put your terraform code into version control
H. test rollback & roll foward infrastructure changes

Amazon provides CloudFormation as it’s foundational templating system. You can use JSON or YAML. Basically you can describe every object in your account, from IAM users, to VPCs, RDS instances to EC2, lambda code & on & on all inside of a template file written in JSON.

Terraform is a sort of cloud-agnostic version of the same thing. It’s also more feature rich & has got a huge following. All reasons to consider it.

Once you’ve got all your objects in templates, you can checkin these files into your git or CodeCommit repository. Then updating infrastructure is like updating any other pieces of code. Now you’re self-documenting, and you can roll-forward & backward if you make a mistake!

Related: How I use terraform & composer to automate wordpress on AWS

5. Learn serverless

I. get familiar with lambda & use serverless framework

Building applications & deploying only code is the newest paradigm shift happening in cloud computing. On Amazon you have Lambda, on Google you have Cloud Functions.

Related: 30 questions to ask a serverless fanboy

6. Bonus: database skills

J. Learn RDS – MySQL, Postgres, Aurora, Oracle, SQLServer etc

For a bonus page on your resume, dig into Amazon Relational Database Service or RDS. The platform supports various databases, so try out the ones you know already first. You’ll find that there are a few surprises. I wrote Is upgrading RDS like a sh*t storm that will not end?. That was after a very frustrating weekend upgrading a customers production RDS instance. 🙂

Related: Is Amazon about to disrupt your data warehouse?

7. Bonus: Data warehousing

K. Redshift, Spectrum, Glue, Quicksight etc

If you’re interested in the data side of the house, there is a *LOT* happening at AWS. From their spectrum technology which allows you to keep most of your data in S3 and still query it, to Glue which provides an ETL as a service offering.

You can also use a world-class columnar storage database called Redshift. This is purpose built for reporting & batch jobs. It’s not going to meet your transactional web-backend needs, but it will bring up those Tableau reports blazingly fast!

Related: Is Amazon about to disrupt your data warehouse?

8. Now go find that cloud deployment job!


With the above under your belt there’s plenty of work for you. There is tons of demand right now for this stuff.

Did you do learn all that? You’ve now got very very in-demand skills. The recruiters will be chomping at the bit. Update those buzzwords (I mean keywords). This will help match you with folks looking for someone just like you!

Related: Why I don’t work with recruiters

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

What’s the *real* way to deploy on Google Cloud?

via GIPHY

I was talking to a customer recently and they asked about deployments. They wanted to do things the real way. Here’s a snippet…

I’m helping out a company called Blue Marble and they are getting ready to deploy a new POS system. The app has been built using a Node.js back-end and Google Cloud Datastore for storage. The current dev build is hosted on AWS and connects to Google for the data bits.

Join 38,000 others and follow Sean Hull on twitter @hullsean.


For prod launch, they are interested in migrating to the “real” way of deployment on Google for everything.

They are pressed on time and looking for someone who can jump in quickly. Are you available? Do you have Google Cloud expertise?

Here’s what I said.

Cultural hurdles


Yep, I’ve have used Bigquery & GCE.

What are they looking for specifically? Full deployment automation? Multiple deploys per day?

I’ve found that sometimes the biggest hurdle to fully automated deploys can be cultural issues.

In other words yes you can automate your deployment so it is push button, get all the artifacts & moving parts automated. Then deploy without much intervention. But to go from that to the team having *faith* in the system, that is a challenge.

Also: Why would I help a customer that’s not paying?

Unit testing


Once the process has been streamlined, a lot often still needs to happen around unit & smoke tests.

If the team isn’t already in the habit of building tests for each bit of code, this may take some time. Also building tests can be an art in itself. What are the edge cases? What values are out of bounds?

Consider for example odd vulnerabilities that show up when hackers type SQL code into fields that devs were expecting. Sanity checking anyone?

Read: Is AWS too complex for small dev teams? The growing demand for Cloud SRE

Integraton testing

What makes this all even more complicated is integration testing. Today many application use various third party APIs, service-based authentication, and even web-based databases like Firebase. So these things can complicate testing.

Related: How to build an operational datastore on Amazon Redshift with S3

Getting there

Although your project, startup or business may be pressed for time, that may not change the realities of development. Your team has to become culturally ready to be completely agile. Many teams choose a middle ground of automating much of the deployment process, but still having a person in the loop just in case.

Same with testing. Sure automating can make you more agile & more efficient. But you’ll never automate out creative thinking, problem solving & ownership of the product.

Related: Why did Flatiron School fail?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Is maintenance as sexy as innovation?

via GIPHY

A recent NYT piece on our aging american infrastructure got me thinking. It seems that roads, bridges, airports & city sewer systems are all in need of repair. Sadly as budgets to maintain these systems in good repair are often short, they become larger problems to fix as their status becomes critical.

Join 37,000 others and follow Sean Hull on twitter @hullsean.

“Americans have an impoverished and immature conception of technology, one that fetishizes innovation as a kind of art and demeans upkeep as a mere drudgery.”

I’m not sure this is an American-only phenomenon. However I do see it a lot with technology companies & startups.

1. Do we have to manage ops in the cloud?

The cloud has enabled infrastructure automation in some pretty phenomenal ways. Code pipelines can deliver changes to a repo, through automated unit testing, and out to customers all without human intervention. This makes teams more agile, and ultimately businesses faster & more profitable.

We might be distracted enough to stop worrying about operations altogether. After all Amazon knows how to manage broken servers & alert us right? I write do we have to manage operations in the cloud previously, as this sentiment seems to be growing.

Modern applications have a ton of interdependencies. Even with decent integration testing, the full stack is complex, and requires monitoring. Co-tenancy can complicate your performance tuning efforts as neighboring customers may directly affect your application. Third party services may be delivered from smaller or less experienced companies, whose SLA may be limiting besides. And hey if Amazon goes down, I can just tell my customers it was their fault, right?

Also: Is Amazon too big to fail?

2. Do you know Dustin Moskovitz?

Chances are I’m guessing you’ll say no. He was part of the original Facebook team alongside Zuckerberg. You don’t know his name? He had the sexy job of, you guessed it maintenance! He was the operations guy. Did he write the application code? More than likely he knew that code very well as he had to fix & maintain it. Along with the infrastructure to scale & support Facebook’s massive growth.

Read: Is AWS too complex for small dev teams? The growing demand for Cloud SRE

3. Is a little technical debt ok?

Ward Cunningham has an excellent interview about technical debt. Is a little bit ok? Maybe. But each amount is kicking the can down the road. As the NYT article on maintenance makes clear, you can move the responsibility on to the next administration, the next term, or someone else, but eventually you’ll have a critical problem on your hands, which will be much more expensive to fix.

Related: How to build an operational datastore on Amazon Redshift with S3

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

How to interview an amazon database expert

via GIPHY

Amazon releases a new database offering every other day. It sure isn’t easy to keep up.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

Let’s say you’re hiring a devops & you want to suss out their database knowledge? Or you’re hiring a professional services firm or freelance consultant. Whatever the case you’ll need to sift through for the best people. Here’s how.

Also: How to interview an AWS expert

What database does Amazon support for caching?

Caching is a popular way to speed up access to your backend database. Put Amazon’s elasticache behind your webserver, and you can reduce load on your database by 90%. Nice!

The two types that amazon supports are Memcache & Redis. Memcache is historically more popular. These days Redis seems a clear winner. It’s faster, and can maintain your cached data between restarts. That will save you I promise!

Also: Is AWS too complex for small dev teams?

How can I store big data in AWS?

Amazon’s data warehouse offering is called Redshift. I wrote Why is everyone suddenly talking about Redshift?. Why indeed!

When you’re doing large reports for your business intelligence team, you don’t want to bog down your backend relational database. Redshift is purpose built for this use case.

I’ve see a report that took over 8 hours in MySQL return in under 60 seconds in Redshift!

A new offering is Amazon Spectrum. This tech is super cool. Load up all your data into S3, in standard CSV format. Then without even loading it into Redshift, you can query the S3 data directly. This is super useful. Firstly because S3 is 1/10th the price. But also because it allows you to stage your data before loading into Redshift itself. Goodbye Google Big Query! I talked about spectrum here.

Related: Which engineering roles are in greatest demand?

What relational database options are there on Amazon?

Amazon supports a number of options through it’s Relational Database Service or RDS. This is managed databases, which means less work on your DBAs shoulders. It also may make upgrades slower and harder with more downtime, but you get what you pay for.

There are a lot of platforms available. As you might guess MySQL & Postgres are there. Great! Even better you can use MariaDB if that’s your favorite. You can also go with Aurora which is Amazon’s own home-brew drop in replacement for MySQL that promises greater durability and some speedups.

If you’re a glutton for punishment, you can even get Oracle & SQL Server working on RDS. Very nice!

Read: Can on-demand consulting save startups time & money?

Does AWS have a NoSQL database solution?

If NoSQL is to your taste, Amazon has DynamoDB. According to . I haven’t seen a lot of large production applications using it, but what he describes makes a lot of sense. The way Amazon scales nodes & data I/O is bound to run into real performance problems.

That said it can be a great way to get you up and running quickly.

Read: Can on-demand consulting save startups time & money?

How do I do ETL & migrate data to AWS?

Let’s be honest, Amazon wants to make this really easy. The quicker & simpler it is to get your data there, that more you’ll buy!

Amazon’s Database Migration Service or DMS allows you to configure your old database as a data source, then choose a Amazon db solution as destination, then just turn on the spigot and pump your data in!

ETL is extract transform and load, data warehouse terminology for slicing and dicing data before you load it into your warehouse. Many of todays warehouses are being built with the data lake model, because databases like Redshift have gotten so damn fast. That model means you stage all your source data as-is in your warehouse, then build views & summary tables as needed to speed up queries & reports. Even better you might look a tool like xplenty.

Amazon’s new offering is called Glue. Five ways to get data into Amazon Redshift. This solution is purpose build for creating a powerful data pipeline, complete with python code to do transformations.

Read: Is data your dirty little secret?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

What does the fight between palantir & nypd mean for your data?

via GIPHY

In a recent buzzfeed piece, NYPD goes to the mat with Palantir over their data. It seems the NYPD has recently gotten cold feet.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

As they explored options, they found an alternative that might save them a boatload of money. They considered switching to an IBM alternative called Cobalt.

And I mean this is Silicon Valley, what could go wrong?

Related: Will SQL just die already?

Who owns your data?

In the case of Palantir, they claim to be an open system. And of course this is good marketing. Essential in fact to get the contract. Promise that it’s easy to switch. Don’t dig too deep into the technical details there. According to the article, Palantir spokeperson claims:

“Palantir is an open platform. As with all our customers, their data & analysis are available to them at all times in an open & nonproprietary format.”

And that does appear to be true. What appears to be troubling NYPD isn’t that they can’t get the analysis, for that’s available to them in perpetuity. Within the Palantir system. But getting access to how the analysis is done, well now that’s the secret sauce. Palantir of course is not going to let go of that.

And that’s the devil in the details when you want to switch to a competing service.

Also: Top serverless interview questions for hiring aws lambda experts

Who owns the algorithms?

Although the NYPD can get their data into & out of the Palantir system easily, that’s just referring to the raw data. That’s the data they ingested in the first place, arrest records, license plate reads, parking tickets, stuff like that.

“This notion of how portable your data is when you engage in a contract with a platform is really, really complex, and hasn’t really been tested” – Tal Klein

Palantir’s secret sauce, their intellectual property, is finding the needle in the haystack. What pieces of data are relevant & how can I present the detectives the right information at the right time.

Analysis *is* the algorithms. It’s the big data 64 million dollar question. Or in this case $3.5 million per year, as the contract is reported to be worth!

Related: Which engineering roles are in greatest demand?

The nature of software as a service

The web is bringing us great platforms, like google & amazon cloud. It’s bringing a myriad of AI solutions to our fingertips. Palantir is providing a push button solution to those in need of insights like the NYPD.

The Cobalt solution that IBM is offering goes the other way. Build it yourself, manage it, and crucially control it. And that’s the difference.

It remains to be seen how the rush to migrate the universe of computing to Amazon’s own cloud will settle out. Right now their in a growth phase, so it’s all about lowering prices. But at some point their market muscle will mean they can go the Oracle route a la Larry Ellison. That’s why customers start feeling the squeeze.

If the NYPD example is any indication, it could get ugly!

Read: Can on-demand consulting save startups time & money?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Do we have to manage ops in the cloud?

via GIPHY

One of the things that is exciting about the cloud is the reduced need for operations staff. There seem to be two drivers of this trend. One is devops, and all the automation that comes with it. As we formalize configurations, things become repeatable, and fewer people can manage greater armies of servers.

The second is by moving to a cloud hosting provider, we essentially outsource the operations to their team.

1. Pretty abstractions? still hardware buried somewhere

That’s right, beneath all the virtual EC2 instances & VPCs there is physical hardware. Huge datacenters sit in North Virginia, Oregon, Ireland, London and many other cities. Within them there are racks upon racks of servers. The hypervisor layer, the abstraction built on top of that, orchestrates everything.

Although we outsource the management of those datacenters to Amazon, there are still responsibilities we carry. Let’s dig into those more.

Also: Top serverless interview questions to ask an expert

2. Full-stack dev – demand for generalists?

These days we see the demand for a full stack developer. That is someone who does not only front end dev, but also backend. In turn, they are often asked to wear the had of ops. Spinup EC2 instance, decide on the capacity & size, choose proper disk I/O, place it within the right subnet & vpc & then configure the security groups properly.

All of these tasks would previously been managed by a dedicated ops team, but now those responsibilities are being put on developers shoulders. In some cases, such as with microservices, devs also carry the on-call duties of their applications.

Lastly there is likely ops to handle automation. Devops will formalize configurations, into ansible playbooks or chef recipes, so they can be checked into version control. At this point infrastructure can even be unit tested.

Read: Build an operational datastore on aws S3 with Spectrum

3. Design, resiliency, instrumentation, debugging

In previous eras, ops teams would be heavily involved with design of applications & architecture to support that. Now that may be handed to devs, but it still needs to happen.

Furthermore resiliency is said to be the customers responsibility. In the pre-cloud days, hardware was more reliable. It had a slower failure rate. With virtual machines, they’re expected to fail, and all the components to make your applications resilient are given to you. But it’s your job to architect them together.

That means your applications need to be self-healing. Failures need to be detected, taken out of autoscaling groups, and replaced. All automatically. Code or not, that is certainly operations.

Check this: Which engineering roles are in top demand?

4. It’s amazon’s fault we’re down!

I’ve seen quite a few outages in the past year, from Dropbox to Airbnb, and DYN themselves. Ultimately these outages could be tied back to a failure with Amazon. But when your business customers are relying on your service, it is *YOUR* business that answers to it’s own SLA.

In the news we see many of these firms pointing the finger at Amazon, “hey it’s not our fault, our cloud provider went down!”. Ultimately your customers don’t care. They don’t want excuses. If using multiple regions in AWS is not sufficient, you’ll need to build your application to be multi-cloud.

Also: 30 questions to ask a serverless fanboy

5. It’s hard to outsource your expertise

Remember, while you outsource your operations to Amazon, you’re getting very professional management of those systems. However they will optimize for their many customers. Your particular problems are less of a concern.

Read this: What can startups learn from the DYN DNS outage?

6. Only you can thinking holistically about interdependancies

Your application more than likely uses a number of APIs to capture data, perhaps do single sign on or even a third party database like Firebase. It’s your responsibility to do integration testing. All that becomes more complex in the cloud.

Also: How to lock down systems from outgoing employees

7. How do services complicate things?

SaaS solutions are everywhere now. auth0, firebase and an infinite variety of third party apis complicate reliability, security, storage, performance, integration testing & debugging?

Security is a traditional responsibility held by the operations hat. Much of that becomes more complex in the cloud. With serverless applications for example you may use a few APIs, plus an authentication broker, and a backend database. As this list of services grows, the code you write may decrease. But testing & securing it all becomes much more complex.

With more services like this, the attack vector or surface area becomes greater. Each of those services, can and will have bugs. What if a zero day is found in the authentication broker, allowing a hacker to break into a broad cross section of applications across the internet? How do you discover this? What if your vendor hasn’t found out yet?

Read: Is Amazon cloud too complex for small dev teams?

8. How does co-tenancy impact performance tuning?

Back to point #1 above, all these virtual servers sit on real physical servers. That affects customers in two ways. One you may be sharing the same host. That is if you use a very small vm, it may sit along side another customer with a small vm. If those eat up CPU cycles or network on that box, neighbors or co-tenants will suffer.

There are many other instance types where you get your own dedicated hardware. With those you have your own nic as well, so no competition. Except wait there’s network storage! That’s right all the machines in the AWS environment use EBS now, which is all co-tenant. So your data is sitting alongside other customers, and you are all fighting for usage of the same disk read heads.

One way to mitigate this is to configure specific provisioned IOPS for your servers. But that costs more. It’s normally reserved for database instances where disk I/O is really crucial.

Granted the NewRelics of the world will certainly help us with this process. But they’re not giving us a hypervisor or global view of those servers, network or storage. So we can’t see how the overall systems performance may be impacted.

Related: Is AWS a patient that needs constant medication?

9. Operations can be invisible

When security is done well, you don’t have breakins, you don’t have data stolen, everything just runs smoothly. Operations is like that too. When it is done well it can be invisible.

It can also be invisible in a different way. When you deploy your application on serverless, all the servers & autoscaling is completely abstracted away. So when you get some weird outage because the farm of servers is offline, or because you hit some account limit in the number of functions you can run at once, then it quickly comes into focus.

Beware of invisible operations, because it’s harder to see what to monitor, and know how to stay ahead of outages.

Read: Is amazon too big to fail?

10. We can’t oursource true ownership

At the end of the day you can’t outsource ownership of your application or your business. The holistic view of your application in totality can only be understood by your engineers.

And that in the end is what operations is all about, no matter who’s wearing the hat!

Also: 5 reasons to move data to amazon redshift

Get my monthly newsletter for more thoughts on data, startups & innovation. Scalability. Automation. Amazon cloud.