3. Tuning across languages can be very challenging
Different languages have different tools available. Can your favorite tool work across all the languages you’ve decided to support in your microservices?
Even though you’re going to go with microservices, think about standardizing across those services. Use tools that work on all of them. And when building your services, make sure it outputs the same dashboard metrics.
What do you think? Is it foolhardy? Or is there a hidden wisdom here?
Here’s my take.
1. Do you have one million customers testing your datacenter?
If you’re comparing the cost of the cloud to the raw numbers of running your own datacenter, the hardware costs are not enough. You’ll need to include the ops teams & other engineers. Right, you probably guessed that.
But did you factor in the costs of a legion of testers. This is the hidden cost that commercial software carries, even while open source software gets this benefit for free.
With a public cloud like AWS you have millions of customers testing the product everyday, and running into edge cases long before you do. So you get a better service, that’s more reliable, all invisibly for free.
2. Do you have 66 datacenters spread across 21 regions and a free network between them?
Anybody who was building web applications in the year 2000 will remember how websites didn’t load the same for different customers. Depending on where in the world they were located, they could experience a very different user experience.
These days we assume that we can be global from day one. But how exactly do we achieve this? Remember with a public cloud, you’re getting tons of things for free, without knowing it. Moving data between AZs or regions? That’s all going across a private interconnect.
And that’s not even including the 180 nodes inside cloudfront that give you a global CDN footprint too!
3. Do you have an engineering team automating away job roles?
I remember the days of DBA job role, do you? Probably not. I specialized in this for years, and there were tons of companies hiring me to help them with it. First Oracle, then MySQL, then Postgres.
Then along came Amazon RDS. Guess what, companies don’t really hire for that role anymore. They do need help with it from time to time, but not as a primary specialization.
What do I mean? Well by hosting your application on AWS, you’re benefiting from the work of teams of engineers in different departments, all expanding on APIs and automating things that those one million customers are asking for.
You’re not going to be able to innovate that well and that quickly in your own datacenter. So you’ll pay more!
A couple of years ago I worked at a startup in the online publishing space. But this story isn’t about online publishing. This story is about deadlines, and not meeting them.
For those who don’t know me, I’m a professional services consultant. I’ve worked on a project basis for 90% of two plus decades of my professional career. I mentioned this to give my opinions and perspectives some context. Although I’m not a manager, I’ve worked under more managers than most. Because I work on 5-10 projects per year, that comes to close to two hundred in my career.
My career path has given me a unique perspective, on teams, leadership and motivation. Of course like anyone who’s worked in tech, I’ve see a lot of agile.
Daily standups are de rigueur of course. As are breaking up your work into two week sprints, and assigning story points to those tasks.
I have to admit there are times when Agile seemed the most fashionable way I saw teams working. And I guess I believe it was more trendy than functional.
Doesn’t everybody already work off tight todo lists, break tasks down into manageable pieces, and always deliver what they promised? It may be because I’ve spent a lot of time managing myself, and surviving as a freelancer that I’ve picked up some of these habits. But I digress…
1. Crunch time
While we as a team had been working on two week sprints, something happened to derail us. Suddenly a major customer was in jeopardy, because we had not delivered on sales promises.
Although what was being promised, we *could* build, we were stumbling over the details.
As an emergency plan, we dropped our current sprint tasks, and marshaled our forces towards this new goal. Everyone on the team knew the customer was hanging by a thread.
While this edict looks great on paper, it didn’t work out so well. Developers and ops weren’t clear which systems were included in “production”, and how they needed to be connected together.
Each engineer ended up interpreting the goal in their own way, often assuming that others were shouldering ultimate responsibility of delivery.
“Well I did my part, this other part of the team is responsible for those other pieces…” was the refrain I heard. Sadly the deadline was missed, and everything was a mess. Only after management later dug in and sifted through the rubble, did they actually break up the tasks, assign story points, and give each engineer actionable items to work on.
This doesn’t work because “production” is not an actionable item.
Actionable is much more specific
o deploy this container
o setup these five servers
o fix these three bugs and push code
o setup these new environment variables
Why is “production” not specific?
Which application? Which layer or API must work? Are there intervening steps before production will work? Are their individual tasks for each engineer?
To me you need to “break things down” further if you have tasks that span multiple people, and multiple sessions. I think of a session as a 2-3 hour bucket of productive work. For me it is also the length of time my laptop battery takes to drain from 100% to 0%. When that happens I know I need a break.
And I know that’s how I get chunks of work done.
So to take this to a more specific level, if Friday is 5 work days away, I figure I have 12 increments of work I can do in that time, and if I have 3 engineers, then I need 36 chunks of work.
If you assign 36 chunks of work I believe you are much more likely to get 25-30 of those chunks done.
If you stick to the one macro goal of “get production to work”, engineers may secretly drop the ball figuring, well there are dependent tasks that are not done, so we’re not gonna get there. And also since the goal points at everyone, nobody saddles the failure.
Whereas if you have 36 chunks of work, individual engineers are less likely to drag their feet because it’ll be clear that the hold up was three of john’s tasks…
All of this gave me a new appreciation for Agile. It truly does keep teams on track, and focuses everyone on actionable work. This allows management more transparency on whats working, and engineers the feedback they need to get to the finish line.
Just to give some context, I’ll start by a quick walk through compute history. From the server cabinet in the back office, to the early managed hosting providers and then on to today’s modern cloud offerings, I’ll explain how we got here.
With that new context in mind, I’ll talk about that evolution one step further, to managed functions. What’s that you ask? Just hand over your code to the cloud, and let them handle running the servers, provisioning load balancers, and reacting to your customers when they hit the endpoint.
No presentation is complete without a proper diagram. My reference architecture makes use of Amazon’s many cloud services, including API endpoint, cognito for user authentication, lambda for serverless functions, dynamodb to store state information, S3 for storing objects, CloudFront for the edge caching network, and Route53 for the domain name.
Each of the components I mention above, requires some explanation. I’ll talk about how to setup a serverless project, how to define and manage your API endpoint. This is where users first touch your application. I’ll introduce user authentication with Amazon’s own service or a third party like OneLogin or Auth0. From there you’ll see how Amazon’s nosql database Dynamodb works, and how you can store your original & edited images in S3. And no site would be complete without an edge cache, and we’ll have that setup too. Then store your domain name in Route53 and point it to your API.
I have to admit, though I’m not a management consultant, I do pickup the big ones from time to time. Good to Great, How to Make Friends & Influence People, The Lean Startup, Innovators Dillemma & Who Moved My Cheese among my favorites.
I’d seen Ben Horowitz’s book “The Hard Thing about Hard Things” in the news. But I also really love the a16z podcast, and although I don’t know a ton about the VC business, I thought it would be a good read.
Boy is that an understatement! The book is so readable & so accessible, there are nuggets of value in there for anyone in the startup world, or building their career, CEO or not!
On the efficient market hypothesis
I had always assumed Adam Smith’s invisible hand was a good theory, almost as scripture. So to see a different perspective on this, and one backed up by real experience. That’s cool.
“No, markets weren’t “efficient” at finding the truth; they were just very efficient at converging on a conclusion — often the wrong conclusion”. p52
What’s more for investors out there, it means good old fashioned investigative work can still turn up gems, that are worth investing in. Word to the wise.
Freaky Friday was a movie way back in the 80’s. In it a mother & daughter are at each others throats, frustrated with the each other. They end up switching places, and quickly learn to sympathize with the other’s plight in life.
Horowitz decided to put this method to use between two of his managers. Pretty ingenious.
“After just one week walking in the other’s moccasins, both executives quickly diagnosed the core issues causing the conflict. They then swiftly acted to implement a simple set of processes that cleared up the combat and got the teams working harmoniously. p253
“In life, everybody faces choices between doing what’s popular, easy, and wrong versus doing what’s lonely, difficult, and right.” p212
Every time you make the hard, correct decision you become a bit more courageous and every time you make the easy, wrong decision you become a bit more cowardly. p213
Boy, can I relate to these bits of wisdom. Running my own business all these years, it hasn’t been easy. There have been ups and downs, and times when people told me to take a different road. But that courage. When you find it, it can be real fuel for you moving forward.
I realized that embracing the unusual parts of my background would be the key to making it through. It would be those things that would give me unique perspectives and approaches to the business. p276
When I work with entrepreneurs today, this is the main thing that I try to convey. Embrace your weirdness, your background, your instinct. p276
I hadn’t really thought of this, and it’s an interesting point. It may be one of the reasons why customers hire me, that I hadn’t realized. Certainly I can give an original viewpoint. But I think I will try to put this to work in the future.
This is a curious & fascinating point about the history of venture capital. Ripe for disruption indeed!
Marc discovered that the original venture capital firms in the late 1940s and early ’50s were modeled after the original investment banks such as J.P. Morgan and Rothschild. Those banks also did not do PR for a very specific reason: The banks funded wars—and sometimes both sides of the same war—so publicity was not a good idea. p271
Alexa & Kyra offer us a tantalizing question. Could it be that we could learn a lot from oddball innovators at the edge of the economy? When I say edge, I really mean it. She interviews Sam Hostetler who is building a business around milking camels, and then there’s Abdi Hasan a pirate from Galkayo northern Somalia. Yeah really! Or what about the German copycats Wimdu who built a complete replica of Airbnb by reverse engineering it.
1. Hack the cold call
Take the example of Lance Weiler. Early on the industry was very against digital. They didn’t see it as really making films.
“Part of [Lance] Weiler’s success was due to his ability to work the system. He wrote letters to major production companies telling them he wanted to make the first digital motion picture. After he didn’t hear back, he took a page from the con man’s handbook and wrote the same letters but intentionally misaddressed them so they were sent to the wrong companies. Sony for example would get a letter intended for Barco.”
He was later able to bring digital projection to Cannes & Sundance!
“For Weiler his big epiphany was when he realized he could be creative across all of it [the business]. Not just in the art product, but in financing, distribution, and business aspects of artistic production.”
The german brothers Oliver, Marc & Alexander Samwer make a superb example of how copying can bring building prowess to compete against innovators that were first to market.
“in 1998 Marc Samwer had an instinct that eBay would thrive in the German market… his brothers agreed… they contacted eBay via email numerous times, recommending that the company replicate it’s platformin Germany. Claiming that eBay failed to respond, the brother’s started their own German-language auction site, Alando, which was then purchased by eBay for 38 million euros (over $50 million) only 100 days after it’s debut. Had the Samwers not copied, eBay might have remained complacent, not realizing its potential within the german market.”
Although not mentioned in the book, Inditex the wildly successful firm behind fashion brand Zara did much the same thing to the fashion industry. By mastering the supply chain, they enabled their company to take designs from the runway & replicate them, turning designs into real clothing in stores, in just two weeks! And indeed they really do replicate, borrow & straight copy those designs from what they see at fashion week. Sad & brilliant at the same time.
“In the lexicon of the Misfit Economy, we define “hustle” as making something out of nothing. To move fast, to trade one thing for another, and to proactively create your own opportunities rather than waiting for opportunity to come your way. To hustle means getting your hands dirty, being lean and facile, working hard, being resourceful and resilient, and showing or having gumption, chutzpah, or mojo.”
And after all, isn’t that everything the startup industry aspires to? Agile teams? Growth hackers? Scrappy startups & innovation?
When I have a long todo list and a million things on my plate, my usual tactic is to just plow through it. Take short break to eat, but then get right back to work. My feeling is, if it’s weighing on the back of my mind, I won’t enjoy downtime anyway.
Recently though I had a very different experience. And it surprised me.
1. Too much to do
A colleague of mine asked me to meetup for beers. We planned to talk technology, and to catchup on what we were both working on.
As the night rolled on he had some delays, and I wanted to cancel too. After all I had a ton of work to do, and didn’t think I would enjoy myself. I really felt like I’d be worrying about all this work on my plate. It’s like taking a vacation when you have a deadline. It doesn’t feel quite right.
We ended up meeting anyway. At first I wasn’t totally relaxed, but then we started talking.
Our conversation turned to the evolution of datacenters. How they used to be on premise, then there were lots of hosting companies. And then Amazon changed everything!
We talked about evolution of tooling & automation. Although system administrators of old have been writing bash scripts forever to make their jobs easier, the proliferation of tools for deployment has allowed smaller ops teams to control fleets of servers. As my friend & colleague was newly starting a job on Amazon Web Services, a lot of this cloud stuff was new to him. So talking about it from a teaching vantage point, made me realize how strong I was in a lot of this stuff.
We talked about docker & containerization too. Even the origins back in the late 70’s with Unix chroot all the way up to Docker today. I explained to him that he could think of a container almost like a unix user, but with a more self-contained view of the whole system. In many ways a container acts like a vm, with it’s own filesystem and processes.
We talked a lot about aws, how S3 was an evolution of FTP in the old days, but much much better, how VPCs worked and the virtualization of networking, how VMs in the AWS world match with bare metal or not, how they share EBS storage. How Amazon has built a database service RDS around popular platforms like Oracle, MySQL & Postgres.
We shared a lot of ideas & brainstorming. About coding, C versus Java versus Python, package management, dependencies and on and on. He also mentioned he needed to build a test script to talk to an Amazon queue. I explained that it should be quite easy, and which libraries to look for.
I’ve had a mailbox with the postal service for over a decade. It’s been great. It’s a convenient address for a business, and it doesn’t change. Great for receiving bills, tax refunds, contracts, pay checks and everything else.
I picked the postal service years back because I figured they are solid. A corner deli or other mailbox service might go out of business. And the idea was to have a box that doesn’t have to change often.
In all those years I’ve always been timely with paying the rent, which can be every six months or twelve months as you choose. Recently I mixed up the renewal. When I went in to pickup my mail, the lock had been changed. I went to the customer service counter to ask about it. Amanda came to help me. She was cordial.
She explained however that the box had been closed. Since it was the day before the holiday, she wouldn’t be able to help with this, and suggested I come back on Monday.
As I like to be agreeable, I agreed to this request. In the meantime she was able to give me my mail. It was nice to see they had been keeping it. After all I was only a few weeks overdue.
Bring me the documents!
On Monday I had planned to return right away. But as my day got going I wondered if I would have enough time. This probably won’t be just a quick stop in. For some reason I sensed the process would be complicated.
Tuesday I set my alarm clock to wake up two hours early. I got dressed & headed straight to the post office. I wanted to get an early start. I went to the customer service desk again, and met Michael. I explained the mixup. Yes I was late playing my rent. Could I pay a penalty of some kind and then renew for another year?
“No I’m sorry, your box is closed now. You’ll have to reopen it.”, he explained.
“That’s no trouble let’s do that. What do we need to do?”, I asked.
“Well, you’ll need to bring all the documents that you opened the box with originally. Driver’s license, second form of ID, and the business document.”, Michael explained.
“Sure I said. I have my driver’s license right here. I can bring the business certificate. Great. And second form of ID?”, I asked.
“A passport would work.”
“That’ll be fine.”, I said. “So if I bring the business certificate & passport, and my driver’s license here, we’ll be able to open the box up again.”, I asked.
“Yes. If you’re able to bring those today or tomorrow, since nobody has opened another box, you can get that same box number again.”
From there I took another train trip home. The passport was easy enough to find. However the business certificate I had misplaced. It took me quite a while rifling through folders, but eventually I found it. Great I thought, I’ll just return to the post office, and have this task done for the day!
One thought occurred to me, I wonder if they might need a proof of address. He didn’t mention it, but I’ll just bring a recent gas bill just in case. Little did I know what would follow…
As I returned to the post office, Amanda greeted me. Hi there. Yes, I remember you from last week she said. Thx! Michael is at lunch now. He will be back in ten minutes.
So I waited. And waited. Twenty minutes go by. I go back to the counter, and Amanda says she’s not sure where he is, but she is going to lunch now too. When you see him return, just buzz the bell and he can help you.
Another twenty minutes go by and I see Michael return. I buzz the door, and he comes out.
Hi there Michael. I’ve managed to track down the documents you asked for, and I’ve got them all right here. Hopefully everything looks right and we can get this taken care of.
He asks me to fill out another form. On it includes my current address. Which has changed & is different from the original business address. Now I had thought to just write the same address down, as that might “simplify” things. But then I thought, that’s probably not prudent. Better to be honest. Right? I mean honesty is rewarded eh?
He sees the address & explains, I’m sorry but you’ll need a proof of address.
“Umm. Yes of course. I was careful to be very attentive to your requests earlier. Did I understand you correctly that you needed two items, the passport & business certificate.”, I asked?
“Yes but I didn’t see that you didn’t have your address here. You’ll need that too”
“No problem sir. Although it isn’t something we discussed, I did also bring my gas bill, because I thought that might be helpful here.”
“No we don’t accept that. Don’t you have a lease? Do you rent an apartment? You’ll need to bring me the right documentation!”
At this point it really does feel like a scene out of Terry Gilliam’s famous 1985 classic Brazil, where beaurocrat’s control your life. We must have form 27b-6 or else!
At this point I’m getting internally quite furious. It is now my third trip to the post office, and still I don’t have the right forms. Remember I’ve also been a customer for over ten years! But that seems to have no bearing. What’s more I have all the documents required, even a proof of address, but it is the wrong one!
My agitation is increasing, and I’m kind of shaking with frustration. I want to scream or yell at this point. But I realize that will only make things worse.
“With all do respect sir, I’ve returned here three times already. It’s quite a trip back and forth. I’ve brought the documents you requested. “, I explain
It seems his tension is rising too. I don’t know if this is the usual day but he is not bending.
“I’ve explained what you need to do. That’s it!”
“Can you please explain to me again sir.”, I ask
“One more question. That’s all I’ll take from you!”, he says holding up his finger menacingly to me.
“How do I know if I return with one more document, we won’t get further through this process and find something else missing. Then I may have to return again.”
I am despondent at this point. Close to giving up. I have less confidence now that I’ve had throughout. I think he sees my pain at this point. I’m practically crawling on the ground beaten. That must have been enough for pity.
“Well you could talk to Miss Adams. Maybe she can help you.”, he says
“Yes miss Adams you say? Sure let’s talk to miss Adams.”
I go around to the main post office and ask after Miss Adams. I meet Kelly at the front desk. She explains that miss Adams is at lunch, and will be back soon. But asks what is the problem.
I explain my mixup. About paying the rent late, and how the box must be reopen. I further explain that I’ve returned three times and show her the documents I have. She says they look fine, what’s the problem. Michael explained that a National Grid bill is not sufficient proof of address.
“Why not”, she asks?
It is at this point I have a revelation. There is no rhyme or reason to any of this madness. There may be a rule, it may or may not be followed. Depending on the day, the moon, the alignment of planets. Who knows.
“I’m sure that should be fine. Let’s ask miss Adams”, she says.
And then as if by magic, Miss Adams materializes out of nowhere. After such a trial, I imagine I felt much like fraternity pleges feel as they’ve been beaten and abused for days. Miss Adams is like a saint arriving from the heavens.
“Michael doesn’t take the gas bill as proof of address? Well I do.”, she says.
And with one simple wave of the wand, everything is resolved. Just like that.
When I think of this experience, yes I’m frustrated as anyone would be.
But it also really stands out for me in stark contrast. For I have worked with innovation, entrepreneurs & startups for many years. We all approach business from the perspective of solving problems. There understanding your customers, helping them, and simplifying processes is the rule of the day.
When I think of a government agency like the Post Office, I think of FedEx. Their market cap is 60 billion dollars. They exist with the sole purpose of moving packages. Customers will pay an incredible amount of money to avoid every having to deal with the post office.
This is a testament to innovation. And to startups. It’s why I’ve enjoyed working in the startup space all these years. The hard work & the creative problem solving. I live for that.
You’ve heard it before. Automate your builds. That means putting everything in version control, from environment building scripts, to configs, artifacts & reference data. Once you can do that, you’re on your way to automating production deploys completely.
While everyone is scrambling to figure out why part of the internet went down … wait is S3 is part of the internet, really? While I’m figuring out if it is a service of Amazon, or if Amazon is so big that Amazon *is* the internet now…
I loved Drew Bell’s story of stumbling into home ownership, attempting to fix a doorbell, and falling down a familiar rabbit hole. With parallels to legacy software systems… aka any older then oh say five years?
If you grew up on the virtual world of the cloud, you may have never touched hardware besides your own laptop. Developing in this world may completely remove us from understanding those pesky underlying physical layers. Yes indeed folks containers do run in “virtual” machines, but those themselves are running on metal, somewhere down the stack.