Categories
All Consulting CTO/CIO Devops Startups

I’m building a startup. What should I watch out for?

via GIPHY

I’ve worked with close to 200 startups over the years. So I’ve seen a lot of stuff. Some of the get-rich-quick managed to get broke even quicker. And others saw great engineers leave behind a clunky maze of an architecture with little or no documtation.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

Here are some thoughts on what to be mindful of…

1. Keep the stack simple

Are you asking yourself questions like these:

o if we build x,y,z it will save us time later
o we need kubernetes because everyone is using it
o let’s deploy on Amazon’s Cloud, everyone is there

If so you may have sidestepped keeping the stack simple. If you are building a proof of concept, mvp, chances are you do not need to build all the bells and whistles.

Relentlessly apply the discipline to keep things simple.

Read: Are pioneers and process people different breeds?

2. Use boring technology

You may be tempted to use some exotic new language. I know this new framework solves all the problems that have come before. No!

Let someone else live on the bleeding edge. I’m not suggesting to use COBOL, though there is still a surprising amount of that still around. What I’m saying is 5 year old tech is *not* legacy.

Not sure? Do a quick search on a job site for the language or technology. How many candidates do you get? If they are rarer you will pay more. And replacing your genius developer will be even harder.

But boring technology is not just about finding talent. It means many of the problems have already been solved. This is huge. There are proven ways to implement with the tried and true.

Related: Why is Martin Weiner special?

3. Beware of sharks in the water

Sharks can come in many forms. Investors that want to take 50%, a google or an Amazon that could rearchitect your solution in a week. These are real and present dangers.

Let’s not forget overspending on co-working spaces, and hidden costs around benefits, and surprising fees on your data or aws bill.

Related: How can 1% of something equal nothing?

4. Outsourcing – trust but verify

You may be able to hire one or two devs locally in new york, but if you need a larger team, you’ll likely look to outsourcing. There are tons of talented and brilliant engineers worldwide. So it makes sense to leverage this.

Beware though of handing all control to a team outside our shores. That’s because if you stumble into a legal trouble, you cannot use legal methods to resolve them. Some examples…

o what if you lost your aws keys and the outsource team holds you hostage?
o what if you are hacked, and the outsource team abandons the project?
o what if the outsource team tries to steal your IP?

I’m not suggesting here that outsource teams are any more likely to do these things than some firm locally. I think these events are rare in business. But if they happen local you can use the levers of the justice system to resolve. Remote may not lend to that.

Related: The art of resistance – when you have to be the bad guy

5. Relentlessly manage time

Your time is going to be short, and you’re trying to do the impossible. Use the tech wisely:

o use monitoring systems to alert you – no need to manually check things
o beware of slack and email – as they can destroy productivity
o automate where you can see real benefits
o keep the architecture simple!
o network for hiring people

Related: Is maintenance a forgotten art?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Categories
All CTO/CIO Startups

Are pioneers & process people different breeds?

via GIPHY

I was just reading Oliver Eidel’s blog. He had a great post, with some provocative ideas. Pioneers versus Process people.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

Now since I love to play devils advocate, after reading this thought provoking article, I thought I would write a bit of my take.

1. Are pioneers and process really characteristics of two different people?

As Oliver describes it, there are two completely different humans. Are they different breeds? Different species perhaps? Whatever the case, he argues that people either have one characteristic or the other.

To me the pioneer, is the engineering expression of creativity. They want to create new things, run with an idea, and see if they can make something happen. But even the most stodgy people have a bit of creativity in them, even if they don’t always express it. Yes I believe everyone has a tiny pioneer buried inside.

And so too, the pioneer, can should the need arise, buckle down and get tasks done. Yes they can be disciplined too, if you want to apply a different word.

Read: How can 1% of something equal nothing?

2. Do pioneer stage startups need more discipline?

I’ve worked at many a startup. I love them. Riding skateboards to work, and having dogs around the office is certainly a laid back downtown atmosphere, that appeals a lot more that the financial district buttoned up scene.

Also pioneer startups, often have a lot of leeway, to, as Oliver says “walking off in random directions, developing other great products”. This is how new ideas are hatched.

That said, many of these startups will one day fail. And it’s not for lack of idea, or market need. It’s often team related. That’s right, some of the best and coolest startups fail for lack of discipline. They’re too pioneering!

Related: Is Fred Wilson right about dealing in an honest, direct and transparent way?

3. Do process stage companies need more pioneering spirit?

I’ve also worked at quite a few larger, somewhat stodgy firms. These are so process heavy, that simple things can take forever. Everything requires a signoff. Deploying code can take weeks. Yes indeed they do things the old fashioned way, and that’s because they always have done that.

Could they use a pioneer or two to shake things up? Totally! Creative energy, drive and spirit could help them find faster ways to do things, and become unstuck when their old school process mentality is limiting their growth.

Related: What mistakes did you make when starting as a consultant?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Categories
All Consulting CTO/CIO Devops Startups

What do the best engineers do better?

via GIPHY

I’m fascinated by this topic.

I recently found another thread on HN about it What do top engineers you know do that others don’t.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

As always for hacker news, there’s a feisty debate about what such character includes.

Here’s my take.

1. Tackle learning quickly

Whether it means getting up to speed on a new service that AWS has launched, building a new api for an application that has never been built before, or getting up to speed with a new platform. Learning is ongoing.

Top engineers can make this a seemless part of their daily routine. Getting going quickly with new concepts and technologies, means wading into the water at first, to gain the general lay of the land. Now you can talk intelligently about the features, limitations and challenges.

From there he or she can dive in quickly to the specific area required for the project, and move forward with that technology comfortably.

Related: How do I migrate my skills to the cloud?

2. Customer & product perspective

When building code, it’s easy to get mired in libraries, sorting algorithms, and API minutiae. And all of that is very important. But what are you building, and why are you building it?

Understanding your customer, what they do day-to-day is not always easy. It means using the product yourself, and also talking with sales teams regularly to hear what they are hearing.

Then pouring all this into your user stories. For top engineers it will inform their decisions, and help them communicate to product & project managers about what issues their encountering. Tradeoffs about features, coding, performance and technical debt can be better evaluated with more information.

Read: Is Fred Wilson honest about transparency?

3. Dig Deeper

Does your code run slowly? Have you tried to figure out why?

Is it related to:

o latency in production that doesn’t appear our your laptop
o untuned production database queries
o untuned connection pooling
o slow API calls
o weird kubernetes or orchestration issues
o web host issue with memory shortage
o web host issue with slow unoptimized code
o issues on the client side

Top engineers have seen applications slow down or fail in a myriad of ways. This allows them to imagine how a new application might be failing, and investigate those.

Related: What mistakes did you make when starting as a consultant?

4. Great communicators

In startups, your engineers need to communicate to many folks who don’t have an engineering background. Product & UI/UX folks probably are quite technical on their own. But what about sales teams who are dealing directly with customers? Or C-suite folks who watch the business bottom lines, but may not have the same low level technical understanding?

Great communicators can find the right metaphor to explain hurdles and holdups, technical debt, or the latest performance challenges. And explaining those in terms that resonate for others is incredibly valuable to the team and business velocity.

Related: Can Mailchimp fraudulently charge your credit card?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Categories
All Consulting CTO/CIO Startups

What mistakes did you make when starting as a consultant ?

via GIPHY

I was recently reading a Hacker News thread on mistakes made in consulting. While most of the discussion I didn’t agree with folks *at all*, it did get me thinking about my own lessons.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

Since these threads in HN often end up in debate, I decided to blog my answers instead. Here they are!

1. Not getting a deposit

I wrote before about When clients don’t pay. It doesn’t happen often, but it does on occasion.

Most of the time, you have a solid relationship because you’re referred by someone who you have also worked with. And a bad actor will get a bad rep among others in the network. This also by the way keeps consultants honest too, as your reputation is at stake with every new customer.

But there have been a few. And there are other good reasons Why I ask clients for a deposit.

It can be a nominal amount of a few hundred dollars. But it gets you into the finance system, provides a small hurdle that the client must also jump over, and generally provides good will to both parties. It says “we’re serious”.

And that’s important!

Read: What I learned from 10 years of blogging

2. Giving away free advice

In the first few years of consulting, I worked for so many great companies, that I figured they were all great. Then along comes one shady shop, for whom I was called in to help with scaling an application.

For a couple of hours we met face to face to discuss their challenges. What I found was an application that had grown beyond it’s original ambitions. I suggested they evaluate the product, provide new service levels to their customers, at different price points. And then for the higher paying customers, build out new hardware just for them.

It was a great solution, if I do say so myself. The customer thought so too. They went and implemented it themselves, without hiring me! I think they even offered to pay me a couple of hours for the meeting.

Suffice it to say I was pissed. There wasn’t much I could do because we didn’t have any agreement in place at that stage. I had no idea people would do stuff like this.

But I learned the hard way. Sad to say it’s the few bad actors that make us all play more carefully in business. Buyer beware!

Related: 6 Devops interview questions

3. Billing hourly

Billing hourly is how many start out. Some think of it as an industry standard. Lawyers do it, hey why not?

But hourly billing can be very confusing to customers. If you work 10 hours to solve a problem one week and 60 the next, they will find these invoices confusing and frustrating. What’s more on the 10 hour end of the spectrum you’ll be answering questions why the work was so “easy” and on the 60 hour end, what did you do wrong?

Customers often just want a problem solved. They want you and they want your availability. And the value of that may vary quite a lot. If you can figure out a way to bill weekly or monthly such that the client is happy with the value you’re providing, this will simplify your life immeasurably. Customers will be happier, and so will you.

I also recommend keeping daily notes and providing progress reports to your client.

Read: High availability what is it and why is it important?

4. Don’t step on toes

As a consultant, you will often be working with a team of fulltime folks. Not always, but sometimes there can be a tiny bit of resentment. Maybe because you’re outside opinion is threatening, or maybe for a million other reasons.

So I recommend treading carefully. Try to reassure your teamates that you aren’t trying to outshine them. Inevitably you’ll be in meetings with folks smarter than you. Certainly they will know more as they have boots on the ground, but sometimes, they are just plain & simple smarter than you. 🙂

Feel things out, and don’t step on toes. Some need to be right. Let them be that.

Read: High availability what is it and why is it important?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Categories
All Devops Scalability Startups

How crazy can Kubernetes get?

I was flipping through Reddit and found this hilarious post referencing a Scott Adams Dilbert strip on Kubernetes.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

What I found even funnier were the comments on the Reddit thread. Read on for fun!

1. Watch your memory

One engineer said he had a dockerized app running with 3GB of memory serving just 7 customers.

Not to be outdone another ops guy pipes in that he has one using 180Gb of memory serving just a few hundred customers.

Of course this is the internet, and along comes a guy who has an app using 1TB of memory, with only one user!

Optimization be damned!

Read: Infrastructure provisioning – what is it and why is it important?

2. Beware growing application complexity

As you dockerize your application, you can support multiple versions of software and packages. This can keep you flexible but also enable engineers to kick the can down the road. Technical debt is real!

What’s more each microservice *can* be on a different stack, using a different language and framework. But just because you *can* do something doesn’t mean you should.

Though docker & kubernetes will enable the above, keep in mind your team has to support it. Using some cool new language that hasn’t really achieved critical mass? Remember the engineer who championed that, and built your business crown jewels on top of that, will eventually leave. And when he or she does, you will be faced with the challenge of finding someone who knows the stuff!

Related: 6 Devops interview questions

3. What is a microserved monolith?

Well it’s not really a thing, except it sounds fun. And a bit absurd. If all those docker containers never get optimized, they probably have layer upon layer of useless stuff. Start with a smaller base image, dont include debug stuff, and extra layers. And cleanup after installing packages.

Here’s a more detailed howto optimizing Docker image sizes.

Read: Is zero downtime even possible on RDS?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Categories
All CTO/CIO Devops Software Development Startups

How to think like a senior engineer

via GIPHY

I just read this story
the Art of interrupting engineers. While much of what I read was pretty obvious from the engineer’s perspective, the product & project manager perspective was illustrative.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

There are definitely things that senior or more experienced engineers do differently. And those things can be learned.

Here are my thoughts…

1. Document all the things

By documenting all the little pieces you are working on you gain in a few ways. You communicate to management complexity they may not see. You buy yourself more time.

Finally you may even help yourself see implicit tasks more clearly. Whenever I hear myself or someone else utter the phrase “that should be easy”, I know I’m onto one of those mysterious tasks that seems simple but never is.

Be relentless. Break big tasks into smaller ones, and ticket each and every darn one!

Also: How can 1% of something equal nothing?

2. Communicate more and better

If you’re doing agile, chances are you’re probably joining a standup everyday.

Those are opportunities to share what is blocking.

o What tradeoffs are you struggling with?
o What technical debt is slowing you down?
o What new discoveries may require a rework of the timeline?

There is surely an art buried in communication. You want to be descriptive. You don’t want to come off complaining. You want to educate stakeholders. Beware of coming off as dismissive.

Related: Is maintenance sometimes a forgotten art?

3. Anticipate. Under promise & over deliver.

If you’ve gotten in the weeds with a particular API before, you’re likely to have a gut feeling about how new features and changes may go well or poorly. Or you may have dug through the comments. Maybe you didn’t find any!?

Or maybe the particular codebase sits on top of an interface or library you haven’t worked with before. So there will be a bit more of a learning curve. Whatever the case, try to promise less than you think you can really deliver on.

You can always finish extra tickets and over deliver later!

Read: What tools and technologies are devops engineers using today

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Categories
All CTO/CIO Startups

How can 1% of something equal nothing?

via GIPHY

I just read this painful story
My company sold for $100 million and I got zilch. How can that be?.

I felt apalled. I was a little sick to my stomach actually.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

Is it possible to work for years, putting blood sweat & tears into a firm, and end up with nothing?

I’ve been working with startups since 1996. While I wish this was an aberration, I’ve seen it happen a half dozen times over the years.

Here are my thoughts…

1. Have a lawyer review legalese

If your intuition tells you this additional compensation is worth something, then use logic to prove it.

The first step is to have a lawyer review the fine print. They know things you don’t. Since this isn’t your domain of expertise, you’ll want someone who can see around the corners, to give you fair advice.

Related: Is maintenance sometimes a forgotten art?

2. Have a financial expert review it

Ask a financial expert to review your compensation package. Think of it like buying a house, you’re commiting to something big, with a slow long term return.

What are the chances statistically that the company will succeed? What are the chances it may fail? If it does get bought, what are the chances the fine print will eat up your portion? When in doubt be conservative with your guesses.

Also: Is Kubernetes going bye bye?

3. Use your math skills

Compare these numbers to other financial investments, such as real estate, or the S&P 500. Use your math skills to choose the best place for your money.

Be relentlessly logical. It’s very easy to get emotional or be irrational when you are dealing with money. But remember it is just math in the end. Evaluate. Test. And execute rationally.

Read: What tools and technologies are devops engineers using today

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Categories
All Scalability Startups

What Matt Ranney learned scaling Uber to 1000 services

via GIPHY

I was recently watching Matt Ranney’s 2016 talk from GOTO conference. He’s an excellent speaker, not least because of his formidable experience at the helm of Uber.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

Jump to 11:45 to start out…

1. What’s the cost of many languages

As they embraced microservices, they discovered surprising problems around having so many languages.

For example it became hard to share code and furthermore it became hard to move between teems.

The upshot is that the fragmentation of languages siloed teams, and built up tribes around those languages, and expertise or interest in them.

Related: How can we keep cloud architectures simple

2. What’s the cost of using RPC for everything

In the microservices world, everything becomes a remote procedure call. For starters there are problems with HTTP and related semantics.

Also there are challenges with JSON. Since there aren’t any types, you don’t have a schema to lock you into formats. So you get weird behaviors down the line.

The upshot: Servers are not browsers. Indeed. Function calls are wayyyy better *within* a datacenter. That’s because you control both servers, so you can tune accordingingly.

Also: What hidden things does a deposit reveal?

3. Tuning across languages can be very challenging

Different languages have different tools available. Can your favorite tool work across all the languages you’ve decided to support in your microservices?

Even though you’re going to go with microservices, think about standardizing across those services. Use tools that work on all of them. And when building your services, make sure it outputs the same dashboard metrics.

Read: Can communication mixups sour an engagement?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Categories
All Cloud Computing CTO/CIO High Availability Startups

Are you as good as the public cloud?

via GIPHY

According to Lyft’s recent public filing, they plan to spend 300 million buckaroos in the next 2.5 years on AWS.

Did I hear that right?

Join 38,000 others and follow Sean Hull on twitter @hullsean.

Perhaps that is their estimate, or the maximum amount they want to budget for. Regardless that’s a lot of money any way you slice it. A lot of folks are commenting about how crazy that is, and how much datacenter you could build yourself with that much money.

What do you think? Is it foolhardy? Or is there a hidden wisdom here?

Here’s my take.

1. Do you have one million customers testing your datacenter?

If you’re comparing the cost of the cloud to the raw numbers of running your own datacenter, the hardware costs are not enough. You’ll need to include the ops teams & other engineers. Right, you probably guessed that.

But did you factor in the costs of a legion of testers. This is the hidden cost that commercial software carries, even while open source software gets this benefit for free.

With a public cloud like AWS you have millions of customers testing the product everyday, and running into edge cases long before you do. So you get a better service, that’s more reliable, all invisibly for free.

Related: How can we keep cloud architectures simple

2. Do you have 66 datacenters spread across 21 regions and a free network between them?

Anybody who was building web applications in the year 2000 will remember how websites didn’t load the same for different customers. Depending on where in the world they were located, they could experience a very different user experience.

These days we assume that we can be global from day one. But how exactly do we achieve this? Remember with a public cloud, you’re getting tons of things for free, without knowing it. Moving data between AZs or regions? That’s all going across a private interconnect.

And that’s not even including the 180 nodes inside cloudfront that give you a global CDN footprint too!

Also: What hidden things does a deposit reveal?

3. Do you have an engineering team automating away job roles?

I remember the days of DBA job role, do you? Probably not. I specialized in this for years, and there were tons of companies hiring me to help them with it. First Oracle, then MySQL, then Postgres.

Then along came Amazon RDS. Guess what, companies don’t really hire for that role anymore. They do need help with it from time to time, but not as a primary specialization.

What do I mean? Well by hosting your application on AWS, you’re benefiting from the work of teams of engineers in different departments, all expanding on APIs and automating things that those one million customers are asking for.

You’re not going to be able to innovate that well and that quickly in your own datacenter. So you’ll pay more!

Read: Can communication mixups sour an engagement?

4. Do you have APIs that tons of engineers have already written code for?

A quick peek at Terraform’s community modules on Github and you’ll probably blush. From VPCs to bastion boxes, key management to load balancers, lots of code has been written and open sourced.

By deploying on a platform that a lot of other devs are using, you’ll benefit from all this open source code. That means you won’t have to write that stuff yourself.

Sure you’ll have integration work to do, but the hidden benefit of being on a popular platform saves you money.

Check out: How I use 5 daily habits to stay on track

5. Can you do disaster recovery for free?

If you build your own datacenter, you have to buy all your capacity. So there are no spare servers sitting around waiting for your use. In the public cloud there is always spare capacity.

What that means is you can write automation code to spinup copies of your application stack in alternate regions, at the push of a button. Thus you effectively get disaster recovery for free!

Also: Can daily notes help you work better with clients?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Categories
All CTO/CIO Software Development Startups

I have a new appreciation for Agile and not because it worked

via GIPHY

A couple of years ago I worked at a startup in the online publishing space. But this story isn’t about online publishing. This story is about deadlines, and not meeting them.

For those who don’t know me, I’m a professional services consultant. I’ve worked on a project basis for 90% of two plus decades of my professional career. I mentioned this to give my opinions and perspectives some context. Although I’m not a manager, I’ve worked under more managers than most. Because I work on 5-10 projects per year, that comes to close to two hundred in my career.

Join 38,000 others and follow Sean Hull on twitter @hullsean.

My career path has given me a unique perspective, on teams, leadership and motivation. Of course like anyone who’s worked in tech, I’ve see a lot of agile.

Daily standups are de rigueur of course. As are breaking up your work into two week sprints, and assigning story points to those tasks.

I have to admit there are times when Agile seemed the most fashionable way I saw teams working. And I guess I believe it was more trendy than functional.

Doesn’t everybody already work off tight todo lists, break tasks down into manageable pieces, and always deliver what they promised? It may be because I’ve spent a lot of time managing myself, and surviving as a freelancer that I’ve picked up some of these habits. But I digress…

1. Crunch time

While we as a team had been working on two week sprints, something happened to derail us. Suddenly a major customer was in jeopardy, because we had not delivered on sales promises.

Although what was being promised, we *could* build, we were stumbling over the details.

As an emergency plan, we dropped our current sprint tasks, and marshaled our forces towards this new goal. Everyone on the team knew the customer was hanging by a thread.

Related: When you have to take the fall

2. We need to deliver production by Friday

While this edict looks great on paper, it didn’t work out so well. Developers and ops weren’t clear which systems were included in “production”, and how they needed to be connected together.

Each engineer ended up interpreting the goal in their own way, often assuming that others were shouldering ultimate responsibility of delivery.

“Well I did my part, this other part of the team is responsible for those other pieces…” was the refrain I heard. Sadly the deadline was missed, and everything was a mess. Only after management later dug in and sifted through the rubble, did they actually break up the tasks, assign story points, and give each engineer actionable items to work on.

Related: When clients don’t pay

3. Please work together to make that happen

This doesn’t work because “production” is not an actionable item.

Actionable is much more specific
o deploy this container
o setup these five servers
o fix these three bugs and push code
o setup these new environment variables

Why is “production” not specific?

Which application? Which layer or API must work? Are there intervening steps before production will work? Are their individual tasks for each engineer?

To me you need to “break things down” further if you have tasks that span multiple people, and multiple sessions. I think of a session as a 2-3 hour bucket of productive work. For me it is also the length of time my laptop battery takes to drain from 100% to 0%. When that happens I know I need a break.

And I know that’s how I get chunks of work done.

So to take this to a more specific level, if Friday is 5 work days away, I figure I have 12 increments of work I can do in that time, and if I have 3 engineers, then I need 36 chunks of work.

If you assign 36 chunks of work I believe you are much more likely to get 25-30 of those chunks done.

If you stick to the one macro goal of “get production to work”, engineers may secretly drop the ball figuring, well there are dependent tasks that are not done, so we’re not gonna get there. And also since the goal points at everyone, nobody saddles the failure.

Whereas if you have 36 chunks of work, individual engineers are less likely to drag their feet because it’ll be clear that the hold up was three of john’s tasks…

Related: Why i ask for a deposit

All of this gave me a new appreciation for Agile. It truly does keep teams on track, and focuses everyone on actionable work. This allows management more transparency on whats working, and engineers the feedback they need to get to the finish line.

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters