Category Archives: Software Development

Marching towards continuous deployment

cute code pipeline

If you’re like a lot of small dev teams & startups, you’ve dreamed of jumping on the continuous deployment train, but still aren’t quite there.

Join 32,000 others and follow Sean Hull on twitter @hullsean.

You’ve got your code in some sort of repository. Now what? As it turns out the concepts aren’t terribly complicated. The hardest part is figuring out the process that works for your team.

1. Make a single script for deployment

Can you build easily? You want to take steps to simplify the build process & work towards everything being done from a single script. This might be an ant or maven script. It might be rake if you’re using ruby. Or it might be a makefile. Either way it organizes dependencies, checks your system & compiles things if necessary.

Also: Do startups need techops?

2. Do nightly builds

If you’re currently doing manual builds, work towards doing them nightly. Once you have that under your belt you can actually schedule these to happen automatically every night. But you’re not there yet. You want to work to improve the build process first. Work on the performance of this process. Quality is also important. Is the build quality poor?

Related: Is there a devops talent gap?

3. Is your build process slow?

If it takes a long time to do the build, it takes a long time to get to the point where you can smoke test. You want to shorten this time, so you can iterate faster. Look at ways to improve the overall performance of the whole chain. What’s it getting stuck on?

Read: 3 things devops can learn from aviation

4. Is your build quality poor?

Your tests are going to verify application functionality, do security checks, performance or even compliance checks. If these are often failing, you need dig in to find the source of your trouble. Tests aren’t specific enough? Or are you passing your tests, but finding a lot of bugs in QA?

It may take your team some time to get better at building the right tests, and reducing the bugs that show up in QA. But this will be crucial to increasing confidence level, where you’ll be ready to automate the whole pipeline. As you become more confident in your tests, then you’ll be confident to automatically deploy to production.

Also: How to deploy on Amazon EC2 with Vagrant

5. Evaluate tools to help

Continuous deployment is a lot about process. Once you’ve gotten a handle on that, you’ll have a much better idea of what you want out of the tools. Amazon’s own CodePipeline is one possible build server you can use. Because it’s a service, it’s one less server you don’t have to manage. And of course Jenkins is a popular option. Even with Jenkins there is a service based offering from CloudBees. You might also take a look at CircleCI, & Travis which are newer service based offerings, which although they don’t have all the plugins & integrations of Jenkins, they’ve learned from bumps in the road, and improved the formula.

We like CircleCI because it’s open source, smaller footprint than Jenkins, integrates with Slack & Hipchat, and has Docker support as well.

Also: 5 Tips for better database change management

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Are you getting errors building Amazon lambda functions? Don’t fret I got you!

aws lambda python

Why does Amazon make lambda functions so hard to create? Well my guess is that when you live at the bleeding edge you should expect to get scrapes!!

Everybody is trying to build lambda functions these days. And it’s no wonder. Once you get them running, Amazon takes care of all the infrastructure drudge work! So cool.

Join 32,000 others and follow Sean Hull on twitter @hullsean.

I’ve been spending some time trying to get answers out of AWS support, and let me tell you it’s no fun. Yes all this stuff is new technology, and nobody has expertise in it in the way say you might in Linux or Oracle or another technology that’s been around for a decade.

Still you’d hope the techs would have some clue. In the end it was a slog dealing with support, and I think I was the one teaching them!

I did find Matt Perry’s howto which is pretty good.

Hopefully my own notes can help someone, so read on!

1. No lambda_function?

The very first issue you’re gonna run into is if you name the file incorrectly, you get this error:


Unable to import module 'lambda_function': No module named lambda_function

If you name the function incorrectly you get this error:


Handler 'handler' missing on module 'lambda_function_file': 'module' object has no attribute 'handler'

On the dashboard, make sure the handler field is entered as function_filename.actual_function_name and make sure they match up in your deployment package.

If only the messages were a bit more instructive that would have been a simpler step, but oh well!

Also: Is Amazon too big to fail?

2. No module named MySQLdb

This is a very tricky one. I mean after all you just spent all this time building your deployment package specifically for lambda, so what gives??


"Unable to import module 'lambda_function': No module named MySQLdb"

Turns out when you use a virtualenv, files will be installed into proj/lib/python2.7/site-packages/ or lib64. However Lambda wants them in the root proj/ directory! So move them there. I know I know. Weird, but that’s what they want.

Related: When hosting on Amazon turns bloodsport

3. Can’t find libmysqlclient

If you’re using the MySQLdb library like I was, you’ll eventually bump into this error:


Unable to import module 'lambda_function': libmysqlclient.so.18: cannot open shared object file: No such file or directory

Turns out that /usr/lib/libmysqlclient.so.18 needs to be COPIED from /usr/lib. Don’t do “mv” or your system won’t have the mysql lib anymore!

Related: Are SQL databases dead?

4. Use the Amazon Lambda environment

One thing the support pointed out is that AWS as *supported images* for lambda development.

After all the errors above were resolved, it’s not clear to me that the supported AMI’s are truly required. However if you’re hitting intractable problems building a properly lambda deploy, you might wanna look at building one of these boxes.

Read: Why dropbox didn’t have to fail

5. Build your lambda deploy package

Now let’s roll it all together. Here’s are all the steps to build your deploy package.


- SSH to the instance
- mkdir test
- virtualenv test
- source proj/bin/activate
- sudo yum groupinstall 'Development Tools'
- sudo yum install mysql
- sudo yum install mysql-devel
- pip install MySQL-python
- cd test
- emacs -nw lamdba_function.py
- add your code to that file
- save the lambda_function.py
- mv proj/lib/python2.7/site-packages/* proj/
- mv proj/lib64/python2.7/site-packages/* proj/
- rm -rf proj/lib (don't need dist-packages in the deploy pkg)
- rm -rf proj/lib64 (don't need dist-packages
- zip -r proj.zip *

Also: How to hire a developer that doesn’t suck?

6. Upload your code

Uploading your code via the AWS dashboard is fine when you’re first testing things. But after a while it’ll get tiring going in the front door.

Create a new lambda function by specifying the basics as follows:


aws lambda create-function \
--function-name testfunc1 \
--runtime python2.7 \
--role arn:aws:iam::996225510001:role/lambda_basic_execution \
--handler lambda_function_file.handler_name \
--zip-file file://proj.zip

And when you want to update your function, do the following:


aws lambda update-function-code \
--function-name testfunc1 \
--zip-file file://proj.zip

Also: How to deploy on EC2 with Vagrant

Good luck with lambda. Once you get past Amazon’s weak documentation it’s pretty cool to be in a serverless computing environment. Happy deploying!

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Some thoughts on 12 factor apps

12 factor app

I was talking with a colleague recently about an upcoming project.

In the summary of technologies, he listed 12 factor, microservices, containers, orchestration, CI and nodejs. All familiar to everyone out there, right?

Join 28,000 others and follow Sean Hull on twitter @hullsean.

Actually it was the first I had heard of 12 factor, so I did a bit of reading.

1. How to treat your data resources

12 factor recommends that backing services be treated like attached resources. Databases are loosely coupled to the applications, making it easier to replace a badly behaving database, or connect multiple ones.

Also: Is the difference between dev & ops a four-letter word?

2. Stay loosely coupled

In 12 Fractured Apps Kelsey Hightower adds that this loose coupling can be taken a step further. Applications shouldn’t even assume the database is available. Why not fall back to some useful state, even when the database isn’t online. Great idea!

Related: Is Amazon too big to fail?

3. Degrade gracefully

A read-only or browse-only mode is another example of this. Allow your application to have multiple decoupled database resources, some that are read-only. The application behaves intelligently based on what’s available. I’ve advocated those before in Why Dropbox didn’t have to fail.

Read: When hosting data on Amazon turns bloodsport

Conclusion

The twelve-factor app appears to be an excellent guideline on building cleaning applications that are easier to deploy and manage. I’ll be delving into it more in future posts, so check back!

Read: Are we fast approaching cloud-mageddon?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Are open source projects run like a democracy or an oligarchy?

aristocracy

I was reading Fred Wilson’s comments recently on The Bitcoin XT Fork. In it he discussed how open source developers manage their projects.

“A group of open source core developers are a democratic system.”

I was surprised by this comment because I had never thought if it as democratic. Here are my thoughts…

Join 28,000 others and follow Sean Hull on twitter @hullsean.

1. Indefinite tenure

Open source projects typically have a leader with indefinite tenure. He can’t be voted out. When developers are unhappy with how things are run, or how they’re evolving, they typically “fork” the project and go their own way.

That would be where Texas secedes from the union if they’re not happy with how things are run in Washington.

Also: Is the difference between dev & ops a four-letter word?

2. Inherited rights

Like an aristocracy, leaders of an open source project typically have rights inherited. This could be due to merit, or seniority. They are the ones with admin rights on the git account.

Divine rights indeed!

Related: Is automation killing old-school operations?

3. Appointments made by merit

Developers join open source projects, and move through the ranks mostly by merit. Sure there’s some back scratching, and massaging that helps too. Personality surely matters, but primarily skill at contributing code & architecture ideas are paramount.

Read: Do managers underestimate operational cost?

4. Power rests with a small elite

For sure, all people cannot vote on open source project direction. It’s a small group of elite, who are admittedly closest to it, and most knowledgable. These are the ones who control it’s direction.

Also: Is the difference between dev & ops a four-letter word?

5. Oligarchy or Aristocracy?

From wikipedia an Oligarchy is “a form of power structure in which power effectively rests with a small number of people”. That sounds closest.

While open source projects do have the indefinite political tenure of an authoritarian regime, they lack the strict obedience aspect.

However, open source projects do look a bit like an Aristocracy. Aristocracy is “a form of government that places power in the hands of a small, privilged ruling class. The term derives from the Greek aristokratia meaning ‘rule of the best’. ”

Also: Are SQL Databases dead?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

What happens when you combine devops & continuous delivery into a card game?

release devops game

Join 25,000 others and follow Sean Hull on twitter @hullsean.

Alex Papadimoulis & the guys at Inedo put together CodeMash The Game an interesting game for a new twist to conference going.

Now they’re at it again with a kickstarter to build Release! a game about devops & continuous delivery.

1. Bring your team together

Weekly standups are great, but what about throwing a quick card game in to mix things up? It’s an interesting twist and one that’s sure to help with team building.

Read: Why has no-one heard of Moskovitz but everyone knows Zuckerberg?

2. Learn more about cutting edge software development

Weak on your agile or want to raise your teams software quality. Release seems like a new and surprising way to do just that.

Related: Why I ask clients for a deposit

3. Learn about software development luminaries

Many of the important folks in the evolution of software development are featured in the game, such as Patrick Dubois, Jez Humble & Dan North.

Also: Is Amazon RDS hard to manage?

Why startups need techops

devops divide

I was at a talk recently on node.js. Even if I’m not working with a technology directly, it’s exciting to see what’s out there, and node.js is bringing some hyper fast performance to a certain category of web applications.

During the keynote, the speaker mentioned a service to deploy applications on. I can’t name names unfortunately but it was a cloud solution on top of which you could deploy your application. Go this route
and you can do without an operations team. Avoid overhead of hiring ops, he claimed. And hey, then you can hire more developers!

To be fair I’ve heard much of the same thing at DBA or linux conferences. I can’t count the number of stories that start with “what some idiot developer did that took down our production systems…”.

Yes, it seems dev & ops are still just a tad bit adverserial.

Join 13,000 others and follow Sean Hull on twitter @hullsean.

1. My little known origins as a developer

Many colleagues and clients I’ve met in the New York City startup industry know me primarily as an operations & scalability guy. I tune databases, infrastructure and components to make things lightening fast.

I spent my earliest years at university on the computer lab operations staff. We watched and managed, made sure level zero backups were taken care of, and moved the tapes. Directly after college, I started at a software firm. I did C++ GUI development on the Mac, using the toolbox libraries with Metrowerks Codewarrior. I built split windows, and scroll bars, and displayed rows of data with nice resizable columns. All this wasn’t built into the class library, so for a lot of it we needed to roll our own solution.

We always had a long list of features coming from the business units. I also fielded many support calls, often from the windows platform as the code there hadn’t been managed and built as carefully. But that too was instructive as you could feel the pain of customers day-to-day challenges. It also illustrated the tradeoffs between new code and features, and existing bug fixes and support.

Also: Why generalists are better at scaling the web

2. A trip through the dot-com bubble as Oracle DBA

Through a circuitous path, I moved to New York in the mid-nineties and joined a startup. I had the opportunity to wear a lot of hats there, and apply my computer lab and Linux operating systems experience to the challenge of managed Oracle. I got a lot more involved with operations quick.

As the dot-com bubble grew, I saw a hot and growing demand for Oracle DBAs as most startups used Oracle, but the talent was in short supply. In one startup 80 million dollars was on the line as performance hobbled the website, and investors feared the worst.

Read: Why the Twitter IPO made a shocking admission about scalability

3. Different priorities & mandates

I remember working at Starmedia a media darling at the time. I was analyzing the database & server systems, and finding some code & jobs running during peak daytime hours. Management claimed that could not be the case. Yet for the next days and weeks I saw the same jobs running. I held strong and spoke truth to power as they say. That’s not always easy when you have a lot of investors, screaming CTOs and 100+ hour weeks. But eventually the source of the job was located, and disabled. And the website returned to it’s speedy self.

These experiences though do underline in my mind the different priorities and focus that developers and operations staff have.

Techops, system administrators & DBAs are typically averse to change. They fight it tooth and nail. That isn’t because they like to be curmudgeons though. They are typically very concerned about the business, but from a dramatically different perspective of stability, and reliability, even at 2am in the morning. They are concerned about the longevity of data, consistency, and durability of it.

Developers on the other hand have a different mandate. They are responsible for new business features, solutions to business requirements. Rapid prototyping & reactive or agile is embraced because it means you can deliver quicker to the business.

Crucially, both of these folks care very much for the business. Just with very different priorities.

Check this: Why AirBNB didn’t have to fail

4. Can developers do operations for you?

In a lot of small startups, the initial phase is obviously on building a product. That’s the build phase, and not surprisingly you hire a lot of developers. As you should. But as you grow you may find the operational tasks that are defaulting to one or more developers are taking more and more of their time. As your customer base grows and you’ve seen your first few spikes, it’s time to start thinking about hiring for a real ops role.

In summary, yes they can, but perhaps not well.

Related: How to hire a developer that you can work with

5. Volume discount, made to order or instant coffee

You may choose to go with instant coffee, by bringing someone in-house. You may find the right talent is hard to find. I wrote about this: Why techops and DBAs are in short supply.

Alternatively you may prefer a volume discount from one of the larger remote DBA or managed support solutions such as Oracle’s, Pythian or Percona. These guys all provide great service, but keep in mind how big of a fish you are. You’ll likely work through a ticketing system, and in some cases different engineers will look at your systems at different times. You will likely need either a very hands-on technical CTO or other in-house person to take ownership, and manage things closely.

The third option is a made-to-order coffee. Yes you pay more for Toby’s, Blue Bottle, or Ninth Street Espresso but you get what you pay for as they say. A boutique shop or independent consultant will provide a lot more hand holding, help your internal staff get up to speed, and communicate intimately about the process. If you’re a more non-technical CTO, or you’re very busy running the business, this solution may make a lot of sense for you.

Also: Why cloud detractors need a history lesson

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Understand the High Cost of Technical Debt by Ward Cunningham (Video)

A week or two ago, I got into a conversation on Twitter about technical debt, and someone shared this superb video by Ward Cunningham (youtube). Here is Ward’s Interview website.

Join 10,000 others and follow Sean Hull on twitter @hullsean.

Waterfall, agile, developer or operations, devops, managers, CTOs… everyone should watch this video, for it cuts to the heart of the challenges we face doing modern software development, in a fast paced and always changing environment.

Though it’s not mentioned in the video, I would argue using an ORM is an expensive form of debt, one that is improperly calculated by teams, managers & startups, and one that bites hard into future scalability. Stay tuned for a post about that!

Enough said, here’s the video.

Here’s the transcript of much of the dialog. Appologies for any typos…

1. Metaphors & Thinking

I became interested in the way metaphors influence the way we think

after reading George Lakoff & Mark Johnson’s Metaphors We Live By (Amazon).

An important idea is that we reason by analogy through the metaphors that have entered our language.

Related: 5 Things Toxic to Scalability.

2. Coining Debt Metaphor

I coined the debt metaphor to explain the refactoring we were doing on a product.

This was an early product done in smalltalk. It was important to me that we accumulate the learnings we did about the application by modifying the program to look as if we had known what we were doing all along. And to look as if it had been easy to do in smalltalk.

The explanation I gave to my boss, and this was financial software, was a financial analogy I called the debt metaphor and that said that:

[quote]
If we failed to make our program align with what we then understood to be the proper way to think about our fin objects, then we were going to continue to stumble on that disagreement which is like paying interest on a loan. -Ward Cunningham
[/quote]

Also: AirBNB didn’t have to fail when Amazon went down.

3. Need for Speed

With borrowed money you can do something sooner than you might otherwise, but until you pay back that money you will pay interest.

I thought borrowing money was a good idea. I thought that rushing software out the door to get some experience with it was a good idea. But that of course you would eventually go back and as you learned things about that software you would repay that loan by refactoring the program to reflect your experience as you acquired it.

Read this: Why generalists are better at scaling the web.

4. Understand the Burden

I think that there were plenty of cases where people would rush things soft out the door & learn things, but never put that learning back into the program. That by analogy was borrowing money thinking you never had to pay it back. of course If you do that you know say with your credit card, evemtually all your income goes to interest & your purchasing power goes to zero.

By the same token if you dev a program for a long period of time, by only adding features, and never reorganizing it to reflect your understanding of those features, then eventually that program does not contain any understanding and all efforts to work on it take longer and longer. In other words the interest is total. You’ll make zero progress.

Check out: How to hire a developer that doesn’t suck.

5. Achieving Agility

a lot of bloggers at least have explained the debt metaphor and confused it i think with the idea that you could write code poorly with the intention of doing a good job later. and thinking that that was the pirmary source of deb.t I’m never in favor of writing code poorly. But I am in favor of writing a code to reflect your current understanding of a problem even if your understanding is partial.

If you want to be able to go into debt that way by dev soft you odn’t completely understand, you’re wise to make that software reflect your understanding as best you can. So that when it does come tiem to refactor it’s clear what your thinking was when you wrote it. and making it easier to refactor it to what your thinking is now.

in other words the whole debt metaphor or lets say the ability to pay back debt, and make the debt metaphor work for your advantage depends upon you writing code that is clean enough to be able to refactor as you come to understand your problem.

i think that’s a good methodology it’s at the heart of extreme programming. the debt metaphor is an explanation, one of many explanations as to why extreme programming works.

Read our popular interview guides for candidates, managers & recruiters: MySQL DBAAWS Expert (Amazon Cloud) and Cloud Deployment & Automation Engineer.

Don’t forget, hiring is a numbers game

Get some in your inbox: Exclusive monthly Scalable Startups. We share tips and special content. Here’s a sample

Why the Android ecosystem is broken

Six months ago I got this crazy idea. Why not leave the mothership? Give up on iPhone and try Android. This is what being tech-agnostic is about, I thought––not being wedded to a single platform. Besides, the iPhone’s digital keypad just wasn’t working for me.

I got a monthly Boost Mobile plan, which uses Sprint. Service was stellar and I mean really good. I could call anywhere and always had a signal, even inside all these pre-war buildings you find in downtown Manhattan. How is this possible I thought? Service is one thing, but thats where the fun ends. A few months into Android hyperspace and I find myself grappling with a system that just doesn’t seem to understand what users want.

Shock and Awe

On Android – first Samsung Transform Ultra then Sidekick 4G I found the app store was like wild, wild west. Buggy apps sat along well tested ones, and only a very discerning eye and mobile guru might know the difference. Syncing was absolutely horrible. The whole platform assumes you want to sync up with Google accounts. I went with Missing Sync from Mark/Space. My addressbook started getting corrupted, duplicate records appeared, syncs would fail halfway through.

What’s more there were tons of started services I didn’t even use like Smart Navigation, Group Texting, and so on. These services seemed to run in the background, hog & bleed memory, and slow down my phone til it started crashing. I actually had to download an app called Easy Task Killer. Apparently a very popular app on the Android phones, I wonder why?

Later on I found out that T-mobile was no longer supporting the Sidekick. No wonder it was so buggy. I can’t believe they’d ship something like this.

My full list of beefs:

  • corrupted data
  • slow to non-functional syncing
  • dangerous apps
  • sharing of private data
  • ineffective calendaring
  • no support for addressbook groups
  • apps not remembering context & position
  • no good email app
  • weaker less feature rich apps
  • kludgy interface

I’ve long since quelled my desire for a physical keyboard. I was struggling with every other thing I would do with the device. Sometimes I’d just give up.

I really wanted to get along with my Android phone but my experience with it only gave me an insight into three crucial areas where it falls short.

  1. An Iron Fist
  2. People complain about Apple’s iron fist in app store approval. You can’t have it both ways. Android completely lacks discipline and users suffer hugely because of it. That weakens the platform.

  3. A manageable set of devices
  4. Developers building for Android must test on a huge spectrum of hardware. Smaller shops are likely to choose a few of the biggest ones only. Phones with a smaller user base likely have a lot of bugs just in the Android version they run. All this bodes badly as users just see buggy software, they don’t know why or how. This perception further weakens the platform.

  5. Affordable development

Building apps costs businesses money. Businesses must balance the costs of building features, test, debug, troubleshoot & release. That’s cheaper on the iphone because you have one device that is much more mature. This has a pile on affect as it strengthens the platform, users spend more on their devices, more users pile onto the iphone platform, so more money can be made building an iphone apps.  On Android higher costs, lower margins further weakens the platform.

A new love for Apple

This whole experience has brought me back to the iPhone 4S, and I have a whole new love and appreciation for the platform and the device.

  • calendar reminders make me more productive
  • data is always right, and where I need it
  • I don’t fumble with menus – I’m faster & less frustrated
  • cross-platform apps are more feature rich on the iphone
  • complex features are hidden in plain view – superb interface
  • pulse, hootsuite, yelp, notes & calendar are all integrated & productive

I also learned that sometimes less is more… much more!

Scale Quickly Like Birchbox – Startup Scalability 101

One of the great things about the Internet is how it has made it easier to put great ideas into practice. Whether the ideas are about improving people’s lives or a new way to sell and old-fashioned product, there’s nothing like a good little startup tale of creative disruption to deliver us from something old and tired.

We work with a lot of startup firms and we love being part of the atmosphere of optimism and ingenuity, peppered with a bit of youthful zeal – something very indie-rock-and-roll about it. But whether they are just starting out or already picking up pace every startup faces the same challenges to scale a business. Recently, we were reminded of this when we watched Inc’s video interview with Birchbox founders, Hayley Barna and Katia Beauchamp. Continue reading Scale Quickly Like Birchbox – Startup Scalability 101

5 Tips for Better Database Change Management

Deploying new code that includes changes to your database schema doesn’t have to be a process fraught with stress and burned fingers. Follow these five tips and enjoy a good nights sleep.

1. Deploy with Roll Forward & Rollback Scripts

When developers check-in code that requires schema changes, that release should also require two scripts to perform database changes. One script will apply those changes, alter tables to add columns, change data types, seed data, clean data, create new tables, views, stored procedures, functions, triggers and so forth. A release should also include a rollback script, which would return tables to their previous state. Continue reading 5 Tips for Better Database Change Management