Are you getting errors building Amazon lambda functions? Don’t fret I got you!

aws lambda python

This post has gotten popular. So I wrote a better easier guide to diving into serverless lambda here. It’s quicker. I promise!

Why does Amazon make lambda functions so hard to create? Well my guess is that when you live at the bleeding edge you should expect to get scrapes!!

Check out: How daily notes help me work better with clients

Everybody is trying to build lambda functions these days. And it’s no wonder. Once you get them running, Amazon takes care of all the infrastructure drudge work! So cool.

Join 32,000 others and follow Sean Hull on twitter @hullsean.

I’ve been spending some time trying to get answers out of AWS support, and let me tell you it’s no fun. Yes all this stuff is new technology, and nobody has expertise in it in the way say you might in Linux or Oracle or another technology that’s been around for a decade.

Still you’d hope the techs would have some clue. In the end it was a slog dealing with support, and I think I was the one teaching them!

I did find Matt Perry’s howto which is pretty good.

Hopefully my own notes can help someone, so read on!

1. No lambda_function?

The very first issue you’re gonna run into is if you name the file incorrectly, you get this error:

Unable to import module 'lambda_function': No module named lambda_function

If you name the function incorrectly you get this error:

Handler 'handler' missing on module 'lambda_function_file': 'module' object has no attribute 'handler'

On the dashboard, make sure the handler field is entered as function_filename.actual_function_name and make sure they match up in your deployment package.

If only the messages were a bit more instructive that would have been a simpler step, but oh well!

Also: Is Amazon too big to fail?

2. No module named MySQLdb

This is a very tricky one. I mean after all you just spent all this time building your deployment package specifically for lambda, so what gives??

"Unable to import module 'lambda_function': No module named MySQLdb"

Turns out when you use a virtualenv, files will be installed into proj/lib/python2.7/site-packages/ or lib64. However Lambda wants them in the root proj/ directory! So move them there. I know I know. Weird, but that’s what they want.

Related: When hosting on Amazon turns bloodsport

3. Can’t find libmysqlclient

If you’re using the MySQLdb library like I was, you’ll eventually bump into this error:

Unable to import module 'lambda_function': cannot open shared object file: No such file or directory

Turns out that /usr/lib/ needs to be COPIED from /usr/lib. Don’t do “mv” or your system won’t have the mysql lib anymore!

Related: Are SQL databases dead?

4. Use the Amazon Lambda environment

One thing the support pointed out is that AWS as *supported images* for lambda development.

After all the errors above were resolved, it’s not clear to me that the supported AMI’s are truly required. However if you’re hitting intractable problems building a properly lambda deploy, you might wanna look at building one of these boxes.

Read: Why dropbox didn’t have to fail

5. Build your lambda deploy package

Now let’s roll it all together. Here’s are all the steps to build your deploy package.

- SSH to the instance
- mkdir test
- virtualenv test
- source proj/bin/activate
- sudo yum groupinstall 'Development Tools'
- sudo yum install mysql
- sudo yum install mysql-devel
- pip install MySQL-python
- cd test
- emacs -nw
- add your code to that file
- save the
- mv proj/lib/python2.7/site-packages/* proj/
- mv proj/lib64/python2.7/site-packages/* proj/
- rm -rf proj/lib (don't need dist-packages in the deploy pkg)
- rm -rf proj/lib64 (don't need dist-packages
- zip -r *

Also: How to hire a developer that doesn’t suck?

6. Upload your code

Uploading your code via the AWS dashboard is fine when you’re first testing things. But after a while it’ll get tiring going in the front door.

Create a new lambda function by specifying the basics as follows:

aws lambda create-function \
--function-name testfunc1 \
--runtime python2.7 \
--role arn:aws:iam::996225510001:role/lambda_basic_execution \
--handler lambda_function_file.handler_name \
--zip-file file://

And when you want to update your function, do the following:

aws lambda update-function-code \
--function-name testfunc1 \
--zip-file file://

Also: How to deploy on EC2 with Vagrant

Good luck with lambda. Once you get past Amazon’s weak documentation it’s pretty cool to be in a serverless computing environment. Happy deploying!

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Also published on Medium.

  • Guy

    Thanks for writing this! For step number 2, I did the following:
    Went to python2.7/site-packages/ and took everything related to MySql

    Then I zipped the content with my, and then I get the following:
    Unable to import module ‘lambda_function’: /var/task/ invalid ELF header

    If I remove that file, then I’m back to:
    Unable to import module ‘lambda_function’: No module named MySQLdb

    Do you know why?

    • Sean Hull

      Did you also copy /var/lib/ into proj/ and then symlink proj/ to it?

  • Si Vis Pacem, Para Bellum

    Any idea how I would call on a .wav file stored on s3? I just can’t seem to get that working. I found an example of reading an image using blob=response[‘Body’].read() where filename would be, but that’s not worked for me.

    • Sean Hull

      Are you in python or node?

      • Si Vis Pacem, Para Bellum

        Dang, you’re fast. I’m in python. I just found something that looks like I can just use boto3 to s3.download_file(bucket, key, ‘/tmp/’+key) and then use‘/tmp/’+key), or at least I’m thinking that would work.

        • Sean Hull

          yeah boto3 is there by default.

          • Si Vis Pacem, Para Bellum

            Appreciate your quick response, very nice of you. It did work, I’m getting error down the line a bit but that’s a separate matter. I hadn’t realized there was a local filesystem I could pull files to from s3.

          • Sean Hull

            Yeah. Inside lambda you’re basically inside a container. Like docker. I think they use Linux LXC project

          • Si Vis Pacem, Para Bellum

            Right, I’m picking up on that now, I’m new to aws lambda and just didn’t realize it.

          • Sean Hull

            You’re not alone. Everybody’s new to lambda ๐Ÿ™‚

  • Anonymous Eagle

    Turns out that /usr/lib/ needs to be COPIED from /usr/lib.

    Copied from /usr/lib TO ??

  • Sille kyatha

    Facing this error, while trying to connect to RDS postgresql- “module initialization error: name ‘conn’ is not defined”

    • Sean Hull

      Sounds like you didn’t instantiate the db connection object.

  • Praveen Vasireddy

    I’m getting an error “Unable to import module ‘lambda_function’: No module named paramiko”. I’m trying to import paramiko to SSH into an EC2 instance. How to overcome this? Thanks in advance.

    • Sean Hull

      I’m not sure what you mean by “ssh into ec2 instance”. With lambda you write your code, package it up & send it to AWS using API call. Then you hit an endpoint or other AWS event will trigger it to run. Not sure when/how you would have an ec2 instance in the mix here.

  • Sean Hull

    If you’re jumping through all those hoops, perhaps lambda is not the best option for you? Why not just spinup the ec2 instance & run the code there. Yes lambda has limitations. It probably always will.

  • David Jorjani

    Hi @praveen_vasireddy:disqus I’m doing the same thing you were trying and getting the same error. Lambda doesn’t recognize the folder structure. Can you please share how you resolved the issue?

    • Sean Hull

      Hi Praveen thx for asking. Lambda is tricky at first. So tell me more about what you’re doing. What library are you trying to load?

      One thing I can recommend. I was digging through the awslabs GitHub repo, and found a project called serverless-image-resize. I wanted to try it out so I checked out the code. Turns out they SPINUP A DOCKER CONTAINER just for the purposes of building their lambda artifact cleanly. They use the amazon-linux docker image, and then npm install packages, then zip it up. Lastly they use cloudformation to push the code to AWS.

      • Sean Hull

        Hi @Praveen, also have you looked at the serverless framework?]

        It is a node package, so install node on your local system first, then do “npm install -g serverless”

        From there do this:
        serverless create –template aws-python –path myService

        Inside of the new folder myService you’ll find two really cool files: <– skeleton of your new python lambda code
        serverless.yml <– template controlling all objects around your new lambda code

        The great thing about the serverless framework is that it organizes your thinking around everything in the serverless universe. Need an API endpoint? a swagger template does the job. Need IAM users with certain permissions or a specific S3 bucket? You'll also define those. When you further use the serverless command to deploy, it will put together the right cloudformation code necessary to build those objects. It will also make sure your names are consistent, to avoid problems like the one you described above.

        Let me know if this help you.