Adding to an SQS Queue Using AWS Lambda and a Serverless API Endpoint

Last October, I heard this crazy idea from a guy at a startup event. The gist: run an application without setting up any servers. Just set up your code on AWS (Amazon Web Services) Lambda, and set up your API using AWS API Gateway. When an API endpoint is hit, you respond by executing code in AWS Lambda. Yeah. No servers. Just use the services AWS has to complete the request (of which there are many). I was thinking, "Yeah, right, that's just nuts."

After about 4 months, I found a use case to give it a try. I wanted to set up an endpoint, and do some processing on the payload, and then put some of the data into SQS (Simple Queue Service) for processing by another service. If successful, we'd be able to shut off 2 servers and do this small task in the cloud.

Note: This looks like a long, complex article, but it's easy as pie. There are just a lot of pictures.

AWS API Gateway to AWS Lambda to AWS SQS Insertion Flow

The project files are available on Github:

Here are the components, and what they do:

  • AWS API Gateway - API hosting and management service
  • AWS Lambda - execute code without provisioning server resources
  • AWS SQS (Simple Queue Service)
  • AWS IAM (Identity & Access Management)

This tutorial assumes you already have an AWS account, and you know a little something about SQS.

Creating the API

Head over to AWS API Gateway and create a new API:

creating a new AWS API

The name here is not part of your API url. Give it a nice name, and click Create API.

Adding a resource

Now click Create Resource. The resource creates the url path in the API.

This is how you build the url. So if you want /v1/something you need to make a v1 resource and then a something resource under that.

making an AWS API resource

Since I just need an endpoint called /message, I'll make a message resource.

making a resource

Enter your resource name and click Create Resource.

Now that the resource has been made, you'll see the path on the left:

resource made

The resource is nothing without a way to interact with it. I'll make a POST for posting a message to the /message endpoint.

Click the Create Method button. You'll get a little HTTP method dropdown under the resource on the left.

http methods

Select POST, and click the Checkbox to save it. Now we are presented with the options to handle the invocation of the resource using that method.

For this, I'm going to use a Lambda Function.

API handler options - lambda selected

I'm a big fan of us-east-1 so I chose it as my region. Then a little message pops up since I don't have any lambdas yet. Click the link in the message.

no lambdas yet, create one

Creating a Lambda

Don't worry too much about this part. It's super easy.

You'll see a big list of blueprints to start with, and you're free to select one that fits your purpose. There wasn't one for my needs so I clicked skip.

AWS Lambda blueprints

Now we'll configure the function.

Give the function a name. My lambda function will insert the API payload into a queue. So I called it enqueue.

You'll need to choose the language to write the function in. Currently AWS Lambda supports:

  • Node.js
  • Java version 8
  • Python 2.7

I'm using Python. Many more languages are on the roadmap.

AWS Lambda supported languages

About Python module support

I'm not going to get into too much detail here, but there are many packages supported. This includes the boto3 package, which is the go-to Python package for interacting with AWS APIs. One of the great things is that you don't need to specify any of your AWS keys, since you're working within the context of AWS.

Provide the code for your function in the text area of the page. Use the editor if your code does not require custom libraries. If you need custom libraries, you can upload your code and libraries as a .ZIP file.

Read more about packaging your application here.

The lambda code

Now I'll specify the Python code in the box. Mine is very simple, and doesn't require any self-made packages. This is it in its entirety:

import boto3  
import json

def lambda_handler(event, context):

    sqs = boto3.resource('sqs')

    queue = sqs.get_queue_by_name(QueueName='test-messages')

    response = queue.send_message(MessageBody=json.dumps(event))

You can see I'm importing boto3 and json, both included in AWS.
I set up a connection to my SQS queue, called "test-messages".
The final line encodes the event payload and adds it to the queue.

The important parts of this are:

  • lambda_handler - this is the function we must define that will be called when a trigger event happens. The trigger event in this case is the call to our API endpoint.
  • event - the payload received in the API call
  • context - runtime information about the call

You'll notice I'm not specifying any AWS keys. As I mentioned above:

One of the great things is that you don't need to specify any of your AWS keys, since you're working within the context of AWS.

The code in place:

AWS lambda code in place

Setting up the IAM Role

The second part of making the Lambda function is creating a special role for your Lambda so that it can run AWS actions on your behalf. You don't make an IAM user for this. You make a role. This page has more info, but the key part is the second paragraph:

Regardless of how your Lambda function is invoked, AWS Lambda always executes the function. At the time you create a Lambda function, you specify an IAM role that AWS Lambda can assume to execute your Lambda function on your behalf. This role is also referred to as the execution role. If your Lambda function accesses other AWS resources during execution (for example, to create an object in an Amazon S3 bucket, to read an item from a DynamoDB table, or to write logs to CloudWatch Logs), you need to grant the execution role permissions for the specific actions that you want to perform using your Lambda function.

We'll handle that now. Below where you specified the lambda code, you'll see this section:

basic execution role

Do as I've done here, and choose lambda_basic_execution (the recommended option).

create a new IAM role

It provides a a very basic security policy that only covers logging the Lambda activity to CloudWatch.

Edit the policy and add additional permissions as needed, as I've done here to allow for queue insertion:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "arn:aws:logs:*:*:*"
    },
    {
      "Action": [
        "sqs:SendMessage",
        "sqs:GetQueueUrl"
      ],
      "Effect": "Allow",
      "Resource": "arn:aws:sqs:us-east-1:SOME_ID_HERE:test-messages"
    }
  ]
}

Once you save the Lambda, we'll head back at the API Gateway UI.

lambda complete

Now you can select the new lambda by typing its name.

select lambda at AWS API UI

Once saved, you're presented with a nice flow diagram:

AWS API flow with Lambda

Testing the API

Now you can run a little test by clicking the lightning bolt:
test lightning bolt

Enter a bit of JSON in the little box:

api test flow

Then test it to make sure you don't get errors.

Deploy the API

Let's take this live!

Click the Deploy API button.

deploy api button

I made a stage called production. For business purposes, you will likely have a development stage, testing stage, pre-production stage, and production stage. Each of this will have its own URL, and you can deploy changes to your API to one stage without affecting the other stages.

make a deployment stage

Once you click Deploy, your API is up and ready for traffic.

stage created

The base URL of your new API endpoint is in blue at the top.

The API url will have this pattern:

https://YOUR_API_ID.execute-api.THE_REGION.amazonaws.com/STAGE_NAME  

To get the path of your endpoint to hit, click to open the tree on the left.

path in stage

Unfolding this will show you the path to the specific resource.

In my case, the API URL is:

https://THE_API_ID.execute-api.THE_REGION.amazonaws.com/production/message  

where "/message" is the path of my api as seen in the gateway API UI.

There are a couple options here I want to point out:

stage options

If you enable the API cache, you can cache responses to avoid the need to hit your Lambda. I won't use it for this project. Investigate that on your own.

I enabled CloudWatch metrics so I can see how often the endpoints in this API are accessed.

Testing the Endpoint

On the command line, run this, replacing with your endpoint:

curl -H "Content-Type: application/json" -X POST -d '{"test":1}' https://YOUR_API_ID.execute-api.THE_REGION.amazonaws.com/YOUR_STAGE_NAME/message  

If you get null as a response, it's probably fine. Just check to make sure the Lambda did what you expected. In my case I checked the queue to make sure the payload from the call was in it.

If you get an error about a missing authorization token, make sure to re-deploy the API to pick up any changes you've made. It is a general error, so it can also mean you're hitting a 404.

For a very simple Lambda for testing a GET call, add another method to your resource, with this as the Lambda:

def lambda_handler(event, context):  
    return 'hello there'

Then hit that endpoint with a GET (by just putting the url in your browser) and you'll see "hello there"

API Keys

By default, your API is open to the public (if they know the full url path). You can lock it down by creating API keys and setting the API to always require a key. I won't cover that here.

Caveats

Lambda Container Initialization

When a Lambda function starts up, it is run in a container, so it can take up to 2 seconds to execute because the container has to start running. At scale, you shouldn't run in to this problem, because the container will not shut down immediately. More API calls soon after will keep the container "warm" so it will be very responsive. But an idle function will close down after a few minutes. Then you would run into that container start lag on the next invocation. If this is a problem and you have sporadic traffic, you can set up a process to periodically hit your endpoint to keep it warm.

Pricing

You're only billed for the time your Lambda function is running. You are billed in 100 ms increments. There is a free tier, so it's extremely affordable, but keep your eye on the bills. AWS Lambda Pricing

Want to Know More?

This AWS webinar from December 15, 2015 was a nice introduction to the services listed above:

Thank to Joel Sutherland for his really nice Github widget.

comments powered by Disqus