Last October, I heard this crazy idea from a guy at a startup event. The gist: run an application without setting up any servers. Just set up your code on AWS (Amazon Web Services) Lambda, and set up your API using AWS API Gateway. When an API endpoint is hit, you respond by executing code in AWS Lambda. Yeah. No servers. Just use the services AWS has to complete the request (of which there are many). I was thinking, "Yeah, right, that's just nuts."
After about 4 months, I found a use case to give it a try. I wanted to set up an endpoint, and do some processing on the payload, and then put some of the data into SQS (Simple Queue Service) for processing by another service. If successful, we'd be able to shut off 2 servers and do this small task in the cloud.
Note: This looks like a long, complex article, but it's easy as pie. There are just a lot of pictures.
The project files are available on Github:
Here are the components, and what they do:
- AWS API Gateway - API hosting and management service
- AWS Lambda - execute code without provisioning server resources
- AWS SQS (Simple Queue Service)
- AWS IAM (Identity & Access Management)
This tutorial assumes you already have an AWS account, and you know a little something about SQS.
Creating the API
Head over to AWS API Gateway and create a new API:
The name here is not part of your API url. Give it a nice name, and click Create API.
Adding a resource
Now click Create Resource. The resource creates the url path in the API.
This is how you build the url. So if you want /v1/something you need to make a v1 resource and then a something resource under that.
Since I just need an endpoint called /message, I'll make a message resource.
Enter your resource name and click Create Resource.
Now that the resource has been made, you'll see the path on the left:
The resource is nothing without a way to interact with it. I'll make a POST for posting a message to the /message endpoint.
Click the Create Method button. You'll get a little HTTP method dropdown under the resource on the left.
Select POST, and click the Checkbox to save it. Now we are presented with the options to handle the invocation of the resource using that method.
For this, I'm going to use a Lambda Function.
I'm a big fan of us-east-1 so I chose it as my region. Then a little message pops up since I don't have any lambdas yet. Click the link in the message.
Creating a Lambda
Don't worry too much about this part. It's super easy.
You'll see a big list of blueprints to start with, and you're free to select one that fits your purpose. There wasn't one for my needs so I clicked skip.
Now we'll configure the function.
Give the function a name. My lambda function will insert the API payload into a queue. So I called it enqueue.
You'll need to choose the language to write the function in. Currently AWS Lambda supports:
- Node.js
- Java version 8
- Python 2.7
I'm using Python. Many more languages are on the roadmap.
About Python module support
I'm not going to get into too much detail here, but there are many packages supported. This includes the boto3 package, which is the go-to Python package for interacting with AWS APIs. One of the great things is that you don't need to specify any of your AWS keys, since you're working within the context of AWS.
Provide the code for your function in the text area of the page. Use the editor if your code does not require custom libraries. If you need custom libraries, you can upload your code and libraries as a .ZIP file.
Read more about packaging your application here.
The lambda code
Now I'll specify the Python code in the box. Mine is very simple, and doesn't require any self-made packages. This is it in its entirety:
import boto3
import json
def lambda_handler(event, context):
sqs = boto3.resource('sqs')
queue = sqs.get_queue_by_name(QueueName='test-messages')
response = queue.send_message(MessageBody=json.dumps(event))
You can see I'm importing boto3 and json, both included in AWS.
I set up a connection to my SQS queue, called "test-messages".
The final line encodes the event payload and adds it to the queue.
The important parts of this are:
lambda_handler- this is the function we must define that will be called when a trigger event happens. The trigger event in this case is the call to our API endpoint.- event - the payload received in the API call
- context - runtime information about the call
You'll notice I'm not specifying any AWS keys. As I mentioned above:
One of the great things is that you don't need to specify any of your AWS keys, since you're working within the context of AWS.
The code in place:
Setting up the IAM Role
The second part of making the Lambda function is creating a special role for your Lambda so that it can run AWS actions on your behalf. You don't make an IAM user for this. You make a role. This page has more info, but the key part is the second paragraph:
Regardless of how your Lambda function is invoked, AWS Lambda always executes the function. At the time you create a Lambda function, you specify an IAM role that AWS Lambda can assume to execute your Lambda function on your behalf. This role is also referred to as the execution role. If your Lambda function accesses other AWS resources during execution (for example, to create an object in an Amazon S3 bucket, to read an item from a DynamoDB table, or to write logs to CloudWatch Logs), you need to grant the execution role permissions for the specific actions that you want to perform using your Lambda function.
We'll handle that now. Below where you specified the lambda code, you'll see this section:
Do as I've done here, and choose lambda_basic_execution (the recommended option).
It provides a a very basic security policy that only covers logging the Lambda activity to CloudWatch.
Edit the policy and add additional permissions as needed, as I've done here to allow for queue insertion:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": "arn:aws:logs:*:*:*"
},
{
"Action": [
"sqs:SendMessage",
"sqs:GetQueueUrl"
],
"Effect": "Allow",
"Resource": "arn:aws:sqs:us-east-1:SOME_ID_HERE:test-messages"
}
]
}
Once you save the Lambda, we'll head back at the API Gateway UI.
Now you can select the new lambda by typing its name.
Once saved, you're presented with a nice flow diagram:
Testing the API
Now you can run a little test by clicking the lightning bolt:
Enter a bit of JSON in the little box:
Then test it to make sure you don't get errors.
Deploy the API
Let's take this live!
Click the Deploy API button.
I made a stage called production. For business purposes, you will likely have a development stage, testing stage, pre-production stage, and production stage. Each of this will have its own URL, and you can deploy changes to your API to one stage without affecting the other stages.
Once you click Deploy, your API is up and ready for traffic.
The base URL of your new API endpoint is in blue at the top.
The API url will have this pattern:
https://YOUR_API_ID.execute-api.THE_REGION.amazonaws.com/STAGE_NAME
To get the path of your endpoint to hit, click to open the tree on the left.
Unfolding this will show you the path to the specific resource.
In my case, the API URL is:
https://THE_API_ID.execute-api.THE_REGION.amazonaws.com/production/message
where "/message" is the path of my api as seen in the gateway API UI.
There are a couple options here I want to point out:
If you enable the API cache, you can cache responses to avoid the need to hit your Lambda. I won't use it for this project. Investigate that on your own.
I enabled CloudWatch metrics so I can see how often the endpoints in this API are accessed.
Testing the Endpoint
On the command line, run this, replacing with your endpoint:
curl -H "Content-Type: application/json" -X POST -d '{"test":1}' https://YOUR_API_ID.execute-api.THE_REGION.amazonaws.com/YOUR_STAGE_NAME/message
If you get null as a response, it's probably fine. Just check to make sure the Lambda did what you expected. In my case I checked the queue to make sure the payload from the call was in it.
If you get an error about a missing authorization token, make sure to re-deploy the API to pick up any changes you've made. It is a general error, so it can also mean you're hitting a 404.
For a very simple Lambda for testing a GET call, add another method to your resource, with this as the Lambda:
def lambda_handler(event, context):
return 'hello there'
Then hit that endpoint with a GET (by just putting the url in your browser) and you'll see "hello there"
API Keys
By default, your API is open to the public (if they know the full url path). You can lock it down by creating API keys and setting the API to always require a key. I won't cover that here.
Caveats
Lambda Container Initialization
When a Lambda function starts up, it is run in a container, so it can take up to 2 seconds to execute because the container has to start running. At scale, you shouldn't run in to this problem, because the container will not shut down immediately. More API calls soon after will keep the container "warm" so it will be very responsive. But an idle function will close down after a few minutes. Then you would run into that container start lag on the next invocation. If this is a problem and you have sporadic traffic, you can set up a process to periodically hit your endpoint to keep it warm.
Pricing
You're only billed for the time your Lambda function is running. You are billed in 100 ms increments. There is a free tier, so it's extremely affordable, but keep your eye on the bills. AWS Lambda Pricing
Want to Know More?
This AWS webinar from December 15, 2015 was a nice introduction to the services listed above: