Boto3 APIs and Serverless Lambdas are a kickass combination that can help you automate activities on AWS Cloud
Boto3 is the AWS Python SDK!
You can essentially perform any or all of the actions on AWS resources programmatically through python API calls. This already is an extremely powerful automation tool in the hands of a developer.
The idea of this blog is not to teach you how to use boto3. The AWS boto3 documentation does a great job in explaining the nitty gritty of the SDK and its usage. Let us instead look at how (and when) to combine boto3 with Serverless Lambda functions. There are some specific details that could really come handy while doing so and we will also highlight these as well.
You can write python scripts locally and run them against your AWS account. Here are the simple steps to do that:
- Install python3.x
- Install boto3 module
pip install boto3
- Write your code
import boto3 s3_client = boto3.client('s3') response = s3_client.list_buckets() print(response)
If you run this piece of code against your AWS account, it should yield you the results of listing a bucket programmatically.
There are a multitude of things you could do with the power of programmatically performing actions on AWS resources. For example, you could attempt to list out all unencrypted SQS queues and send out a notification, even remediate them through code. You could generate a report to to list out compliant/non-compliant resources and you could configure what these compliance requirements are programmatically in your code.
Lets take it a step further now and look at what happens if we run this code as a lambda function. First of all, its extremely convenient to run the code as lambda functions. Why?
- AWS provides Python lambda runtimes with support for variety of python versions
- Boto3 SDK is bundled as part of the python lambda runtime
- Lambdas are serverless functions that can scale automatically
Secondly, lambda functions can be developed and deployed as part of CI/CD pipelines. This allows us to integrate them as part of regular software development cycles.
Lambdas can be triggered on a schedule, when an event (using eventbridge) occurs, be integrated with API gateway and triggered on a REST API call, on an SNS notification and a wide variety of triggers. There are a ton of possibilities and this leads to a wide range of creative use-cases where we could use them:
- Start and Stop EC2 instances that are tagged with a time schedule
- Detect and report non-compliance and apply automated remediation – for example, when a S3 bucket is created, check whether for encryption and apply encryption if required.
- Identify incidents and report them (or remediate them automatically) – for example, if an AWS resource (like an AMI image) is shared publicly or to another account, send an email notification or remove the share
There are some limitations of lambda and intricacies of boto3 calls that once understood can help you design them better:
- Lambda time out: By default, lambda functions time out after 15 mins. You may want to keep this in mind when designing your functions. For example, if your organization has hundreds of accounts and you are trying to list out all resources, this could take more than 15 mins and you may want to redesign this by splitting your function into multiple functions that list only a certain type of resource
- Some of the boto3 API calls take few minutes to complete. For example, if you are launching a DB cluster, you may want to wait until the cluster comes up (which might take a few minutes) before attempting to update the cluster for encryption. If you think the wait could take beyond 15 mins, may be use queuing to trigger another lambda that does the next set of updates to the resource
Using lambda and boto3, you can build robust and complex automation and when designed well, these can also turn out to be quite reasonable in terms of cost. In fact, the benefits you can extract is virtually limitless and far outweighs the cost in this case.