AWS VPC-Lambda Security: Internet always on
/0 Definitions
Let’s define a couple of relevant terms for the purpose of this article:
- “Public AWS services”: services in a AWS account which can have Incoming and/or Outgoing Internet connectivity directly through AWS, without being part of any VPCs and Internet Gateways.
- “VPC-Lambdas” or “VPC-enabled Lambdas”: Lambdas configured in one of your VPCs and subnets (implemented through AWS hyperplane ENIs).
/1 Intro
In the vast majority of cases, VPC-bound AWS services follow the same behavior: both incoming and outgoing Internet traffic require the usage of an Internet/NAT Gateway.
VPC-Lambdas, on the other hand, behave differently in such a way that it might importantly impact the security of customers’ AWS infrastructures. In this article, we take a look at how and why.
/2 The Lambda Compute Layer
The entire Lambda compute layer in AWS is part of what is referred to as the “AWS Lambda Service VPC”, an infrastructure which consists of VPCs run by AWS itself (outside of any actual AWS customer accounts). When creating Lambdas in our consumer AWS accounts, they can be configured in one of the following 2 different methods:
1) As a Public AWS services mode
or,
2) As a VPC-Lambda mode
In the first scenario (Public AWS Services mode) incoming and outgoing internet traffic is offered purely through AWS. In other words, it makes no difference if an Internet or NAT gateway is configured, since these components are actually VPC-bound services. There is not any way to restrict or limit Internet connectivity to public AWS service Lambdas as far as I am aware.
In the second scenario (and the goal of this article), is when things get a bit “cloudy”, and we shall take a look why. Let’s dive into it.
/3 Lambda triggers
Lambda triggers can be configured differently and individually for each Lambda in our account(s). They include for example, API Gateway, EventBridge, SQS and, wait for it: Lambda-URLs. When these triggers are configured, they are automatically added by AWS to the Lambda’s own resource policy.
Whether through a VPC-Lambda, or through the public AWS services mode, Lambda triggers are actually part of the AWS Control Plane, and as such, they obviously do not go through any VPCs or Internet Gateways. Nothing new here. So what is special about VPC-Lambdas then?
Let’s first take a look at an EC2 configured as part of a VPC. If we completely disable all Internet Gateway(s) in that VPC (directly within the same account, or, indirectly through other network accounts), then there will be no direct incoming or outgoing Internet traffic possible to that EC2. However, we are of course still able to call the Control Plane APIs, such as starting the EC2 instance itself.
When we compare all of this to a VPC-Lambda, the Internet access behavior is actually different:
- Outgoing internet traffic would only be allowed if Security Groups allow it, as well as having an Internet Gateway enabled. Nothing special here, as this holds true for VPC-EC2s as well.
- Incoming Internet traffic however will still be possible through Lambda triggers, even if the Security Groups have zero inbound rules, and, no Internet Gateways are attached. The output or actions performed will of course depend on the code of the Lambda itself, but it is important to know and take note of this behavior.
Stated differently, unless we completely limit and/or disable Lambda triggers, VPC-Lambdas (and of course public-service lambdas) will always have incoming Internet connectivity, through for example, Lambda-URLs configured with No-Auth, or, an API Gateway integration. There is no other way to stop or limit that behavior, as far as I am aware.
/4 Why do we use Lambda in a VPC then?
Simple answer: to allow VPC-Lambdas to be able to talk to other VPC services, such as EC2 and RDS. We quickly realize that inbound Security Group rules for Lambda don’t really serve any purposes at all, because Lambdas are meant to be triggered (invoked through the control plane APIs). They are not really listening on a consumer-accessible application-level port like for example EC2s or RDSs do.
/5 Lab
A simple proof of concept could be quickly done:
- Create a Lambda as part of a VPC, in any Region or subnet combination. You can use the simple and default “Hello from Lambda” code
- Attach a Security Group with no inbound rules. Outbound rules don’t matter for this lab.
- Disable any Internet Gateways for that region and VPC
- Configure a Lambda-URL with No-Auth and test it from anywhere on the Internet: it will output the Lambda code
- Alternatively, create a Regional or Edge-Optimized API Gateway, and configure it to proxy traffic to your Lambda’s ARN. Same behavior: the Lambda code is executed and given back to the user through the AWS API GW public endpoint.
/6 Solutions and recommendations
Every AWS account and AWS infrastructure I have worked with uses Lambdas. It is a service everyone loves and implements, highlighting the infinite possibilities of serverless architectures in the Cloud.
Most Cloud or Security Engineers would believe that placing Lambdas in a VPC would naturally offer a complete “Internet-isolation and security”, but in reality this would only be partially true, meaning, only in terms of outgoing traffic from VPC-Lambda to the Internet.
Lambda triggers are however a critical part of the AWS Lambda service itself. I don’t personally believe that limiting some or all Lambda triggers is a practical solution. I have written another article about Lambda-URLs with No-Auth, and those could be SCP-restricted. But for all other available Lambda triggers, it could be a very tough or even impossible decision to make, if we plan to actually use the AWS Lambda service itself.
At this point, keeping an eye and monitoring which Lambda triggers are configured (through Lambda’s resource policy for example), for each and every single Lambda (whether part of a VPC or not), will be yet another important contributing factor to your corporate AWS security posture. With hundreds or thousands of Lambdas configured in a typical AWS infrastructure (across many AWS accounts), this might prove to be a challenging task.
/7 Conclusion
Understanding how “Public AWS services” behave, on a service per service level, is a critical part of any AWS security architecture and design consideration.
VPC-Lambdas inhibits unique behaviors when it comes to Inbound Internet connectivity when compared to other VPC-bound services.
Looking further ahead, I would really like to see one or more of the following steps implemented in the future, in order to make the subject less of a somewhat undocumented mystery:
- Console-based warnings within the Lambda service itself
- Implementing new Lambda functionalities, such as a “block Internet” switch within the consumer Lambdas available in their own accounts (if that is a possibility)
- I believe that we should have a list of all public AWS services available, to be documented and provided by AWS, so customers/clients can investigate or secure them as needed. This is important, as new features and services are pushed very frequently by AWS
- Lastly, I believe we could definitely use additional AWS documentations, specially in the AWS docs regarding VPC networking for Lambda, to explain and clarify the overall behavior we discussed here.
Thank you for reading, and I hope you enjoyed this article!
References:
