Welcome to Part 3 of the blog series on NSX Multitenancy. In this article, we will discuss NSX Virtual Private Clouds (VPCs) that is an extension to NSX projects (discussed in Part 2) by offering an additional management plane isolation within the context of an NSX project. Using VPCs, we segment the management plane even further and we offer the consumers (who are mainly application owners or developers), a simplified self-service portal where they can consume the networking and security objects independently in their own private space.
If you were not following along, please check out the previous parts below:
Part 1 – Introduction & Multitenancy models :
https://vxplanet.com/2023/10/24/nsx-multitenancy-part-1-introduction-multitenancy-models/
Part 2 – NSX Projects :
https://vxplanet.com/2023/10/24/nsx-multitenancy-part-2-nsx-projects/
Let’s get started:
Virtual Private Cloud (VPC) Architecture
In the previous part (Part 2), we created a project called “Project_AppSec” for the Application Security team who required a dedicated tenant for their workloads. We now have a requirement from the application owners of Project_AppSec mainly from the “Ondemand Testing Service (OTS)” applications and “Insite Security (IS) ” applications to:
- Offer them with a simplified self-service management UI so that they can rapidly onboard their workloads.
- Ensure management plane isolation between the application workloads within the tenant.
We accomplish this with NSX Virtual Private Clouds (VPCs). We create two VPCs under the project called AppSec_OTS_VPC and AppSec_IS_VPC.
RBAC policies are then configured at the VPC level so that application owners can access and manage their own VPCs only.
The below sketch shows the logical architecture of the two VPCs – AppSec_OTS_VPC and AppSec_IS_VPC that are created under the project.
As you can see, there are three different networks within a VPC, we call it as VPC subnets. A subnet is nothing, but an overlay segment created in the default overlay transport zone. Workloads in a VPC can be attached to any of the below subnets:
- Public subnet : This subnet is advertised from the VPC to the Provider T0 gateway and is routed to the external fabrics by the T0 gateway. The Enterprise administrator creates the external (public) IP block in the default space and will be shared to projects which can be then consumed within the project VPCs. A CIDR is carved out from this block for use in public subnets and NAT rules within the VPC.
- Private subnet : This subnet is private and can be used for communication between workloads within the same VPC only. Workloads on private subnet can communicate with workloads on other private and public subnets in the same VPC. Optionally workloads on private subnets can have outbound access to external networks using SNAT.
- Isolated subnet : This subnet is isolated and does not have upstream gateway connectivity. Workloads in an isolated segment can communicate with workloads in the same isolated subnet only (ie, only L2).
Both public and private subnets are attached to an auto-created VPC T1 gateway. This is system managed and will be destroyed once the VPC is deleted. Each VPC will have it’s own dedicated T1 gateway.
A couple of things to note:
- VPCs consume the T0 / VRF gateways which are assigned to the project. Anything that is not assigned to the project cannot be consumed by the VPCs.
- VPCs consume the edge clusters which are assigned to the project. This can be the Provider shared edge cluster or a dedicated edge cluster for the project.
- A VPC gateway (T1 gateway) is automatically created under the project whenever a VPC is created. This gateway will be read-only to the project admins and is not visible inside the VPC. This T1 gateway will be upstreamed to the T0 / VRF gateway specified during the creation of VPC.
- Only one T0 / VRF gateway and one edge cluster can be associated to a VPC.
- Security policies created in a VPC will be applicable only to the workloads within the VPC.
Whitelisting VPC public subnets in Project route filter
Remember? In Part 2, we implemented Project Route Advertisement control for Project_AppSec where we authorized only the subnet 192.168.20.0/24 to be advertised from the project T1 gateways. This means that the external IP blocks we are going to define for the VPC’s public subnets need to be whitelisted before we can assign them to subnets and workloads.
Note : This needs to be done before creating the VPC.
Let’s login to the default space as hari@vxplanet.int who is a member of “NSX_Enterprise_Admins” and verify that we have the external IP block named “Project_AppSec_Public” (which we created in Part 2). This IP block is used for the public subnets in VPCs.
Optionally, we could add additional IP blocks (eg: Project_AppSec_Public_2), and associate to project “Project_AppSec”.
In order to whitelist the public subnets, we will update the prefix list (created in Part 2) named “project_appsec_whitelist” with the VPC public subnets 192.168.10.0/24 and 192.168.11.0/24.
As, we have already associated this prefix-list to the Project_AppSec’s route filter in Part 2, the change should take effect immediately. Once the VPCs are set up in the next section, we should see the public subnets from the VPCs advertised to the Provider T0 gateway.
Creating Virtual Private Clouds (VPCs)
Now that we created the external IP blocks and whitelisted them in the project route filter, let’s log back in as alec@vxplanet.int who is the project admin for “Project_AppSec” and create few private IP blocks that can be used for the private subnets in VPCs.
We will now create the two VPCs – AppSec_OTS_VPC and AppSec_IS_VPC.
Note : VPCs are created from the respective projects (Project_AppSec in our case) by the Project Admin or Enterprise Admin.
AppSec_OTS_VPC will use the external IP block “Project_AppSec_Public” and the private IP block “Private_Block_OTS”. Similarly, AppSec_IS_VPC will use the external IP block “Project_AppSec_Public_2” and the private IP block “Private_Block_IS”.
It’s also possible to share IP blocks across multiple VPCs. For eg: the external IP block “Project_AppSec_Public “ can be shared across both VPCs, but for this blog post , let’s keep them as separate.
The T0 / VRF gateway selected in the VPC will be used for attaching the VPC T1 gateways.
The edge cluster selected in the VPC will be used for instantiating the SR construct of the VPC T1 gateway (Active – Standby) if stateful services are enabled.
The option for “Default Outbound NAT” will enable SNAT for the private subnets in the VPC gateway, so that workloads on private subnets can communicate to external networks.
For the preferred IP addressing scheme for workloads, we will choose segment DHCP which is the setting called “Managed by NSX Policy Management”.
We can see the VPC T1 gateways created under the context of the project “Project-AppSec”. They are upstreamed to the Provider T0 gateway. Lifecycle of these T1 gateways is managed by the system.
Assigning RBAC policies to VPCs
RBAC policies are assigned by the NSX Enterprise Administrator. However, the Enterprise Administrator can delegate role assignment privilege to a project Admin, so that the project admin can manage the RBAC policies of the VPCs within the project.
Let’s assign the role assignment privilege to alec@vxplanet.int who is the project admin of Project_AppSec.
Now let’s log back in as alec@vxplanet.int and assign the groups AppSec_OTS_VPC_Admins and AppSec_IS_VPC_Admins as VPC admins to AppSec_OTS_VPC and AppSec_IS_VPC respectively.
Let’s test the RBAC policies. We will login as ben@vxplanet.int who is a member of AppSec_OTS_VPC_Admins group. Notice that ben@vxplanet.int got his own private management portal and has access only to those VPCs where he is assigned access to.
VPC Admins cannot modify the IP blocks assigned to the VPC, but can create subnets, onboard workloads, configure E-W & N-S firewall policies and NAT rules.
We could add users with other roles (non-admin roles) to the VPC:
- The project admin can do this from the respective VPCs (like how we added VPC Admins) or
- Enterprise Admin can delegate role assignment privilege to a VPC Admin who can add respective users to their VPCs.
Now ben@vxplanet.int (VPC Admin) has the privilege to add other users to his VPC.
Now let’s test the RBAC policies on AppSec_IS_VPC. We will login as mario@vxplanet.int who is a member of AppSec_IS_VPC_Admins.
Mario has now successfully logged into his assigned VPC ”AppSec_IS_VPC”.
Assigning Quotas to VPCs
Quotas are assigned to VPCs either by the Enterprise Admins (from the default view) or by the respective Project Admins (from the project view).
Quotas can be applied to:
- Objects that are created inside the respective VPC (from the project view) or
- Objects created across all the VPCs inside the respective project (from the default view)
Let’s login as Project Admin (alec@vxplanet.int) and apply quotas to some objects in AppSec_OTS_VPC.
What we did just now was that we applied a quota for the total number of subnets that can be created inside this VPC (ie: 5 subnets).
Now let’s login as Enterprise Admin and apply quotas to objects across all the VPCs in the project. Note that we cannot apply quota to a specific VPC from the default view.
What we did just now was that we applied a quota for the total number of subnets that can be created across all the VPCs in the project “Project_AppSec” (ie: 3 subnets).
Curious to know which quota takes effect on the VPC “AppSec_OTS_VPC”?
The answer is, if multiple quotas are applied to the same object, the one with lower value takes effect. In this case, the quota applied across all the VPCs from the default view (3 subnets) will override the quota applied at the individual VPC level (5 subnets). Let’s check this.
As you can see, creation of the 4th VPC subnet failed with quota limit reached error which shows that the quota applied from the default view took precedence.
We can also check the quota status from the VPC UI.
Creating VPC Subnets
As discussed in the VPC architecture, there are three subnet access modes available:
- Private subnets
- Public subnets and
- Isolated subnets
Let’s login as ben@vxplanet.int who is the VPC Admin of AppSec_OTS_VPC, and create the three subnets.
For the private subnet, a CIDR block is carved out from the assigned private IP block based on the number of workloads specified:
For the public subnet, a CIDR block is carved out from the assigned external IP block based on the workload size specified:
For the isolated subnet, we need to provide a manual CIDR block. Note that this subnet doesn’t have an upstream connectivity, hence overlapping CIDRs are allowed.
We see that the private and public subnets are automatically attached to the VPC T1 Gateway. This is visible only under the context of projects by the Project Admins.
If we check the route advertisement rules on the VPC T1 Gateway, we see that only the public subnet is advertised.
As these VPC subnets are created on the default overlay transport zone, they are realized on both vsphere compute clusters – VxDC01-C01 and VxDC01-C02
Because we selected “Default Outbound NAT” option while creating the VPC, we should see a default SNAT rule configured to allow private subnets to access external networks (eg: infrastructure services).
Resource sharing to VPCs
Whenever a VPC is created, a default share is created in the parent project which contains the private & external IP blocks, T0 gateway and edge cluster which can be consumed by the VPC.
We can share custom resources to a VPC from:
- the default space by the Enterprise Admin : This will share to all VPCs in the specified project.
- the respective Project by the Project Admin : This will share either to all or to selected VPCs in the project
Let’s check out both options:
We will login as hari@vxplanet.int (Enterprise Admin) to the default space and will share an IP_Restricted security group to Project_AppSec and all the VPCs in that project. Project users and VPC users can optionally use this security group in DFW policies to allow or deny access to networks in this security group.
Note that this option will share the security group with all the VPCs in the project.
We will see the shared security groups under the source / destination fields in the distributed or gateway firewall options in the VPC.
Next, we will login as alec@vxplanet.int who is the Project Admin of Project_AppSec and share another security group with AppSec_OTS_VPC that has the client subnets to access the applications hosted in AppSec_OTS_VPC.
As earlier, we will see the shared security groups under the source / destination fields in the distributed or gateway firewall options in the VPC.
Onboarding workload VMs to VPCs
Now that we have subnets created in the VPCs, lets onboard few workload VMs to VPCs by attaching them to the respective subnets.
We have three VMs – OTS_App01, OTS_App02 and OTS_App03 and we will connect them to public, private and isolated subnets of AppSec_OTS_VPC respectively.
Let’s login as ben@vxplanet.int who is the VPC Admin for AppSec_OTS_VPC, and we see that the three VMs are owned by AppSec_OTS_VPC.
Now, we will login to one of the edge nodes hosting the T0 SR component and check the BGP table to confirm that the public subnets are advertised from the VPC T1 gateways (as we authorized them in the project route filter towards the beginning of this blog post)
Hope you have noticed that, what is being advertised by the Provider T0 gateway to the external fabrics over BGP is the carved out CIDRs (/32, /26 etc) from the external IP blocks, which would make the routing tables on the upstream devices look messy over the time.
Let’s summarize them at the Provider T0 gateway.
Summarizing VPC public subnets at Provider T0 Gateway
Let’s login to the default space as hari@vxplanet.int (Enterprise Admin) and perform route summarization for the public subnets, so that only the summary route is advertised to the external fabrics.
Now, on the external fabric (which is Mikrotik RouterOS in my homelab), we see only the summary route received.
Distributed firewall in VPCs
Whenever a VPC is created, the system creates a default security group for the VPC. This group has a naming format of “PROJECT-<Project_name>-VPC-<VPC_Name>-default”. This group represents the entire VPC and has all the subnets and VMs of the VPC. This default group is available within the VPC as well as on the parent project and the default space.
Remember that in the previous article (Part 2), we discussed about the default group of the project. All members of the VPC’s default group will be automatically added as members of it’s parent project’s default group. That means, if we write a DFW policy using the project’s default group, it will apply to all the VPCs in the project as well. Let’s check this:
We see the VPC subnets and segment ports added as members to the project’s default group.
As the default groups of both projects and VPCs are available to the default space as well, we could use these security groups to write DFW rules in the default space, to override project / VPC policies or to lockdown any specific project including it’s VPCs.
The system auto-creates the below DFW policies and rules within each VPC. With the default policies, workload VMs have outbound access from all subnets in the VPCs to “ANY” destination (that are within the VPC as well as outside the VPC, including external networks).
Inbound access to public subnets is not enabled, but we could write additional policies based on the requirements.
If you recall the default DFW behaviour of projects (which we discussed in Part 2), each workload VM in a project can only communicate with other workload VMs in the same project (unless additional rules are created).
Does that mean, with only the default rules in place, workload VMs in the public subnet of VPC require additional DFW rules at the parent project level for outbound external access? Not exactly, let’s find out:
The default DFW policy of the VPCs are stacked on top of the default policy of the parent project.
Rule ID 1054 (at the top) has more precedence than other rules, hence workload VMs on the VPC public subnets will have outbound external connectivity.
Any custom rules written in the project context will be stacked on top of all VPC default and custom rules.
As you can see rule ID 1092 written at the project context has more precedence than the Rule ID 1093 written at the VPC level.
Rule ID 1093 was written by the VPC Admin to allow access to the IP_Restricted group. Rule ID 1092 was written by the Project Admin to override VPC rule 1093 and thus deny access to the IP_Restricted group. Let’s confirm this by reviewing the rules applied to a workload VM attached to the ots_public subnet.
Rules written in the default space by the Enterprise Admin will ultimately have more precedence than anything written at the project or VPC level. That means an Enterprise Admin can lock down a project or a specific VPC from the default space, if required. Let’s check this:
We have a Rule ID 1094 in the default space that restricts the AppSec_OTS_VPC from accessing the IP_Restricted group.
As you can see, Rule ID 1094 (in the default space) has higher precedence over Rule ID 1092 (project level) and Rule ID 1093 (VPC level)
If you are still awake, we have reached the end of this article 😊.
We will meet again in Part 4 to discuss about stateful active-active gateways in Projects. Stay Tuned!!!
I hope this article was informative. Thanks for reading.
Continue reading? Here are the other parts of this series:
Part 1 – Introduction & Multitenancy models :
https://vxplanet.com/2023/10/24/nsx-multitenancy-part-1-introduction-multitenancy-models/
Part 2 – NSX Projects :
https://vxplanet.com/2023/10/24/nsx-multitenancy-part-2-nsx-projects/
Part 4 : Stateful Active-Active Gateways in Projects
https://vxplanet.com/2023/11/07/nsx-multitenancy-part-4-stateful-active-active-gateways-in-projects/
Part 5 : Edge Cluster Considerations and Failure Domains
https://vxplanet.com/2024/01/23/nsx-multitenancy-part-5-edge-cluster-considerations-and-failure-domains/
Part 6 : Integration with NSX Advanced Load balancer
https://vxplanet.com/2024/01/29/nsx-multitenancy-part-6-integration-with-nsx-advanced-load-balancer/
