When VMware Cloud Director is deployed in public environment setup it is a good practice to restrict the system admin access only for specific networks so no brute force attack can be triggered against the publicly available UI/API end points.

There is actually a relatively easy way to achieve this via any web application firewall (WAF) with URI access filter. The strategy is to protect only the provider authentication end points which is much easier than to try to distinguish between provider and tenant URIs.

As the access (attack) can be done either through UI or API the solution should address both. Let us first talk about the UI. The tenants and provider use specific URL to access their system/org context but we do not really need to care about this at all. The UI is actually using (public) APIs so there is nothing needed to harden the UI specifically if we harder the API endpoint. Well, the OAuth and SAML logins are exception so let me tackle them separately.

So how can you authenticate to VCD via API?

Integrated Authentication

The integrated basic authentication consisting of login/password is used for VCD local accounts and LDAP accounts. The system admin (provider context) uses /cloudapi/1.0.0/sessions/provider API endpoint while the tenants use /cloudapi/1.0.0/sessions.

The legacy (common for both providers and tenant) API endpoint /api/sessions has been deprecated since API version 33.0 (introduced in VCD 10.0). Note that deprecated does not mean removed and it is still available even with API version 36.x so you can expect to be around for some time as VCD keeps backward compatible APIs for few years.

You might notice that there is in a Feature Flags section the possibility to enable “Legacy Login Removal”.

Enabling this feature will disable legacy login both for tenants and providers however only if you use alpha API version (in the case of VCD 10.3.3.1 it is 37.0.0-alpha-1652216327). So this is really only useful for testing your own tooling where you can force the usage of that particular API version. The UI and any 3rd party tooling will still use the main (supported) API versions where the legacy endpoint will still work.

However, you can forcefully disable it for provider context for any API version with the following CMT command (run from any cell, no need to restart the services):

/opt/vmware/vcloud-director/bin/cell-management-tool manage-config -n vcloud.api.legacy.nonprovideronly -v true

The providers will need to use only the new cloudapi/1.0.0/providers/session endpoint. So be careful as it might break some legacy tools!

API Access Token Authentication

This is a fairly new method of authentication to VCD (introduced in version 10.3.1) that uses once generated secret token for API authentication. It is mainly used by automation or orchestration tools. The actual method of generating session token requires access to the tenant or provider oauth API endpoints:

/oauth/tenant/<tenant_name>/token

/oauth/provider/token

This makes it easy to disable provider context via URI filter.

SAML/OAuth Authentication via UI

Here we must distinguish the API and UI behavior. For SAML, the UI is using /login/org/<org-name>/… endpoint. The provider context is using the default SYSTEM org as the org name. So we must filter URI starting with /login/org/SYSTEM.

For OAuth the UI is using the same endpoint as API access token authentication /oauth/tenant vs /oauth/provider. /login/oauth?service=provider

For API SAML/OAuth logins cloudapi/1.0.0/sessions vs cloudapi/1.0.0/sessions/provider endpoints are used.

WAF Filtering Example

Here is an example how to set up URI filtering with VMware NSX Advanced Load Balancer.

1. We need to obviously set up VCD cell (SSL) pool and Virtual Service for the external IP and port 443 (SSL).
2. The virtual service application profile must be set to System-Secure-HTTP as we need to terminate SSL sessions on the load balancer in order to inspect the URI. That means the public SSL certificate must be uploaded to load balancer as well. The cells can actually use self signed certs especially if you use the new console proxy that does not require SSL pass through and works on port 443.
3. In the virtual service go to Policies > HTTP Request and create following rules:
   Rule Name: Provider Access
   Client IP Address: Is Not: <admin subnets>
   Path: Criteria – Begins with:
   /cloudapi/1.0.0/sessions/provider
   /oauth/provider
   /login/oauth?service=provider
   /login/org/SYSTEM
   Content Switch: Local response – Status Code: 403.
   

And this is what you can observe when trying to log in via integrated authentication from non-authorized subnets:

And here is an example of SAML login:

This came up on provider Slack and I want to preserve it for future.

Question: Can the provider create Flex type Org VDC where the tenant would get allocation of certain CPU capacity (in GHz) and they could deploy as many VMs as they wish while limited by the CPU allocation?

Example: the VDC has 10 GHz allocation and the tenant is able to deploy 3 VMs each with 2 vCPUs at 2 GHz (which is in total 12 GHz) but they will altogether be able to consume only up to 10 GHz of physical vSphere resources.

This is basically the old legacy non-elastic allocation pool type VDC where the VDC is backed by single vCenter Server Resource Pool with the limit equal to the CPU allocation. As of VCD 5.1.2 there is ability to enable elasticity of such VDC but it behaves differently and is done at whole VCD instance level. So can you create similarly behaving VDC with the new Flex type (introduced in VCD 10.0).

Answer: Yes. Create non-elastic Flex VDC with the allocation and enable CPU limit. Then create VM sizing policy with vCPU speed set to 0 GHz and make it default for the Org VDC.

Topic of multitenant logging in VMware Cloud Director (VCD) especially with NSX-T came up recently in multiple conversations, so I will summarize the capabilities and options as they are today.

Audit and Event Logs

Information about VCD events such as user logins, vApp operation, VM power-on etc are available in the Audit and Event log. Tenant can access their particular logs via API – AuditTrail Events and Tasks (type=task query) or directly in the UI.

An additional option is to receive events and notifications via MQTT message bus as was described here.

NSX-V Networking Logs

With NSX-V each tenant would have one or more Org VDC Gateways each backed by its own NSX-V Edge Service Gateway (single or two VMs in HA mode). The tenant could configure their own syslog endpoint to send all Edge logs there. As each Edge could have two syslog entries, the provider could set up their syslog endpoint as well to receive the same logs.

And this is how the configuration looks in NSX-V:

Distributed firewall logs are generated in vmkernel of each ESXi host. It means they are sent to syslog configured at ESXi level which is not multitenant. However VCD configures a rule tag which is unique for each Org VDC (its UUID) so it is possible to filter logs for a specific tenant.

NSX-T Networking Logs

Org VDC gateways in NSX-T Org VDCs are backed by NSX-T Tier-1 gateways which are logical entities on shared NSX-T Edge Nodes. This means it is not possible to configure tenant’s syslog on them as was done with NSX-V. Therefore VCD will configure Log label for each GW firewall entry. The log is composit of first 10 characters of the UUID of the GW parent object (Org VDC or Data Center Group), 5 characters of the GW name and 5 characters of the GW UUID.

Note that you need at least NSX-T 3.2 and VCD 10.4.

For distributed firewall the situation is identical to NSX-V. The only difference it that the log label is composit of first 5 characters of the Data Center Group name and 27 characters of its UUID.

Additional improvements are planned in NSX-T 4.1+ to allow multitenant access to other log types.

Log Filtering and Forwarding

To test the functionality vRealize Log Insight supports up to 10 log forwarders. These can be configured in the Log Management section. In the screenshot below you can see Distributed Firewall for NSX-V and NSX-T and Gateway Firewall for NSX-T forwarding to tenant syslog 10.0.2.29 configured.

It is common for service providers to meter internet traffic consumption of their tenants. In NSX-V environment you could get any interface statistics of an Edge Service Gateway via NSX API or vCloud Director Networking API:
GET /api/4.0/edges/{edgeId}/statistics/interfaces

Let us explore the ways you can do similar metering in NSX-T environment.

We have three points where we can meter external traffic:

1. On Tier-0 or VRF uplink (if such object is dedicated to the tenant). The API to use is:
   
   GET https://{{host}}/policy/api/v1/infra/tier-0s/{tier-0 name}/locale-services/default/interfaces/{uplink-name}/statistics
   
   If the Tier-0/VRF is stretched across multiple Edge nodes you will need to aggregate the data across multiple uplinks.
   
2. On Tier-0/VRF router downlink to a specific Tier-1 GW. This is useful if the Tier-0 is shared across multiple tenants and their Tier-1 GWs. The API is not documented in the API docs.
   
   GET https://{{host}}/policy/api/v1/infra/tier-0s/{tier-0 name}/tier-1-interface/statistics/summary?tier1_path=/infra/tier-1s/{tier-1 name}
   
3. On CSP (Service Interface) of a Tier-1 (or Tier-0) GW for use cases where a particular VLAN is connected directly to tenant GW. The API to use is quite long and must include the enforcement point (edge node backing active Tier-1 GW)
   
   GET https://{{host}}/policy/api/v1/infra/tier-1s/{tier-1 name}/locale-services/{locale id}/interfaces/{CSP interface id}/statistics?enforcement_point_path=/infra/sites/default/enforcement-points/default&edge_path=/infra/sites/default/enforcement-points/default/edge-clusters/{edge-cluster-id}/edge-nodes/{edge-node-id}

VMware has just released VMware Cloud Provider Lifecycle Manager version 1.4. I have blogged about the product in the past. I have been testing the version 1.4 in my lab and must say it is now significantly easier to use to deploy, manage new or brown field deployments of the VMware Cloud Director stack.

The UI has now almost full parity with the API. So it is now very easy to use the product from the UI for single environment management (at scale you would probably still want to use API). Additionally, existing VCD environments (either deployed buy VCPLCM or any other way) can be registered under management of VCPLCM. You will just need to provide a few basic information (solution endpoint address, passwords, etc.) which allows VCPLCM to run the discovery process and collect additional details (nodes, certificates, integrations, etc.). Once that is done you simply finish the registration by supplying missing details such as integration passwords or node names and their network port groups in VC inventory that could not be autodiscovered.

With VCD under management of VCPLCM the most annoying operations such as upgrades, certificate replacements or node additions/redeployments are very simple to perform and take just a few minutes.

Give it a try!

Starting with VMware Cloud Director 10.4.1 VM and vApp operations that happen across Provider VDCs backed by different vCenter Server instances require that both vCenter Server trust each other certificates.

Which operations are affected?

Any VM move, clone or vApp move, clone and template deployments that happen via UI or API where source and destinations are managed by different vCenter Server.

If involved vCenter Server do not trust each other certificates, those operations will fail with an error:

Underlying system error: com.vmware.vim.binding.vim.fault.SSLVerifyFault

In the VCD cell logs you will see something similar to:

2022-11-18 17:15:46,023 | ERROR    | vim-proxy-activity-pool-482 | RelocateVmActivity             | Underlying system error: com.vmware.vim.binding.vim.fault.SSLVerifyFault | requestId=9554d239-a457-48bd-988d-4aa7856cdba2,request=POST https://10.196.235.31/api/vdc/ceb9bd94-171e-4d3f-a390-248791cbb218/action/moveVApp,requestTime=1668791732601,remoteAddress=10.30.65.110:53169,userAgent=PostmanRuntime/7.29.2,accept=application/*+xml;version 37.0 vcd=b4856f34-94bb-4462-8d1b-469483f98f69,task=46d10139-134e-4823-a674-dd0e10a5565a activity=(com.vmware.vcloud.backendbase.management.system.TaskActivity,urn:uuid:46d10139-134e-4823-a674-dd0e10a5565a) activity=(com.vmware.vcloud.vdc.impl.MoveVAppActivity,urn:uuid:a8c5d945-9ffc-44b7-aafb-54a634e0dd08) activity=(com.vmware.vcloud.vdc.impl.LinkVMsToTargetVAppActivity,urn:uuid:ed3c8d9f-69db-466e-bbbd-9046d996d3e6) activity=(com.vmware.vcloud.vdc.impl.MoveVmUsingVmotionActivity,urn:uuid:6839cde5-cff4-424f-9d24-940b0b78ef45) activity=(com.vmware.ssdc.backend.services.impl.RelocateVmActivity,urn:uuid:dd9d8304-c6e8-45eb-8ef0-65a17b765785) activity=(com.vmware.vcloud.fabric.storage.storedVm.impl.RelocateStoredVmByStorageClassActivity,urn:uuid:ff420cab-5635-45a0-bdf0-fee6fe33eb7e) activity=(com.vmware.vcloud.fabric.storage.storedVm.impl.RelocateStoredVmByDatastoreActivity,urn:uuid:6cde303d-ae82-4a3c-9ecc-cceb0efcdbe9) activity=(com.vmware.vcloud.val.internal.impl.RelocateVmActivity,urn:uuid:73b44746-52c3-4143-bb29-1d22b532cc55)
com.vmware.ssdc.library.exceptions.GenericVimFaultException: Underlying system error: com.vmware.vim.binding.vim.fault.SSLVerifyFault
        at com.vmware.ssdc.library.vim.LmVim.createGenericVimFaultException(LmVim.java:329)
        at com.vmware.ssdc.library.vim.LmVim.Convert(LmVim.java:445)
        at com.vmware.ssdc.library.vim.LmVim.Convert(LmVim.java:499)
        at com.vmware.vcloud.val.taskmanagement.AsynchronousTaskWaitActivity.getResultIfTaskAlreadyCompleted(AsynchronousTaskWaitActivity.java:449)
        at com.vmware.vcloud.val.taskmanagement.AsynchronousTaskWaitActivity$InitialPhase.invoke(AsynchronousTaskWaitActivity.java:123)
        at com.vmware.vcloud.activity.executors.ActivityRunner.runPhase(ActivityRunner.java:175)
        at com.vmware.vcloud.activity.executors.ActivityRunner.run(ActivityRunner.java:112)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: (vim.fault.SSLVerifyFault) {
   faultCause = null,
   faultMessage = null,
   selfSigned = false,
   thumbprint = B3:E8:35:77:E3:65:20:1A:BD:F2:DB:41:08:34:58:51:07:E2:AA:53
}

The system administrator thus must make sure that when there is more than one resource vCenter Server managed by VMware Cloud Director that there is mutual trust across all such vCenter Server instances. To reiterate, this must be ensured at vCenter Server level – no change is needed in VMware Cloud Director.

Typically, vCenter Servers use their own VMCA to issue their (VC and ESXi) certificates. Each such VMCA certificate must be imported to the other vCenter Server. The alternative is to use publicly signed CA instead of the self-signed VMCA. Then no action would be needed. And finally if you use your own Enterprise CA to issue VC certificates, then each VC just must trust the root Enterprise CA.

vCenter Server provides UI for certificate management in vSphere Client (Administration > Certificates > Certificate Management).

The source VMCA certificate can be retrieved via the download.zip from http://<source-VC>/certs/download.zip, extracted and uploaded via the ablove ADD UI.

Alternatively, the same can be accomplished via API from Developer Center > API Explorer by GET chain from source VC and upload it by POST trusted-root-chains to the target VC.

The third alternative is to use PowerCLI which also provides a way to manage vSphere certificates. An example script would look like this:

$sourceVC="vc01.fojta.com"
$targetVC="vc02.fojta.com"
connect-viserver $sourceVC
connect-viserver $targetVC
$trustedCerts = Get-VITrustedCertificate -server $sourceVC -VCenterOnly
foreach ($Cert in $trustedCerts)
  {Add-VITrustedCertificate -PemCertificateOrChain $cert.CertificatePEM -server $sourceVC -VCenterOnly -Confirm:$false}

As the trust must be mutual, if you have five resource vCenter Server each should have at least five trusted root certificates (one its own VMCA and four for the other VCs).

VMware Cloud Director 10.4.1 (VCD) has just been released and with it comes also the change of the appliance internal PostgreSQL database from version 10 to 14 (as well as replication manager upgrade). This means that the standard upgrade process has changed and is now a bit more complicated and could be significantly longer than in the past. So pay attention to this change. This article goal is to explain what is going on in the background to understand better why certain steps are necessary.

VCD 10.4.1 is still interoperable with PostgreSQL 10-14 if you use the Linux deployment format, so this blog applies only if you use the appliance deployment. PostgreSQL version 10 is no longer maintained so that is the main reason for the switch besides the fact that newer is alway better :-).

• The database upgrade will happen during the vamicli update process.
• All appliance nodes must be up but the vmware-vcd service should be shut down.
• Always recommended is to take cold snaphots of all nodes before the upgrade (make sure the snapshots are done while all DB nodes are off at the same time as you do not want to restore snapshot of primary to older state while secondary nodes are ahead)
• Primary database appliance node is where you have to start the upgrade process, it will most likely take the longest time as new database version 14 will be installed side-by-side and all the data converted. That means you will also need enough free space on the database partition. You can check by running on the primary DB node:
  
  df -h|grep postgres
  /dev/mapper/database_vg-vpostgres 79G 17G 58G 23% /var/vmware/vpostgres
  
  The above shows that the partition size is 79 GB, the DB is currently using 17 GB and I have 53 GB free. So I am good to go. The actual additional needed space is less than 17 GB as the database logs and write ahead logs are not copied over. Those can be deducted. You can quickly get their size by running:
  
  du -B G –summarize /var/vmware/vpostgres/current/pgdata/pg_wal/
  du -B G –summarize /var/vmware/vpostgres/current/pgdata/log/
  
  Or just use this one-liner:
  
  du -sh /var/vmware/vpostgres/current/pgdata/ –exclude=log –exclude=pg_wal
  
  If needed, the DB partition can be easily expanded.
  
• The secondary DB nodes will during the vami upgrade process clone the upgraded DB from the primary via replication. So these nodes just drop the current DB and will not need any additional space.
• After all nodes (DB and regular ones) are upgraded, the database schema is upgraded via the /opt/vmware/vcloud-director/bin/upgrade command. Then you must reboot all nodes (cells).
• The vcloud database password will be changed to a new autogenerated 14 character string and will be replicated to all nodes of VCD cluster. If you use your own tooling to access the DB directly you might want to change the password as there is no way of retrieve the autogenerated one. This must be done by running psql in the elevated postgres account context.
  
  root@vcloud1 [ /tmp ]# su postgres
  postgres@vcloud1 [ /root ]$ psql -c “ALTER ROLE vcloud WITH PASSWORD ‘VMware12345678′”
  
  
  and then you must update vcd-service on each cell via the CMT reconfigure-database command and restarting the service.
  
  /opt/vmware/vcloud-director/bin/cell-management-tool reconfigure-database -dbpassword ‘VMware12345678’ -dbuser vcloud
  
  Note the DB password must have at least 14 characters.
• Any advanced PostgreSQL configuration options will not be retained. In fact they may be incompatible with PostgreSQL 14..

The recently released new version of VMware Cloud Director 10.4.1 brings quite a lot of new features. In this article I want to focus on those related to networking.

External Networks on Org VDC Gateway (NSX-T Tier-1 GW)

External networks that are NSX-T Segment backed (VLAN or overlay) can now be connected directly to Org VDC Edge Gateway and not routed through Tier-0 or VRF the Org VDC GW is connected to. This connection is done via the service interface (aka Centralize Service Port – CSP) on the service node of the Tier-1 GW that is backing the Org VDC Edge GW. The Org VDC Edge GW still needs a parent Tier-0/VRF (now called Provider Gateway) although it can be disconnected from it.

What are some of the use cases for such direct connection of the external network?

• routed connectivity via dedicated VLAN to tenant’s co-location physical servers
• transit connectivity across multiple Org VDC Edge Gateways to route between different Org VDCs
• service networks
• MPLS connectivity to direct connect while internet is accessible via shared provider gateway

The connection is configured by the system administrator. It can use only single subnet from which multiple IPs can be allocated to the Org VDC Edge GW. One is configured directly on the interface while the others are just routed through when used for NAT or other services.

If the external network is backed by VLAN segment, it can be connected to only one Org VDC Edge GW. This is due to a NSX-T limitation that a particular edge node can use VLAN segment only for single logical router instantiated on the edge node – and as VCD does not give you the ability to select particular edge nodes for instantiation of Org VDC Edge Tier-1 GWs it simply will not allow you to connect such external network to multiple Edge GWs. If you are sharing the Edge Cluster also with Tier-0 GWs make sure that they do not use the same VLAN for their uplinks too. Typically you would use VLAN for co-location or MPLS direct connect use case.

Overlay (Geneve) backed external network has no such limitations which makes it a great use case for connectivity across multiple GWs for transits or service networks.

NSX-T Tier-1 GW does not provide any dynamic routing capabilities, so routing to such network can be configured only via static routes. Also note that Tier-1 GW has default route (0.0.0.0/0) always pointing towards its parent Tier-0/VRF GW. Therefore if you want to set default route to the segment backed external network you need to use two more specific routes. For example:

0.0.0.0/1 next hop <MPLS router IP> scope <external network>
128.0.0.0/1 next hop <MPLS router IP> scope <external network>

Slightly related to this feature is the ability to scope gateway firewall rules to a particular interface. This is done via Applied To field:

You can select any CSP interface on the Tier-1 GW (used by the external network or non-distributed Org VDC networks) or nothing, which means the rule will be applied to uplink to Tier-0/VRF as well as to any CSP interface.

IP Spaces

IP Spaces is a big feature that will be delivered across multiple releases, where 10.4.1 is the first one. In short IP Spaces are new VCD object that allows managing individual IPs (floating IPs) and subnets (prefixes) independently across multiple networks/gateways. The main goal is to simplify the management of public IPs or prefixes for the provider as well as private subnets for the tenants, however additional benefits are being able to route on shared Provider Gateway (Tier-0/VRF), use same dedicated parent Tier-0/VRF for multiple Org VDC Edge GWs or the ability to re-connect Org VDC Edge GWs to a differeent parent Provider Gateway.

Use cases for IP Spaces:

• Self-service for request of IPs/prefixes
• routed public subnet for Org VDC network on shared Provider Gateway (DMZ use case, where VMs get public IPs with no NATing performed)
• IPv6 routed networks on a shared Provider Gateway
• tenant dedicated Provider Gateway used by multiple tenant Org VDC Edge Gateways
• simplified management of public IP addresses across multiple Provider Gateways (shared or dedicated) all connected to the same network (internet)

In the legacy way the system administrator would create subnets at the external network / provider gateway level and then add static IP pools from those subnet for VCD to use. IPs from those IP pools would be allocated to tenant Org VDC Edge Gateways. The IP Spaces mechanism creates standalone IP Spaces (which are collections of IP ranges (e.g. 192.168.111.128-192.168.111.255) and blocks of IP Prefixes(2 blocks of 192.168.111.0/28 – 192.168.111.0/28, 192.168.111.16/28 and 1 block of 192.168.111.32/27).

A particular IP Space is then assigned to a Provider Gateway (NSX-T Tier-0 or VRF GW) as IP Space Uplink:

An IP Space can be used by multiple Provider Gateways.

The tenant Org VDC Edge Gateway connected to such IP Space enabled provider gateway can then request floating IPs (from IP ranges) 

or assign IP block to routable Org VDC network which results into route advertisement for such network.

In the above case such network should be also advertised to the internet, the parent Tier-0/VRF needs to have route advertisement manually configured (see below IP-SPACES-ROUTING policy) as VCD will not do so (contrary to NAT/LB/VPN case).

Note: The IP / IP Block assignments are done from the tenant UI. Tenant needs to have new rights added to see those features in the UI.

The number of IPs or prefixes the tenant can request is managed via quota system at the IP Space level. Note that the system administrator can always exceed the quota when doing the request on behalf of the tenant.

A provider gateway must be designated to be used for IP Spaces. So you cannot combine the legacy and IP Spaces method of managing IPs on the same provider gatway. There is currently no way of converting legacy provider gateway to the one IP Spaces enabled, but such functionality is planned for the future.

Tenant can create their own private IP Spaces which are used for their Org VDC Edge GWs (and routed Org VDC networks) implicitly enabled for IP Spaces. This simplifies the creation of new Org VDC networks where a unique prefix is automatically requested from the private IP Space. The uniqueness is important as it allows multiple Edge GWs to share same parent provider gateway.

Transparent Load Balancing

VMware Cloud Director 10.4.1 adds support for Avi (NSX ALB) Transparent Load Balancing. This allows the pool member to see the source IP of the client which might be needed for certain applications.

The actual implementation in NSX-T and Avi is fairly complicated and described in detail here: https://avinetworks.com/docs/latest/preserve-client-ip-nsxt-overlay/. This is due to the Avi service engine data path and the need to preserve the client source IP while routing the pool member return traffic back to the service engine node.

VCD will hide most of the complexity so that to actually enable the service three steps need to be taken:

1. Transparent mode must be enabled at the Org VDC GW (Tier-1) level.

2. The pool members must be created via NSX-T Security Group (IP Set).

3. Preserve client IP must be configured on the virtual service.

Due to the data path implementation there are quite many restrictions when using the feature:

• Avi version must be at least 21.1.4
• the service engine group must be in A/S mode (legacy HA)
• the LB service engine subnet must have at least 8 IPs available (/28 subnet – 2 for engines, 5 floating IPs and 1 GW) – see Floating IP explanation below
• only in-line topology is supported. It means a client that is accessing the LB VIP and not going via T0 > T1 will not be able to access the VS
• only IPv4 is supported
• the pool members cannot be accessed via DNAT on the same port as the VS port
• transparent LB should not be combined with non-transparent LB on the same GW as it will cause health monitor to fail for non-transparent LB
• pool member NSX-T security group should not be reused for other VS on the same port

Floating IPs

When transparent load balancing is enabled the return traffic from the pool member cannot be sent directly to the client (source IP) but must go back to the service engine otherwise asymmetric routing happens and the traffic flows will be broken. This is implemented in NSX-T via N/S Service Insertion policy where the pool member (defined via security group) traffic is instead to its default GW redirected to the active engine with a floating IP. Floating IPs are from the service engine network subnet but are not from the DHCP range which assigns service engine nodes their primary IP. VCD will dedicate 5 IP from the LB Service network range for floating IPs. Note that multiple transparent VIPs on the same SEG/service network will share floating IP.

A brief heads up note that with the release of VMware Cloud Director 10.5 and its corresponding API version 38.0 the legacy API endpoint POST /api/sessions has been removed.

It has been announced for deprecation for some time (actually it was initially planned to be removed in VCD 10.4) and I blogged in detail about the replacement /cloudapi/1.0.0/sessions and /cloudapi/1.0.0/sessions/provider endpoints in the Control System Admin Access blog post.

Any 3rd party tools that use the legacy endpoint will get HTTP response 405 – Method not found.

Note that the GET /api/session method will still work until all legacy APIs will be moved to /cloudapi.

Introduction

Warning: Alpha/experimental feature content follows

If you have installed VMware Cloud Director 10.5 you might have noticed a new disabled Feature Flag functionality called Three Personas. This is a preview of new capability that will be iteratively added to VCD in upcoming releases. However you can already use some of it today after enabling the feature flag.

This new feature creates ability to create sub-provider like organizations that have the right to manage other organizations while not having the full VCD system admin rights. In this first iteration the sub-provider cannot create such organizations or add them any resources, but can operate within their context on their behalf. While this is just scratching the surface of things to come, this already covers some use cases that were requested by many service providers. Namely the ability to enpower provider support team to manage certain tenant level operations in a subset of organizations without the ability to access the infrastructure related objects that only system admins should manage (PVDCs, Provider Gateways, VCs, …)

How to Get Started

After you enable the feature flag you will notice a new Default Sub-Provider Entitlement right bundle is created with new right Access Control: Organization: Administer and traverse into other organizations.

We need to create new global role “Sub-Provider Administrator” that we will clone from the Global Organization Administrator and add this new additional right to it.

Now we can create the Sub-Provider organization. We use the regular New Organization UI workflow. However notice we can see new Enable sub-provider option. When selected the Default Sub-Provider right bundle is automatically assigned to this organization.

We will publish the new Sub-provider Administrator global role to this organization.

Now you can create some users in this Sub-provider organization and assign them the new global role.

Now we can log in into the Sub-provider organization and can see that in the Administration > Organizations we see only our own organization.

This is because we have not yet been entitled to see other specific organizations. This step currently in VCD 10.5 requires API usage.

While logged in as system admin with the API version 39.0.0-alpha run the GET Org OpenAPI call on the organization you want to manage (ACME Org. in my case) as the sub-provider. Notice the new managedBy section in the output JSON.

By default every org is managed by System org. We will change it to our Sub-provider org and use PUT on the same Org endpoint.

You can immediately see in the Sub-provider Org UI that the ACME Org is accessible and we can traverse to it.

ACME Org in the sub-provider context.

Any tasks performed or objects created in the ACME Org within the sub-provider context will by marked/owned by SYSTEM.

VMware Cloud Director relies on NSX Advanced Load Balancer (Avi) integration to offer load balancing as a service to tenants – more on that here. This article discusses a particular workaround to scale above the current NSX-ALB NSX-T Cloud limits which are 128 Tier-1 objects per NSX-T Cloud and 300 Tier-1 objects per Avi Controller cluster. As you can have up to 1000 Tier-1 objects in single NSX-T instance could we have load balancing service on each of them?

Yes, but let’s first recap the integration facts.

• VCD integrates with NSX-T via registration of NSX-T Managers
• The next step is creation of VCD Network Pool by mapping it to Geneve Transport Zone provided by the above NSX-T Managers
• Individual Org VDCs consume NSX-T resources for network creation based on assigned VCD Network Pool
• VCD integrates with NSX ALB via registration of Avi Controller Cluster
• NSX ALB integrates with NSX-T via NSX-T Cloud construct that is defined by NSX-T Manager, Overlay Transport Zone and vCenter Server
• NSX-T Cloud is imported into VCD and matched with an existing Network Pool based on the same NSX-T Manager/Overlay Transport Zone combination
• Service Engine Groups are imported from available NSX-T Clouds to be used as templates for resources to run tenant consumed LB services and are assigned to Edge Gateways

Until VCD version 10.4.2 you could map only single NSX-T Cloud to a Geneve Network Pool. However with VCD 10.5 and 10.4.2.2 and newer you can map multiple NSX-T Clouds to the same Geneve Network Pool (even coming from different Avi Controller clusters). This essentially allows you to have more than 128 Tier-1 load balancing enabled GWs per such Network Pool and with multiple NSX ALB instances could scale all the way to 1000 Tier-1 GWs.

The issue is that VCD currently is not smart enough to pick the most suitable NSX-T Cloud for placement from capacity perspective. The only logic VCD is using is the priorization based on the alphabetic ordering of NSX-T Clouds in the list. So it is up to the service provider to make sure that on top is the NSX-T Cloud with the most available capacity.

As can be seen above I used A- and B- prefixed names to change the prioritization. Note that the UI does not allow the name edit, API PUT call must be used instead.

Note: The assignable Service Engine Groups depend if the Edge Gateway (Tier-1) has already been assigned to a particular NSX-T Cloud or not. So the API endpoint will reflect that:
https://{{host}}/cloudapi/1.0.0/loadBalancer/serviceEngineGroups?filter=(_context==<gateway URN>;_context==assignable)

Lastly I want to state that the above is considered a workaround and a better capacity management will be address in the future VCD releases.

VMware NSX Migration for VMware Cloud Director is a tool that automates migration of NSX-V backed Org VDCs to NSX-T backed provider VDC (PVDC). It is a side-by-side migration that requires standing up a new NSX-T cluster(s) to accommodate the migration. I described the process in my older blog.

However, there might be a situation that prevents this approach – either because hardware is not available or because only one cluster is available (VCD on top of VMware Cloud Foundation management domain).

We have derived and tested a procedure that enables in-place migration for such corner cases. It is not simple and requires the concurrent usage of NSX Migration Coordinator and NSX Migration for VCD. It also has some limitations but it can be an option when the regular side-by-site migration is not possible.

High Level Overview

At a high level the procedure is from VCD perspective still side by side and uses NSX Migration for VCD, but both source PVDC and target PVDC reside on the same cluster (with resource pool backing) while NSX Migration Coordinator is converting individual hosts from NSX-V to NSX-T and temporarily bridging VXLAN and Geneve networks.

Here are the high level steps:

1. Make sure the source PVDC (NSX-V backed) has vSphere resource pool backing (instead of whole cluster). Also make sure VCD version is at least 10.4.2.2.
2. Partially prepare target NSX-T Environment (deploy Managers and Edge Nodes)
3. If load balancing is used, also partially prepare NSX ALB (deploy Controllers and configure NSX-T Cloud and Service Engine Groups)
4. Create target PVDC NSX-T with resource pool backing on the same cluster
5. Migrate network topology of all Org VDCs using the cluster with the NSX Migration for VCD (while using identical VNIs for the Geneve backed networks)
6. Convert cluster ESXi hosts from NSX-V to T and migrate workloads to Geneve backed networks with NSX Migration Coordinator
7. Migrate Org VDC network services with NSX Migration for VCD
8. Migrate vApps with NSX Migration for VCD
9. Clean up source Org VDCs with NSX Migration for VCD
10. Clean up NSX-V from VCD and vCenter

Detailed Migration Steps

PVDC backing with Resource Pools

The source PVDC must be backed by resource pool (RP) otherwise we will not be able to create another PVDC on the same cluster. If the PVDC is using the whole cluster (its root RP) then it must be converted. The conversion is not so difficult but requires DB edit, therefore GSS support is recommended:

1. Create a child RP under the vSphere cluster to be used as a PVDC RP.
2. Move all PVDC child RPs (that were under the cluster directly, including System vDC RP) under this new RP.
3. Note down the VC moref of this new RP.
4. Update VCD vc_computehub and prov_vdc tables in VCD database to point the rp_moref to this new RP moref from VC (step 3).

If necessary, upgrade VCD to version 10.4.2.2 or newer.

Prepare NSX-T Environment

1. Deploy NSX-T Managers
2. Deploy NSX-T Edge Nodes. Note that the Edge node transport zone underlay (transport) VLAN must be routable with the existing VXLAN underlay (transport) VLAN (or it must be the same VLAN). The ESXi hosts when converted by NSX Migration Coordinator will reuse the VXLAN transport VLAN for its TEP underlay.
3. Make sure that Geneve overlay ID VNI Pool interval contains the VXLAN VNIs used in NSX-V

4. Create overlay transport zone
5. Register NSX-T manager to VCD
6. Register VC as Compute Manager in NSX-T Manager
7. Create Geneve backed network pool in VCD (utilizing transport zone from step #4)
8. Create Tier-0 and VRF gateways as necessary based on your desired target topology
9. Create provider gateways in VCD using Tier-0 or VRF GWs created in step #8

Prepare NSX ALB Environment (optional)

This step is necessary only if load balancing is used.

1. Deploy Avi Controllers and import the Avi Controller Cluster into VCD
2. Prepare management segment and Tier-1 GW for Avi NSX-T Cloud
3. Set up NSX-T Cloud in Avi and import it into VCD. Note that the Avi Controller will indicate the NSX-T Cloud is in red state as there are no NSX-T backed ESXi hosts in the environment yet.
4. Set up Service Engine Groups as necessary and import them into VCD

Create Target Provider VDC

1. In vCenter create sibling resource pool next to the RP backing the source PVDC

2. In VCD create the target NSX-T backed PVDC backed by the RP created in step #1 while using the Geneve backed network pool created previously. Yes it is possible this way to create NSX-T backed PVDC on NSX-V prepared cluster.

Topology Migration

With NSX Migration for VCD perform prechecks for all Org VDCs using the cluster. Skip bridging assessment as it is not necessary.

vcdNSXMigrator.exe –filepath=sampleuserinput.yml  –passwordFile=passfile –preCheck -s bridging

If there are no issues, process with creation of target Org VDC network topology again for all Org VDCs on the cluster. In the input YAML file include CloneOverlayIds=TRUE line which will ensure that target networks use the same Overlay IDs as the source networks. This will allow Migration Coordinator associate the right networks.

If the topology includes direct VLAN backed Org VDC networks then also use LegacyDirectNetworks=TRUE. This statement will force NSX Migration for VCD to reuse the existing DVS portgroup backed external networks in the target Org VDCs. You also need to make sure that each such parent external network is used more than just one Org VDC network (for example via a dummy temporary Org VDC), otherwise the migration would be using the “Imported NSX-T Segment” mechanism which would not work as there is no ESXi host available yet that could access NSX-T Segment. See below table from the official doc:

The topology migration is triggered with the following command (must be run for each Org VDC):

vcdNSXMigrator.exe –filepath=sampleuserinput.yml  –passwordFile=passfile -e topology

At this stage workloads are still running on NSX-V prepared hosts and are connected to VXLAN/VLAN backed port groups. The target Org VDCs are created and their network topology (Edge GWs and networks) exists only on Edge Transport Nodes as none of the resource ESXi hosts are yet prepared for NSX-T.

Note that you should not use Edge subinterface connection on source Org VDCs networks as it will not work later with Migration Coordinator. Also note that distributed routed networks on source side will experience extended downtime during migration as Migration Coordinator bridging will not work for them. You can convert them to direct interface Edge connectivity or alternatively accept the downtime.

Routed vApps (that use vApp Edge) should be powered off (undeployed). This is because vApp networks are not migrated during topology migration and thus will not exist during the Migration Coordinator step.

Migration Coordinator

In this step we will convert NSX-V hosts to NSX-T. This is done with NSX Migration Coordinator which will sequentially put each ESXi host into maintenance mode, evacuate the VMs (and connect them to VXLAN or Geneve networks) to other hosts. Then it will uninstall NSX-V and install NSX-T vibs on the host and bridge VXLAN and Geneve overlays directly in VMkernel.

When done, from vCenter perspective workloads will end up on the target networks but from VCD perspective they are still running in the source Org VDCs! The routing and other network services will still be performed by source Edge Gateways. In order to later migrate network services to NSX-T Tier-1 GWs, we must make sure that source Edge Service Gateway VMs stay on NSX-V prepared host so they are still manageable by NSX-V Manager (which uses ESXi host VIX/message bus communication). This is accomplished by creating vCenter DRS affinity groups.

1. Create DRS host group named EdgeHost that contains the last host in the cluster
2. Create DRS host group named WorkloadHosts that contains all other hosts in the cluster
3. Create DRS VM group named EdgeVMs that contains all ESG VMs
4. Create DRS VM group named WorkloadVMs that containts all VCD managed workloads
5. Create DRS VM/Host must rule for all EdgeVMs to run on EdgeHost
6. Create DRS VM/Host must rule for all WorkloadVMs to run on Workload hosts
7. Make sure that DRS vMotions all VMs as required (if you are using ESGs in HA configuration disable antiaffinity on them).

Now we can start the Migration Coordinator conversion:

8. Enable Migration Coordinator on one of the NSX-T Manager Appliances via CLI command start service migration-coordinator

9. Access the NSX-T appliance via web UI and go to System > Lifecycle Management > Migrate and select the Advanced Migration mode DFW, Host and Workload.

10. Import the source NSX-V configuration by connecting the vCenter and providing NSX-V credentials:

11. Continue to resolving configuration issues. Accept the recommendations provided by the Migration Coordinator.
    Note: The Migration Coordinator does not recognize the topology created by VCD hence there may be unknown objects like security groups in case DFW is used in VCD. We accept all the recommendations by the Migration Coordinator which is to skip the DFW rules migration. Not to worry, DFW will be migrated by NSX Migration for VCD in a later step.
12. Next we migrate configuration:
    

13. Prepare infrastructure in the next step:

14. And finally we arrive at Migrate Host step where the conversion of each ESXi host is performed. Make sure the ESG designated host is last in the sequence. Note that there is no rollback from this step on. So make sure you have tested this process in lab environment and you know what you are doing.

Either use Pause in between host option or let the migration run until the last host conversion. We will pause there (or cancel the VC action that tries unsuccessfully to put the ESG designated host into maintenance mode as there are DRS pinned VMs on it.

Network Services Migration and VCD vApp Reconciliation

Next, we need to migrate network services from NSX-V ESGs to Tier-1 GWs, migrate DFW and finally from VCD perspective migrate vApps to target Org VDCs.

For each Org VDC perform network services migration:

vcdNSXMigrator.exe  –filepath=sampleuserinput.yml  –passwordFile=passfile  -e services

N/S network disconnection will be performed. NSX-V Edge Gateways will be disconnected from external networks. Source Org VDC networks will be disconnected from their NSX-V Edge Gateways and target Org VDC networks will be connected to NSX-T Tier-1 GWs and Tier-1 GWs will be connected to Tier-0 gateway. If LB is used the Service Engine deployment will happen during the phase automatically by Avi Controller.

For each Org VDC perform vApp migration step:

vcdNSXMigrator.exe  –filepath=sampleuserinput.yml  –passwordFile=passfile  -e movevapp

Clean-up Phase

1. Clean up all source Org VDCs with:
   
   vcdNSXMigrator.exe  –filepath=sampleuserinput.yml  –passwordFile=passfile  –cleanup
   
   
2. As the previous step deleted all NSX-V ESGs we can go back to NSX Migration Coordinator and convert the remaining host to NSX-T. We can also remove all DRS rules that we created for ESGs and VCD Workloads.

3. Disable and remove source Provider VDC.
4. In case there are no more NSX-V backed provider VDCs in the environment remove VXLAN network pools and unregister NSX-V from VCD as described here.
5. If you migrated your last NSX-V cluster perform NSX-V cleanup from vSphere as described here: https://docs.vmware.com/en/VMware-NSX-T-Data-Center/3.2/migration/GUID-7A07429B-0576-43D0-8808-77FA7189F4AE.html

With the upcoming end-of-life for the VMware NSX Migration fo VMware Cloud Director that has been announced here the source code and all the released builds are now available publicly on Calsoft GitHub repository: https://github.com/Calsoft-Pvt-Ltd/NSX_V2T_Migration_Tool. Calsoft engineers were developing the tool from the beginning so they do have the expertese to maintain the code and support those service providers that have not been able yet to migrate. If you are such a provider I encourage you to contact Calsoft: techsupport@calsoftinc.com.

VMware Cloud Director 10.6 has been released yesterday. It is quite a big release with lots of new functionality especially related to three-tier tenancy model and catalog distribution, but in this blog post I want to focus on the large number of the new network related features.

Provider Gateways

Provider GWs are backed by NSX Tier-0 GWs or by VRFs on those Tier-0 GWs. In the past you however could not use mixed mode of importing both the parent Tier-0 GW and its children VRFs as Provider GWs. This is now possible and can be used together with NSX 4.1 inter VRF routing with BGP route leaking to easily create network topologies that require tenant private Provider GWs without complicated uplink configurations on them. The diagram below shows example of such topology where parent Tier-0 GW with VLAN uplinks provides connectivity to Internet and is used by some VCD Edge GWs (e.g. Services Tier-1 GW). At the same time we have two VRFs (A and B) imported into VCD as Provider GWs as well and they route to Interent internally via inter VRF routing set up with the parent Tier-0. Tenants can have dedicated direct connect networks, NAT/fireall or VPN tunnels running on those Provider GWs or dedicated IP Spaces.

As hinted above the provider can now expose certain Provider GW services to the tenant via Topology Intentions feature. What type of networks can be route advertised, can they use NAT/Firewall or setup IPSec and L2 VPN.

In case the Provider GWs is backed by Tier-0 GW or VRF in A/S mode the tenant can (if allowed) to configure NAT/FW and route based IPSec VPN service there. Note that NSX 4.1 is needed for IPSec VPN on VRF. The BGP configuration for route based IPSec VPN is simplified via the autoconfigure feature:

The provider can also granularly decide which BGP configuration elements will be visible/editable to the tenant. This is configured via BGP Permission Groups.

BGP configuration is again simplified via autoconfiguration that is linked to the IP Spaces.

Both the provider and tenant can download routing table or advertised/learned routes from each BGP peer.

The provider can also configure GRE tunnels on the Provider GW.

Edge Gateways

Edge Gateways (backed by Tier-1 NSX GWs) can now also be deployed in the Distributed only mode without the SR (services router) component. In such case the Tier-1 GW does not provide services that run on SR such as firewalling, NAT, VPN, DNS forwarding or static routes. The distributed nature guarantees high N/S data throughput (no hairpinning of traffic to single edge node running the active SR component). Load balancing, rate limiting or DHCP service in network mode is still supported. The other use case beside performance is to limit the NSX licensing costs.

Tenants can deploy route based IPSec VPN on the Edge Gateway however only with static routes as dynamic routing is not supported on Tier-1 GW. To contrast this with IPSec VPN on Provider GW discussed previously the main difference is that you do not need to make the Provider GW private in this case.

Three-Tier Networking

Three-Tier tenancy feature introduces sub-provider and sub-tenant constructs. Sub-providers are enpowered by the provider to provision and manage their sub-tenants and their resources. If we focus on the networking aspect Network Pools and Edge Clusters are now visible (via grants) to sub-providers who can now provision Org VDCs and Edge Gateways for their sub-tenants. Edge Gateways can be connected to to Provider Gateways owned by the subprovider. While the sub-provider can manage IP Spaces the linkage of (public) IP Space to Provider GW still must be done by the provider. Private sub-provider IP Space cannot be granted to sub-tenants. Tier-0 or VRF backed provider GWs can be used for this purpose.

Data Center group scale has been increased to 2000 participating VDCs which if created by the sub-provider can be shared across sub-tenants. This enables use cases where the sub-provider can manage all networking and services on behalf of the sub-tenants. Strict mode IPAM can be enabled to enforce that sub-tenants cannot use IPs outside the static pools associated with the Org VDC network.

Tenant Network Logging

Aria Operations for Log insight can be attached to VCD for log ingestion where ESXi hosts forward DFW logs and NSX Edge Nodes GW FW logs. The tenant now can browse their logs in the DC Group or GW context via UI or API.

Note that this feature does not require NSX Project integration – filtering is done based on ruleIDs.

Advanced Load Balancing

VCD now exposes to tenants the ability to create custom health monitors for their applications.

In large deployments it is now easier to scale NSX ALB to match NSX Tier-1 maximums. I blogged about workaround how to go above the supported 128 Tier-1 GW in a single NSX Cloud. This is no longer needed – multiple NSX Clouds from a single or multiple ALB Clusters can be associated with single Network Pool (NSX transport zone). When load balancer is enabled at Edge GW automated or manual selection is made in which ALB NSX Cloud the placement will occur.

IPAM Reservations

Tenants can reserve IPs from Org VDC network pool for floating application VIPs, load balancer virtual services, fixed DNS record, NAT rules or for an external service. Such reservations can be time bound or related to vApp life-cycle.

This is what I have been working on recently – Broadcom own internal private cloud backed by VMware Cloud Foundation and Cloud Director.

One of the great capabilities of VMware Cloud Director is to build nested labs for development, testing, education or demonstration purposes. VMworld Hands on Labs used this capability for ages. Recently I got numerous questions on how to build such lab so I want to summarize all the important information here.

Requirements

• self-contained lab runing in VCD cloud environment with equivalent functionality to physical lab
• rapid deployment of multiple instances that do not interfere with each other
• reset, capture and clone capabilities

High Level Design

Each lab is running in a single routed vApp that is connected to the Org VDC Edge Gateway via vApp Edge backed by Tier-1 GW. Inside the vApp there is usually single transit network where all the VMs are connected. Nested VLANs can be used and internal routing is provided by vPod Router which is essentially a router VM (e.g. VyOS or similar). The lab can be accessed from ControlCenter Windows VM that acts as a jumbox to which you can connect via RDP but can also have additional roles (DNS, AD, …).

vApp Edge provides NAT from Org VDC network IP to vPod Router who then NATs the RDP traffic to ControlCenter.

Provider Design

We assume recent VCD version is used with NSX-T providing the networking functionality.

Org VDC needs to have Edge Cluster assigned in order to deploy vApp Edges. The Org VDC network must be overlay backed and cannot be VLAN backed.

If we plan to use vPod Router own DHCP server and nested ESXi workloads a specific NSX Segment Profile Template must be set up. In NSX, MAC Discover segment profile must be created with MAC Change and Learning enabled – this is needed due to VMs or vmkernel ports on nested ESXi hosts will have different MAC addresses than the ESXi NICs. Segment Security segment profile must be created with DHCP Server Block disabled (which is enabled by default).

Then in VCD in Resources > Infrastructure Resources > NSX-T we can create Segment Profile Template utilizing the two above segment profiles.

Then we can assign this Segment Profile Template to our Org VDC for vApp networks.

Finally if we want to enable rapid provisioning of the labs we can enable Fast Provisioning in Policies > Storage for the Org VDC which will deploy all vApps as linked clones from the catalog templates which will take seconds as opposed to minutes for full clones.

Lab Design

Lab design is fully in scope of the tenant. They can create labs as they wish – what follows is my recommended approach.

Create vApp with single routed vApp network called Transit and IP subnet 192.168.0.0/24 and an IP pool. This network will be connected to the Org VDC network. If you plan on using VLANs enable Allow Guest VLAN.

Deploy vPod Router VM. I prefer to use VyOS in a VM. Use at least two network interfaces where the first one needs to use IP allocation from the Transit IP Pool. Set the others to use DHCP. IP Pool allocation mode is necessary to allow later on NAT from vApp Edge. The DHCP config essentially tells VCD not to use its IPAM – we will set static IPs internally in the vPodRouter.

Guest customization should be generally disabled on all lab VMs, you will need to configure IPs manually from within each VM.

configure
set interfaces ethernet eth0 address 192.168.0.2/24
set protoctols static route 0.0.0.0/0 next-hop 192.168.0.1
set service ssh listen-address 192.168.0.2
commit
save

Now we can configure NAT to vPodRouter on the vApp Edge. At the vApp level go to Networks and click on the Transit network. In services we will configure Firewall to enable incoming traffic, make sure NAT is enabled and there is NAT rule to our first vNIC of vPodRouter.

You should now be able to SSH to your lab via the External IP shown above (10.100.11.3).

Note that the above approach will consume two IPs from the Org VDC network IP Pool. One for the vApp Edge and one for the vPodRouter NAT (1:1 SNAT/DNAT or reflexive – can be set in Org VDC Network Policies). The alternative approach is to change the NAT from IP Translation to Port Forwarding and forward only specific ports (22, 3389) to vPodRouter – then only single IP is used from the Org VDC Network Pool (Router External IP)

Now you should configure internal networking of the vApp – set up the other vPodRouter interfaces, create virtual interfaces if VLANs are to be used, configure NTP and optionally DHCP.

For example we will set up 192.168.110.0/24 subnet on the 2nd interface:

configure
set interfaces ethernet eth1 address 192.168.110.1/24
set interfaces ethernet eth1 mtu 1700
commit
save

Once this is done we can deploy ControlCenter – this is usually Windows server VM with AD, CA, DNS and other roles enabled and with enabled RDP service. You would connect it to the transit network but configure its IP to the secondary subnet: e.g. 192.168.110.10. Set VCD IP allocation to DHCP and as said disable guest customization and set IP internally inside the VM.

Lastly we configure NAT of RDP traffic on the vPodRouter and masquerade so all lab VMs can access internet.

configure
set nat destination rule 10 description “Portforward RDP to ControlCenter”
set nat destination rule 10 destination port 3389
set nat destination rule 10 inbound-interface name eth0
set nat destination rule 10 protocol tcp
set nat destination rule 10 translation address 192.168.110.10
set nat source rule 100 outbound-interface name eth0
set nat source rule 100 translation address masquerade
commit
save

You should now be able to RDP to the ControlCenter VM via the same IP as was SSH to the vPodRouter.

Capture this basic vApp to catalog with Make identical copy option.

This concludes the basic lab framework. The next step is to deploy additional VMs and appliances based on the lab needs.

Important Tips

Here is collection of helpful tips on lab setups

• Do not use .local for your domain name suffix. E.g. controlcenter.corp.local. The reason is that some DNS clients do not send .local domain to an external server and your DNS resolution will not work correctly.
• When deploying nested ESXi hosts make sure none of the vmk NICs is matching the MAC address of the ESXi VMNICs. By default vmk0 is using vmnic1 MAC address. This will break NSX MAC learning if you have multiple NICs on ESXi and teaming policy will decide to use other vmnic. The easiest is to deploy and configure ESXi and then reset MAC addresses in VCD.

• If you need storage for your nested ESXi hosts either use vSAN or TrueNAS or similar storage appliance.
• When using nested ESXi hosts you have the option to deploy other lab VMs either nested on those hosts or as regular VMs inside the vApp. The first option is easier as you do not need to modify the OVAs to work under VCD, but the performance will be much slower and your nested ESXi VMs need enough compute/storage. For example I prefer to deploy NSX Edge Nodes as vApp VMs and not as nested – but you cannot use the NSX UI workflow to do so.
• Some OVA appliances must be modified in order to deploy them to VCD as not all features VC support are enabled in VCD. For example ExtraConfig properties or DeploymentOptions must be removed or edited. Extract (unzip) OVA, edit OVF and upload to catalog only OVF and vmdk files.
• When deploying NSX make sure your MTU is set correctly on the vPodRouter. As you are running on physical overlay with potentially nested VLANs set it conservatively to allow for that additional overhead (e.g. 1700).
• Appliance networking is usually configured via Guest Properties. So make sure you configure them before powering on the VM. Always disable guest customization.

• Set vApp Start and Stop Order to start lab VMs in the correct order and account for time they need to start properly

VMware Cloud Provider Lifecycle Manager (VCPLCM) greatly simplifies upgrades of VMware Cloud Director. However, out of the box its interop bundle manifests support and are aware of only GA versions of VMware Cloud Director. So what do you do if you would like to use a non-GA release (beta or hot fix)?

We need to start with a disclaimer, we assume the release you are trying to upgrade to has the same interop as a GA release that VCP LCM already supports, it just has a new build number. If this is not true, you should know what you are doing.

The interop bundles among other information provide VCP LCM with exact file names and checksum values of OVA and upgrade files. So we need to provide this information to VCP LCM in another way – here is the process:

1. Upload your new OVA/upgrade files to repo directory on VCP LCM appliance as usual. In my case I have these two files:
   VMware_Cloud_Director_10.6.0.11554-24142761_update.tar.gz
   VMware_Cloud_Director-10.6.0.11554-24142761_OVF10.ova
2. Now in the same directory create sha256 checksum files for these two files using the same filename with suffix ‘checksum‘. This can be done with these two commands right on the appliance:
   
   sha256sum VMware_Cloud_Director_10.6.0.11554-24142761_update.tar.gz > VMware_Cloud_Director_10.6.0.11554-24142761_update.tar.gz.checksum
   
   sha256sum VMware_Cloud_Director-10.6.0.11554-24142761_OVF10.ova > VMware_Cloud_Director-10.6.0.11554-24142761_OVF10.ova.checksum
3. Now we can proceed with the upgrade. In the target version however we need to specify the long version name that includes both the appliance version and the build number – so in my case it is: 10.6.0.11554-24142761. You will get support warning as this is not release that matches the interop bundle data.

That’s it – the rest of the process is as usual (clicking Validate and OK) and you should be able to upgrade your VCD instance within a few minutes.

In some older releases of vSAN there is an issue where if applications are using AVX-512 CPU instructions data corruption can occur. This was first reported by customers using DB2 but can occur with other applications as well. Here is the KB describing the problem: https://knowledge.broadcom.com/external/article/367589/applications-using-avx512-instructions-i.html

Note that the issue has been already fixed in all ESX/vSAN branches so the best way to remediate is to patch your environment. However, if immediate patching is not an option, and you need a quick workaround before the patch can be applied then read on.

To quickly check if a particular VM can use AVX-512 instructions you can run this linux command:

cat /proc/cpuinfo |grep avx512

flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon nopl xtopology tsc_reliable nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 avx2 smep bmi2 invpcid avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xsaves arat pku md_clear flush_l1d arch_capabilities

For Windows VM you can use HWiNFO.

To reproduce the issue, the simplest test is to use ssh to transfer data and check for corruption. Ssh is using openssl that might leverage AVX-512 instructions.

Here I am using Photon OS 5.0.

$ dd if=/dev/zero bs=1M count=1024 | sha256sum -  > sha256sums.zero # one time checksum generation
$ ( while true; rm -f /tmp/zero.$$.img ; do ssh root@localhost "dd if=/dev/zero iflag=fullblock bs=1M count=1024 status=progress" | dd iflag=fullblock bs=1M status=none of=/tmp/zero.$$.img && sha256sum - < /tmp/zero.$$.img | diff -u sha256sums.zero - || exit 1; done )

After a while the checksum will fail:

...
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.11915 s, 507 MB/s
1013972992 bytes (1.0 GB, 967 MiB) copied, 2 s, 507 MB/s
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.09118 s, 513 MB/s
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2 s, 532 MB/s
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.01729 s, 532 MB/s
--- sha256sums.zero     2024-10-21 10:44:48.142238490 +0000
+++ -   2024-10-21 10:54:17.289072688 +0000
@@ -1 +1 @@
-49bc20df15e412a64472421e13fe86ff1c5165e18b2afccf160d4dc19fe68a14  -
+4e841c1d34a2a051fab082d10a8105cd34e10c95ee74ce20d280a7c022eaaa53  -

The workaround can be done either at the application level (see the DB2 specific instruction in the KB above), OS level where the CPU instructions flags can be masked in kernel (append clearcpuid=304 at the end of the grub commandline) or the best at the virtual hardware level which is obviously the most universal as it is neither application nor OS specific.

The following two Advanced Configuration Parameters will mask the problematic AVX-512 CPU instructions:

 featMask.vm.cpuid.AVX512F = “Max:0”
 featMask.vm.cpuid.AVX512FP16 = “Max:0“

They can be applied for each VM in vCenter, or alternatively in VMware Cloud Director environment we can create Compute Policy (in UI called VM Sizing Policy) that the end-users can apply to their VMs.

This is a description on how to setup Plex Media Server to run as a virtual machine on vSphere. I am using Linux distribution and exposing CPU integrated Intel UHD Graphics 630 for accelerated transcoding of UHD video sources.

1. Go directly to the ESXi host client UI and in Manage > Hardware enable Passthrough on the UHD Graphics 630.
   

2. In vCenter go to the ESX host > Configure > Hardware > Graphics and enable Shared Direct on the GPU.

3. Create VM with 2 vCPU, 4 GB RAM and 40 GB HDD. Set OS to Rocky Linux 64-bit.

4. Add new PCI Device to the VM and select the GPU in DirectPath IO mode.

5. Download latest Rocky Linux minimal ISO mount to the VM and install the OS and configure network (nmtui).

6. Install nfs-utils, create mount point (mkdir /mount/Media) and mount your media libraries by adding line pointing to your NFS NAS export with vi /etc/fstab

10.0.2.246:/Media /mount/Media nfs timeo=20,intr

7. Install Plex Media Server using Plexupdate.

yum install wget
bash -c “$(wget -qO – https://raw.githubusercontent.com/mrworf/plexupdate/master/extras/installer.sh)”

7. (optional) Install Tautulli (for monitoring)

yum install snapd
yum install epel-release
yum install snapd
snap install tautulli

8. Check that GPU is properly detected with lshw -C display

[root@plex-server ~]# lshw -C display
*-display
description: VGA compatible controller
product: SVGA II Adapter
vendor: VMware
physical id: f
bus info: pci@0000:00:0f.0
logical name: /dev/fb0
version: 00
width: 32 bits
clock: 33MHz
capabilities: vga_controller bus_master cap_list rom fb
configuration: depth=32 driver=vmwgfx latency=64 resolution=1280,800
resources: irq:16 ioport:840(size=16) memory:f0000000-f7ffffff memory:ff000000-ff7fffff memory:c0000-dffff
*-display
description: VGA compatible controller
product: CoffeeLake-S GT2 [UHD Graphics 630]
vendor: Intel Corporation
physical id: 3
bus info: pci@0000:02:03.0
version: 02
width: 64 bits
clock: 33MHz
capabilities: pciexpress msi pm vga_controller bus_master cap_list
configuration: driver=i915 latency=0
resources: irq:32 memory:fc000000-fcffffff memory:d0000000-dfffffff ioport:a00(size=64)

VMware Cloud Director stores ISO catalog media in a directory structure on one or more datastores backing the storage policy assigned to the catalog. The directory structure looks like this:

<VCD Instance Name>/media/<org ID>/<org VDC ID>/media-<media ID>.<ISO|FLP>

In case the datastore is backed by vSAN the whole directory tree (starting with the VCD Instance Name root folder) exists within a single vSAN Namespace object that has maximum size 255 GB. This might not be enough if you consider that the same directory is shared across multiple Org VDCs and Orgs (your tenants could altogether upload just about 25 10GB ISO images).

As of vSAN 8.0U1 there is a supported way to extend the VM Namespace limit via PowerCLI. The details are in the vSAN Operations Guide (page 124).

I will show briefly how to perform the resizing.

You need to find the vSAN datastore UUID and the object UUID. Both can be seen in the vCenter UI.

vSAN datastore UUID is seen in the location in the Summary view of the datastore.

The VM Namespace object UUID backing the VCD catalog directory can be seen in the Files view.

With that information we can now resize the object. The following PowerCLI commands performed risizing from 255 GB to 512 GB.

$services = Get-view 'ServiceInstance'
$datastoreMgr = Get-view $services.Content.DatastoreNamespaceManager
$datacenter = get-datacenter
$datastoreMgr.QueryDirectoryInfo($datacenter.ExtensionData.MoRef,"/vmfs/volumes/vsan:6c46b8baef884c56-96c033ba12d72500/175e8566-fcf9-130d-7b77-043201572d40")

Capacity  Used
--------  ----
  261120 14737

$datastoreMgr.IncreaseDirectorySize($datacenter.ExtensionData.MoRef, "/vmfs/volumes/vsan:6c46b8baef884c56-96c033ba12d72500/175e8566-fcf9-130d-7b77-043201572d40", 512000)
$datastoreMgr.QueryDirectoryInfo($datacenter.ExtensionData.MoRef,"/vmfs/volumes/vsan:6c46b8baef884c56-96c033ba12d72500/175e8566-fcf9-130d-7b77-043201572d40")

Capacity  Used
--------  ----
  512000 14737

Note that ISOs cannot be mounted to a VM that is running on a host that does not have access to the ISO datastore. This might be an issue with vSAN if HCI Mesh is not used as in such case vSAN datastore can be accessed only by VMs on the same cluster. So this approach would not be recommended for Elastic Org VDCs spanning multiple clusters.