Just a quick post to elaborate on the feature to add custom links into the H5 vCloud Director Portal that I very briefly mentioned in my What’s New in vCloud Director 9.7 blog post.

The custom links are visible in the drop down under the user name in top right corner. They are specified as part of the branding customization that can be system wide or tenant specific. I have already talked about branding here, so this is just evolution of this feature since version 9.0.

Additionally default links to help, about (under the questionmark icon) and VMRC download (in VM UI element) can be replaced as well through a single API call.

Custom links can contain dynamic elements such as ${TENANT_NAME}, ${TENANT_ID} and ${SESSION_TOKEN} which enables for example easy redirection to legacy (Flex) portal as is show in my example.

You can also see Section elements and Separators to make the links more organized.

The actual API call that I used in my example:

PUT /cloudapi/branding

Headers:

Content-Type: application/json
x-vcloud-authorization: …

Payload:

{

    "portalName": "vCloud Director 9.7",
    "portalColor": "#323843",
    "selectedTheme": {
        "themeType": "BUILT_IN",
        "name": "Default"
    },
    "customLinks": [
        {
            "name": "Legacy Portal",
            "menuItemType": "link",
            "url": "https://vcloud.fojta.com/cloud/org/${TENANT_NAME}"
        },
        {
            "menuItemType": "separator"
        },
        {
            "name": "Additional Services",
            "menuItemType": "section",
            "url": null
        },
        {
            "name": "Backup",
            "menuItemType": "link",
            "url": "https://fojta.wordpress.com/backup"
        },
        {
            "name": "Monitoring",
            "menuItemType": "link",
            "url": "https://fojta.wordpress.com/monitor"
        },
        {
            "name": "Billing",
            "menuItemType": "link",
            "url": "https://fojta.wordpress.com/bill"
        },
        {
            "menuItemType": "separator"
        },
        {
            "name": "Tools",
            "menuItemType": "section",
            "url": null
        },
        {    "name": "VCD CLI",
            "menuItemType": "link",
            "url": "http://vmware.github.io/vcd-cli/"
        },
        {    "name": "API Documentation",
            "menuItemType": "link",
            "url": "https://code.vmware.com/apis/553"
        },
        {
            "name": "help",
            "menuItemType": "override",
            "url": null
        },
        {
            "name": "about",
            "menuItemType": "override",
            "url": null
        },
        {
            "name": "vmrc",
            "menuItemType": "override",
            "url": null
        }
    ]
}

Tenant specific branding is achieved with the following call:

PUT /cloudapi/branding/tenant/acme

With clockwork efficiency after less than 6 months there a is new major release of vCloud Director – version 10. As usual, I will try to summarize all the new functionality compared to the previous release 9.7. I have similar posts about 9.7, 9.5 and 9.1 so you can get quickly up to speed if you are not familiar with them as well.

User Interface

From the tenant UI perspective the HTML5 UI (/tenant) has been evolving to add missing legacy (Flex) UI functionality. You can now customize VM network adapter during VM creation, change user password and user settings.

The top ribbon bar now provides more information and new search option.

New universal tenant login page (/login) was added:

Tenant UI also provides new functionality such as NSX-T network management.

The provider HTML5 UI now contains all the actions the cloud service provider needs to do (various Settings screens, tenant migration, …), so the legacy Flex UI is actually disabled by default. There are still however some missing features like direct VM import from vCenter Server, Org VDC template creation or edit of VM guest properties.

If necessary, you can enable Flex UI with this command (run on any cell and reboot them all):

cell-management-tool manage-config -n flex.ui.enabled -v true

Among some of the new Provider UI features are:

• compute policy management (VM Sizing Policies and Provider VDC specific VM Placement Policies).
  
  
• NSX-T provider actions such as Geneve network pool creation, import of T0 for external networks and Org VDC Edge Gateway management including quite useful quick external IP addresses sub-allocation (available for NSX-V Edge Gateways in API as well).
  
• SDDC Proxy and token management (CPOM feature)

NSX-T Support

As hinted above, NSX-T integration has been improved massively. I am going to deep dive into the topic in a separate article, so let me cover it here very quickly.

In the previous vCloud Director releases the system administrator could only import NSX-T based networks (overlay logical segments) as tenant Org VDC networks and that was it. In the current release the tenants now can create NAT-routed and isolated networks with firewalling, DHCP and DNS forwarding services provided by NSX-T T1 Gateways. The vCloud Director networking objects did not change much which means there should not be major difference between NSX-V backed and NSX-T backed Org VDC from the usability perspective. However, there is not full feature parity between NSX-V and NSX-T functionality; sometimes it is due to NSX-T not providing these features (SSL VPN), sometimes due to vCloud Director not yet caught up. Expect more in the future as this is a journey.

Note: Only NSX-T version 2.5 is supported by vCloud Director 10.0.

API

• API version has been bumped up to 33.0, while versions 27.0-32.0 are still supported but 27.0 and 28.0 are marked for deprecation.
• There is a new API authentication mechanism. The OpenAPI provides two different authentication endpoints (one for provider: /cloudapi/1.0.0/sessions/provider the other for tenants /cloudapi/1.0.0/sessions). You can disable for API version 33.0 the old authentication mechanism (/api/sessions) with the following command:cell-management-tool manage-config -n vcloud.api.legacy.nonprovideronly -v trueThis means it is now quite easy with Web Application Firewall to protect the provider API authentication from the internet.
• OpenAPI provides new (faster) way to collect audit events from vCloud Director via AuditTrail API call. Note that vCloud Director now stores audit events only for limited time in order to keep the database size and query speed manageable.
• The NSX-T related networking APIs are not pass-through as was the case with NSX-V and instead use the OpenAPI calls.
• vCloud Director Appliance API: each appliance node now provides its own appliance API to get database state provide by replication manager. It is also possible to remotely execute database standby node promotion  and thus automate database failover with external tooling or load balance to the active database node for 3rd party database usage.
  GET https://<appliance IP>:5480/api/1.0.0/is_primary
  GET https://<appliance IP>:5480/api/1.0.0/nodes
  POST https://<appliance IP>:5480/api/1.0.0/nodes/<node name>/promote

Other Features

• Improved vRealize Orchestrator (vRO) integration. Two more custom properties vcd_sessionToken and _vcd_apiEndpoint can be passed from vCloud Director to vRO workflow so the workflow during its execution can connect in the particular user context via the vCloud Director Plugin to vCloud Director and provide access only to those objects the user has access to.
  The spelling of two other custom properties was fixed from _vdc_userName and _vdc_isAdmin to _vcd_userName and _vcd_isAdmin (but is still backwards compatible).
  The new vRO vCloud Director Plugin now also supports vRO Clustering so the vCloud Director connection is automatically shared across vRO nodes.
• RBAC support for NSX-V Edge ECMP and DNS features. The former was asked by many providers in order to keep NSX-V licensing at Advanced edition and not to get accidentally bumped to Enterprise edition if tenant enabled ECMP on its Org VDC Edge Gateway.
• Legacy Org VDC allocation models can now be changed to flex allocation model which allows for switching allocation models of existing Org VDCs.
• When system administrator enables Distributed Firewall via UI it is possible to choose if the new tenant firewall section should be created at the bottom (and not on top by default). This was before possible only via API.

• MS SQL is no longer supported as vCloud Director database. To use vCloud Director version 10.0 you must either use the appliance form factor with its embedded PostgreSQL database or an external PostgreSQL. Migration is supported.
• Compatible VCD-CLI version 22.0 and pycloud 21.0 SDK were released as well.

Intro

vCloud Director relies on NSX network virtualization platform to provide on-demand creation and management of networks and networking services. NSX for vSphere has been supported for long time and vCloud Director allows most of its feature to be used by its tenants. However as VMware slowly shifts away from NSX for vSphere and pushes forward modern, fully rewritten NSX-T networking platform, I want to focus in this article on its integration with vCloud Director.

History

Let me start with highlighting that NSX-T is evolving very quickly. It means each release (now at version 2.5) adds major new functionality. Contrast that with NSX-V which is essentially feature complete in a sense that no major functionality change is happening there. The fast pace of NSX-T development is a challenge for any cloud management platforms as they have to play the catch up game.

The first release of vCloud Director that supported NSX-T was 9.5. It supported only NSX-T version 2.3 and the integration was very basic. All vCloud Director could do was to import NSX-T overlay logical segments (virtual networks) created manually by system administrator. These networks were imported into a specific tenant Org VDC as Org VDC networks.

The next version of vCloud Director – 9.7 supported only NSX-T 2.4 and from the feature perspective not much had changed. You could still only import networks. Under the hood the integration however used completely new set of NSX-T policy based APIs and there were some minor UI improvements in registering NSX-T Manager.

The current vCloud Director version 10 for the first time brings on-demand creation of NSX-T based networks and network services. NSX-T version 2.5 is required.

NSX-T Primer

While I do not want to go too deep into the actual NSX-T architecture I fully expect that not all readers of this blog are fully familiar with NSX-T and how it differs from NSX-V. Let me quickly highlight major points that are relevant for topic of this blog post.

• NSX-T is vCenter Server independent, which means it scales independently from vCenter domain. NSX-T essentially communicates with ESXi hosts directly (they are called host transport nodes). The hosts must be prepared with NSX-T vibs that are incompatible with NSX-V which means a particular host cannot be used by NSX-V and NSX-T at the same time.
• Overlay virtual networks use Geneve encapsulation protocol which is incompatible with VXLAN. The concept of Controller cluster that keeps state and transport zone is very similar to NSX-V. The independence from VC mentioned in the previous point means vSphere distributed switch cannot be used, instead NSX-T brings its own N-VDS switch. It also means that there is concept of underlay (VLAN) networks managed by NSX-T. All overlay and underlay networks managed by NSX-T are called logical segments.
• Networking services (such as routing, NATing, firewalling, DNS, DHCP, VPN, load balancing) are provided by Tier-0 or Tier-1 Gateways that are functionally similar to NSX-V ESGs but are not instantiated in dedicated VMs. Instead they are services running on shared Edge Cluster. The meaning of Edge Cluster is very different from the usage in NSX-V context. Edge Cluster is not a vSphere cluster, instead it is cluster of Edge Transport Nodes where each Edge Node is VM or bare metal host.
• While T0 and T1 Gateways are similar they are not identical, and each has specific purpose or set of services it can offer. Distributed routing is implicitly provided by the platform unless a stateful networking service requires routing through single point. T1 GWs are usually connected to single T0 GW and that connection is managed automatically by NSX-T.
• Typically you would have one or small number of T0 GWs in ECMP mode providing North-south routing (concept of Provider Edge) and large number of T1 GWs connected to T0 GW, each for a different tenant to provide tenant networking (concept of Tenant Edge).

vCloud Director Integration

As mentioned above since NSX-T is not vCenter Server dependent, it is attached to vCloud Director independently from VC.

(Geneve) network pool creation is the same as with VXLAN – you provide mapping to an existing NSX-T overlay transport zone.

‘Now you can create Provider VDC (PVDC) which is as usual mapped to a vSphere cluster or resource pool. A particular cluster used by PVDC must be prepared for NSX-V or NSX-T and all clusters must share the same NSX flavor. It means you cannot mix NSX-V clusters with NSX-T in the same PVDC. However you can easily share NSX-V and NSX-T in the same vCenter Server, you will then just have to create multiple PVDCs. Although NSX-T can span VCs, PVDC cannot – that limitation still remains. When creating NSX-T backed PVDC you will have to specify the Geneve Network Pool created in the previous step.

Within PVDC you can start creating Org VDCs for your tenants – no difference there.

Org VDCs without routable networks are not very useful. To remedy this we must create external networks and Org VDC Edge Gateways. Here the concept quite differs from NSX-V. Although you could deploy provider ECMP Edges with NSX-V as well (and I described here how to do so), it is mandatory with NSX-T. You will have to pre-create T0 GW in NSX-T Manager (ECMP active – active is recommended). This T0 GW will provide external networking access for your tenants and should be routable from the internet. Instead of just importing external network port group how you would do with NSX-V you will import the whole T0 GW in vCloud Director.

During the import you will also have to specify IP subnets and pools that the T0 GW can use for IP sub-allocation to tenants.

Once the external network exist you can create tenant Org VDC Edge Gateways. These will be T1 GWs instantiations into the same NSX-T Edge Cluster as the T0 GW they connect to. Currently you cannot chose different NSX-T Edge Cluster for their placement. T1 GWs are always deployed in Active x Standby configuration, the placement of active node is automated by NSX-T. The router interlink between T0 and T1 GWs is also created automatically by NSX-T.

During the Org VDC Edge Gateway the service providers also sub-allocates range of IPs from the external network. Whereas with NSX-V these would actually be assigned to the Org VDC Edge Gateway uplink, this is not the case with NSX-T. Once they are actually used in a specific T1 NAT rule, NSX-T will automatically create static route on the T0 GW and start routing to the correct T1 GW.

Tenant Networks

There are four types of NSX-T based Org VDC networks and three of them are available to be created via UI:

• Isolated: Layer 2 segment not connected to T1 GW. DHCP service is not available on this network (contrary to NSX-V implementation).
• Routed: Network that is connected to T1 GW. Note however that its subnet is not announced to upstream T0 GW which means only way to route to it is to use NAT.
• Imported: Existing NSX-T overlay logical segment can be imported (same as in VCD 9.7 or 9.5). Its routing/external connectivity must be managed outside of vCloud Director.
• In OpenAPI (POST /1.0.0/OrgVdcNetwork) you will find one more network type:  DIRECT_UPLINK. This is for a specific NFV use case. Such network is connected directly to T0 GW with external interface. Note this feature is not officially supported!

Note that only Isolated and NAT-routed networks can be created by tenants.

As you can see it is not possible today to create routed advertised Org VDC network (for example for direct connect use case when tenant wants to route from on-prem networks to the cloud without using NAT). These routed networks would require dedicated T0 GW for each tenant which would not scale well but might be possible in the future with VRF support on T0 GWs.

Tenant Networking Services

Currently the following T1 GW networking services are available to tenants:

• Firewall
• NAT
• DHCP (without binding and relay)
• DNS forwarding
• IPSec VPN: No UI, OpenAPI only. Policy and route based with pre share key is supported. (Thanks Abhi for the correction).

All other services are currently not supported. This might be due to NSX-T not having them implemented yet, or vCloud Director not catching up yet. Expect big progress here with each new vCloud Director and NSX-T release.

Networking API

All NSX-T related features are available in the vCloud Director OpenAPI (CloudAPI). The pass through API approach that you might be familiar with from the Advanced Networking NSX-V implementation is not used!

Feature Comparison

I have summarized all vCloud Director networking features in the following table for quick comparison between NSX-V and NSX-T.

This is just a short blog post to describe an issue you might get with the tenant or portal HTML UI in vCloud Director where you will see errors in the browser related to request header fields too large.

You will see it more likely with Chrome browser and if your cloud domain is shared with other services. The root cause is that the browser API calls will stop working once the request header gets larger than 8 KBs. While 8 KBs seems like big enough size especially as the request headers vCloud Director uses contain only session ID, JWT token and possibly load balancers headers it also includes all the browser cookies applicable to the vCloud Director domain stored by other web services.

The temporary fix is for the end-user to delete her browser cookies. But is there something the provider could do?

In our case we saw the situation where the vCloud Director instance was on *.vmware.com domain and the browser contained lots of large OAM cookies related to VMware Single Sign-On solution. While those cookies are essential for multiple VMware internal applications, there is no reason for vCloud Director to receive them in every API request. One way how to block the cookies and thus decrease the request header size is to remove them at the load balancer. With NSX-V load balancer this can be accomplished by utilizing SSL L7 termination and an application rule (see my older blog post how to configure NSX-V Edge Load balancer).

In my case the application rule I use is:

Update 2019/10/24: The initial rule would remove all Cookies. I have now amended it with another rule that removes all but vcloud_session_id and vcloud_jwt cookies if they are present.

reqirep ^Cookie:\s.*(vcloud_session_id=[^;]*)|(vcloud_jwt=[^;]*) Cookie:\ \1;\ \2
reqidel ^Cookie:.*OAM*

which deletes all cookies from the request header starting with OAM string

One of my vSphere 6.7 U3 environments I am managing was still using external Platform Services Controller (PSC) from times when it was the prescribed architecture. That is no longer the case so to simplify my management I wanted to get rid of the external PSC via so called Convergence to embedded PSC.

Unfortunately although there is a very nice UI to do this it never worked for me. And I did try multiple times. The error I always ended up was:

Failed to get RPMs.

The /var/log/vmware/converge/converge.log log did not show any error, but what was peculiar there was this entry referring to download of VCSA 6.5.0 files?!

2019-10-29T16:02:01.223Z INFO converge currentURL = https://vapp-updates.vmware.com/vai-catalog/valm/vmw/8d167796-34d5-4899-be0a-6daade4005a3/6.5.0.10000.latest/
2019-10-29T16:02:01.223Z INFO converge Manifest file = https://vapp-updates.vmware.com/vai-catalog/valm/vmw/8d167796-34d5-4899-be0a-6daade4005a3/6.5.0.10000.latest/manifest/manifest-latest.xml

These are obviously not correct for my 6.7 U3 VCSA appliance. This VMware Communities thread finally pushed me in the right direction.

Here are the steps how to resolve this:

1. Delete content of /root/vema directory on VCSA
2. Download correct VCSA ISO installation media corresponding to the version of your VC. In my case it was the full 4 GB VMware-VCSA-all-6.7.0-14836122.iso. The patch media VMware-vCenter-Server-Appliance-6.7.0.41000-14836122-patch-FP.iso cca 2 GB large did not work.
3. Mount the ISO to your VCSA
4. Re-run the convergence via the UI

Just a quick announcement that a vCloud Director Object Storage Extension Reference Design that I wrote is now available at this link.

It deep dives into the use cases, architecture, includes recommended deployment options and description of the new feature of 1.0.1 related to multisite deployments. There are also results of performance tests of the overhead the extension adds over the direct use of the native storage platform.

Very brief blog post to document how to install vCloud Director with external database that does not support plain text connections.

In general the process is to do the initial set up with plain text DB connection and then switch to TLS – see the official docs here. That will however not work if the external database supports only TLS connection.

Instead this process must be used:

1. Import DB certificate (unless it is publicly signed) to cell default JRE keystore.
2. Use unattended configuration.

Example:

# /opt/vmware/vcloud-director/jre/bin/keytool --import -trustcacerts -keystore /opt/vmware/vcloud-director/jre/lib/security/cacerts -alias psql -file /opt/vmware/vcloud-director/etc/psql.crt

Enter keystore password: changeit

Owner: CN=10.0.4.64

Issuer: CN=10.0.4.64

Serial number: cb64ae0954184182

Valid from: Fri Nov 22 14:10:39 GMT 2019 until: Sat Nov 21 14:10:39 GMT 2020 Certificate fingerprints:

MD5:  04:4F:8F:C5:9C:CC:D5:E8:F1:50:C1:85:51:D4:FB:AD

SHA1: 22:53:FF:71:A7:EC:9B:D1:74:79:D5:95:46:71:F6:38:A7:E7:F8:4E

SHA256: 08:7C:27:B4:FB:32:04:DE:AF:BB:FE:9D:47:1D:38:46:C8:F4:7C:69:73:DE:8D:CB:BD:2A:A5:B2:11:12:68:DD

Signature algorithm name: SHA256withRSA

Subject Public Key Algorithm: 2048-bit RSA key

Version: 3

Extensions:

#1: ObjectId: 2.5.29.35 Criticality=false AuthorityKeyIdentifier [ KeyIdentifier [

0000: 15 EA 78 3F 71 DD 34 D4   15 F0 C8 03 F7 76 1A 0B  ..x?q.4......v..

0010: 64 B2 A6 6E                                        d..n

]

]

#2: ObjectId: 2.5.29.19 Criticality=false BasicConstraints:[

CA:true

PathLen:2147483647

]

#3: ObjectId: 2.5.29.14 Criticality=false SubjectKeyIdentifier [ KeyIdentifier [

0000: 15 EA 78 3F 71 DD 34 D4   15 F0 C8 03 F7 76 1A 0B  ..x?q.4......v..

0010: 64 B2 A6 6E                                        d..n

]

]

Trust this certificate? [no]:  yes

Certificate was added to keystore

# /opt/vmware/vcloud-director/bin/configure --unattended -dbhost <DB IP address> -dbname vcloud -dbpassword vcloud -dbtype postgres -dbuser vcloud --database-ssl true –dbport 5423 -ip <cell-ip> --primary-port-http 80 --primary-port-https 443 -cons <cell-ip> --console-proxy-port-https 8443 -k /opt/vmware/vcloud-director/etc/certificates.ks -w <keystore password> -g

.......................................\

Database configuration complete.

# /opt/vmware/vcloud-director/bin/cell-management-tool system-setup --email admin@vcloud.com --full-name 'System Admin' --installation-id 33 --password 'VMware1!' -system-name vcd --unattended --user administrator

Creating admin user.

Setting system details.

Completing system setup.

System setup is complete.

Some time ago I blogged about how I automate acquisition of Let’s Encrypt Certificates for my lab (NSX + vCloud Director) with PowerShell. The old script no longer works due to some changes on Let’s Encrypt side therefore the need for part 2.

To quickly summarize my situation. My lab consists of vCloud Director with multiple cells fronted by NSX-V Load Balancer. I need public certificate for vCloud Director which is uploaded to the NSX-V Load Balancer (that does L7 SSL termination) and to vCloud Director public addresses.

Prerequisites:

• Web server on the domain your are getting the certificate for. It is necessary for the DNS challenge that proves you own the domain you are requesting the certificate for. I am using IIS on the machine I trigger the script from and supply the root folder where the challenge file needs to be placed.
• NSX-V API access information – needed to replace the certificate on the NSX-Edge
• Details about the load balancer (on which Edge it is running and what is the LB application profile of vCloud Director)
• vCloud Director API access information – needed to upload new certificate and the full chain to vCloud Director public addresses.
• PowerShell modules: POSH-ACME and PowerCLI

$Username = "admin"
$Password = "default"
$NSXManager = "nsx01.fojta.com"
$LBEdge = 'edge-1'
$ApplicationProfile = 'applicationProfile-1'
$Email = "mailto:admin@fojta.com"
$Domain = "vcloud.fojta.com"
$Vcd = "vcloud.fojta.com"
$VcdAdmin = "administrator"
$VcdPassword = "vcloud"
$IisAcmeRoot = "C:\inetpub\wwwroot\.well-known\acme-challenge"

$RootCert = "-----BEGIN CERTIFICATE-----
MIIDSjCCAjKgAwIBAgIQRK+wgNajJ7qJMDmGLvhAazANBgkqhkiG9w0BAQUFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTAwMDkzMDIxMTIxOVoXDTIxMDkzMDE0MDExNVow
PzEkMCIGA1UEChMbRGlnaXRhbCBTaWduYXR1cmUgVHJ1c3QgQ28uMRcwFQYDVQQD
Ew5EU1QgUm9vdCBDQSBYMzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
AN+v6ZdQCINXtMxiZfaQguzH0yxrMMpb7NnDfcdAwRgUi+DoM3ZJKuM/IUmTrE4O
rz5Iy2Xu/NMhD2XSKtkyj4zl93ewEnu1lcCJo6m67XMuegwGMoOifooUMM0RoOEq
OLl5CjH9UL2AZd+3UWODyOKIYepLYYHsUmu5ouJLGiifSKOeDNoJjj4XLh7dIN9b
xiqKqy69cK3FCxolkHRyxXtqqzTWMIn/5WgTe1QLyNau7Fqckh49ZLOMxt+/yUFw
7BZy1SbsOFU5Q9D8/RhcQPGX69Wam40dutolucbY38EVAjqr2m7xPi71XAicPNaD
aeQQmxkqtilX4+U9m5/wAl0CAwEAAaNCMEAwDwYDVR0TAQH/BAUwAwEB/zAOBgNV
HQ8BAf8EBAMCAQYwHQYDVR0OBBYEFMSnsaR7LHH62+FLkHX/xBVghYkQMA0GCSqG
SIb3DQEBBQUAA4IBAQCjGiybFwBcqR7uKGY3Or+Dxz9LwwmglSBd49lZRNI+DT69
ikugdB/OEIKcdBodfpga3csTS7MgROSR6cz8faXbauX+5v3gTt23ADq1cEmv8uXr
AvHRAosZy5Q6XkjEGB5YGV8eAlrwDPGxrancWYaLbumR9YbK+rlmM6pZW87ipxZz
R8srzJmwN0jP41ZL9c8PDHIyh8bwRLtTcm1D9SZImlJnt1ir/md2cXjbDaJWFBM5
JDGFoqgCWjBH4d1QB7wCCZAA62RjYJsWvIjJEubSfZGL+T0yjWW06XyxV3bqxbYo
Ob8VZRzI9neWagqNdwvYkQsEjgfbKbYK7p2CNTUQ
-----END CERTIFICATE-----
"

#Set-PAServer LE_STAGE
Set-PAServer LE_PROD

## Read https://github.com/rmbolger/Posh-ACME/wiki/%28Advanced%29-Manual-HTTP-Challenge-Validation

New-PAAccount -AcceptTOS -Contact $Email
New-PAOrder $Domain

$auths = Get-PAOrder | Get-PAAuthorizations
$token = $auths[0].HTTP01Token
$toPublish = Get-KeyAuthorization $token

## Upload challenge file to the IIS web server
New-Item -Path $IisAcmeRoot -Name $token -Value $toPublish

$auths.HTTP01Url | Send-ChallengeAck
New-PACertificate $Domain
$cert = Get-PACertificate

$IssuerCert = [IO.File]::ReadAllText($cert.ChainFile)
$PrivateKey = [IO.File]::ReadAllText($cert.KeyFile)
$LBCertificate = [IO.File]::ReadAllText($cert.CertFile)

## Create authorization string and store in $head
$auth = [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($Username + ":" + $Password))
$head = @{"Authorization"="Basic $auth"}

##Upload certificate
$Uri = "https://$NSXManager/api/2.0/services/truststore/certificate/" + $LBEdge
$Body = "
<trustObject>
<pemEncoding>" + $LBCertificate + $IssuerCert + $RootCert + "</pemEncoding>
<privateKey>" + $PrivateKey + "</privateKey>
<description>vCloud Certificate</description>
</trustObject>"
$r = Invoke-WebRequest -URI $Uri -Method Post -Headers $head -ContentType "application/xml" -Body $Body -ErrorAction:Stop
$NewCertificateId = ([xml]$r).certificates.certificate.objectId

##Delete Root and intermediate certificate from the Edge as they are not needed
$Uri = "https://$NSXManager/api/2.0/services/truststore/certificate/" + $NewCertificateId[0]
$r = Invoke-WebRequest -URI $Uri -Method Delete -Headers $head -ContentType "application/xml" -ErrorAction:Stop
$Uri = "https://$NSXManager/api/2.0/services/truststore/certificate/" + $NewCertificateId[1]
$r = Invoke-WebRequest -URI $Uri -Method Delete -Headers $head -ContentType "application/xml" -ErrorAction:Stop

##Replace certificate in the application profile
$Uri = "https://$NSXManager/api/4.0/edges/" + $LBEdge + "/loadbalancer/config/applicationprofiles/" + $ApplicationProfile
$r = Invoke-WebRequest -URI $Uri -Method Get -Headers $head -ContentType "application/xml" -ErrorAction:Stop
[xml]$sxml = $r.Content
$OldCertificateId = $sxml.applicationProfile.clientSsl.serviceCertificate
$sxml.applicationProfile.clientSsl.serviceCertificate = $NewCertificateId[2]
$r = Invoke-WebRequest -Uri $Uri -Method Put -Headers $head -ContentType "application/xml" -Body $sxml.OuterXML -ErrorAction:Stop

##Delete old certificate from the Edge
$Uri = "https://$NSXManager/api/2.0/services/truststore/certificate/" + $OldCertificateId
$r = Invoke-WebRequest -URI $Uri -Method Delete -Headers $head -ContentType "application/xml" -ErrorAction:Stop

##Update vCloud Director with new certificates

$VcdSession = Connect-CIServer $Vcd -User $VcdAdmin -Password $VcdPassword

$Uri = "https://"+$Vcd+"/api/admin/extension/settings/general"
$head = @{"x-vcloud-authorization"=$VcdSession.SessionSecret} + @{"Accept"="application/*;version=33.0"}
$r = Invoke-WebRequest -URI $Uri -Method Get -Headers $head -ErrorAction:Stop
[xml]$sxml = $r.Content

$sxml.GeneralSettings.RestApiBaseUriPublicCertChain = $LBCertificate + $IssuerCert + $RootCert
$sxml.GeneralSettings.SystemExternalAddressPublicCertChain = $LBCertificate + $IssuerCert + $RootCert

$r = Invoke-WebRequest -URI $Uri -Method Put -Headers $head -ContentType "application/vnd.vmware.admin.generalSettings+xml" -Body $sxml.OuterXML -ErrorAction:Stop

A short Powershell script I wrote to demonstrate how to open vCloud Director VM console in the (Windows) Standalone VM Console.

$vm = get-CIVM 'CentOS7'
$vmrcPath = "C:\Program Files (x86)\VMware\VMware Remote Console\vmrc.exe"
$mks = $vm.ExtensionData.AcquireMksTicket()
$thumbprint = $mks.Ticket.Substring($mks.Ticket.get_Length()-61,59)
$parameter = "vmrc://$($mks.Host)/?thumbprint=$($thumbprint)&path=$($mks.Vmx)&websocket=wss://$($mks.Host)/($mks.Port);$($mks.Ticket)"
& "$vmrcPath" $parameter

The script should be pretty self explanatory. You need to have PowerCLI for vCloud Director installed and be logged in (as tenant or system admin).

$vm variable contains the VM for which you want to open the console. $vrmcPath is the path to the locally installed VMware Remote Console application. Next you need to acquire VM’s MKS ticket and with little manipulation send it to the application as a parameter.

vSphere 7 is now generally available and with it came also new vSAN update that introduces vSAN File Service. Cormac Hogan has good overview of the feature on his blog so definitely head there first to understand what it is.

I want to dive into the possibility of using vSAN File Service NFS in vCloud Director environments.

Let me start with current (April 2020) interop – vSphere 7 is not supported yet with vCloud Director. Which means vCenter Server 7 cannot be used as a target for IaaS services. But that is not an issue for the use case I want to discuss today.

vCloud Director deployment needs NFS storage for its Transfer Share directory. vCloud Director architecture consists of multiple cells (VM nodes) that scale out horizontally based on the size of the managed environment. The cells need shared database and shared Transfer Share directory to function properly. The Transfer Share must be NFS mount and is used mostly for OVF import/export operations related to vApp template and catalog management however the appliance deployment mode of vCloud Director also uses transfer share for storing appliance related info, ssh keys, responses.properties file for deployment of additional cells, and embedded database backups.

vCloud Director cell VMs are usually deployed in the management cluster and that can be separate vSphere 7 environment with vSAN. Can we (or should we) use vSAN NFS for vCloud Director Transfer Share?

Current practice is either to use external hardware storage NFS (NetApp) or to deploy Linux VM with large disk that acts as NFS server. The first approach is not always possible especially if you use vSAN only and have no external storage available. Then you have to go with the Linux VM approach. But not anymore.

vSAN File Service NFS has the following advantages:

• no external dependency on hardware storage or Linux VM
• easy to deploy and manage from UI or programmatically
• capacity management with quotas and thresholds
• high availability
• integrated lifecycle

The whole end-to-end deployment is indeed very simple, let me demonstrate the whole process:

1. Start with vSAN FS configuration in vSphere Cluster > Configure > vSAN > Services > File Service
2. Directly download vSAN File service agent (the lightweight container image OVA)
3. Configure vSAN domain and networking
   
   
4. Provide pool of IP addresses for the containers (I used 4 as I have 4 host management cluster).
   
5. After while you will see the agent containers deployed on each cluster node.
   
6. Now we can proceed with NFS share configuration. In the vSphere Cluster > Configure > vSAN > File Service Shares > ADD. We can define name, vSAN storage policy and quotas.
   
7. Enter IP addresses of your vCloud Director cells to grant them access to this share. Set permission Read/Write and make sure root squash is disabled.
   
8. Once the share is created, select the check box and copy its URL. Chose the NFSv4.1.
   In my case it looks like 192.168.110.181:/vsanfs/VCDTransferShare
9. Now you use the string in your vCloud Director cell deployment. I am using the vCloud Director appliance.
   
10. Once the cell is started we can see how the transfer share is mounted:
    
    Notice that while the mount IP address in /etc/fstab is the provided one 192.168.110.171, the actual one used is 192.168.110.172. This provides load balancing across all service node when more exports are created and when NFSv4.1 mount address is used.

It is imported to understand that although we have 4 vSAN FS agents deployed, the TransferShare will be provided via single container – in my case it is the one with IP 192.168.110.172. To find out on which host this particular container is running go to Cluster > Monitor > vSAN > Skyline Health > File Service > File Service Health.

So what happens if the host esx-03a.corp.local is unavailable? The share will fail over to another host. This took in my tests around 60-90 seconds. During that time the NFS share will not be accessible but the mount should persist and once the failover finishes it will become accessible again.

Notice that 192.168.110.172 is now served from esx-04.corp.local.

Also note that maintenance mode of the host will not vMotion the agent. It will just shut it down (and after while undeploy) and rely on the above mechanism to fail over the share to another agent. You should never treat the vSAN FS agents as regular VMs.

I am excited to see vSAN File Services as another piece of VMware technology that removes 3rd party dependencies from running vCloud Director (as was the case with NSX load balancing and PostgreSQL embedded database).

This post is password protected. You must visit the website and enter the password to continue reading.

NSX-T supports Role Based Access Control by integrating with VMware Identity Manager which provides access to 3rd party Identity Sources such as LDAP, AD, SAML2, etc.

When NSX-T version 2.3 is integrated with VIDM you would get a choice during the login which type of account you are going to provide (remote or local).

NSX-T version 2.4 no longer provides the option and will always default to the SAML source (VIDM). To force the login with local account provide this specific URL:

https://<NSX-T_FQDN/IP>/login.jsp?local=true

And it is time for another What’s New in (v)Cloud Director blog post. If you are not up to date you can find the older articles for versions 10, 9.7, 9.5 and 9.1 here.

Let us start with the “important” announcement – a name change. vCloud Director has been re-branded to VMware Cloud Director. Fortunately we keep the same (unofficial) acronym – VCD. The current version is 10.1 which might look like a small increase from the previous 10.0 but that is just marketing numbering so do not put too much emphasis on it and assess yourself if it is big release or not.

Interoperability

VMware has added support for NSX-T 3.0, but vSphere 7 support is missing. It is expected to come shortly in a major patch release. You can upgrade your management clusters and dedicated vCenter Servers, just not those that are backing Provider VDCs.

UI

As previously announced, no more Adobe Flex UI (cannot be even enabled with a secret switch). Shouldn’t be an issue however, as the HTML 5 UI has not only 99.9% parity but in fact is significantly better than Flex UI ever was. There are new features such as VM sizing and VM placement groups, advanced filtering, multiselect actions, badges, quick access to VM console or network cards, tasks and events in vApp details, …

The UI team is no longer in feature parity mode, they are in make-it-better mode and doing a great job as can be seen from the screenshots below.

Platform Security

Certificate validation is now required for VC/NSX endpoints and will be required for LDAP in the next major release as well. It means your endpoints either have to have publicly trusted certificates, or you have to upload their signing certificate, or you have to approve their certificate on the first use (when you add or edit such endpoint). To ease the transition from older vCloud Director releases, you can run cell-management-tool trust-infra-certs command after upgrade that will automatically retrieve and trust certificates of all connected VC/NSX endpoints. If you forget to run this command, Cloud Director will not be able talk to VC/NSX endpoints!

Appliance

While you still can use the Linux installer of Cloud Director with external database the appliance deployment factor has been again improved and there is less and less reasons not to use it especially for green field deployments.

The appliance API (on port 5480) has been enhanced to monitor status of services, to see which node is running active node of embedded database (useful for load balancing access to the database for external tools), monitor filesystem storage and trigger database node switchoever (planned failover) or promotion (after failure).

The appliance embedded appliance has for the first time automated failover functionality. It is disabled by default but you can enable it with the API.The Appliance UI has also been improved and provides some of the API functionality.

Messaging Bus

As you might now, Cloud Director has embedded messaging bus for inter-cell communication. In the past it was using ActiveMQ (ports 61611 and 61616). If the service provider wanted to use blocking tasks, notifications or vCloud API (or is it now VCloud API?!) extensions then an external RabbitMQ messaging bus had to be deployed. In the current release ActiveMQ has been replaced with Artemis and is also available externally for blocking tasks and notifications, so RabbitMQ is no longer needed for these two use cases. Additionally it can also be used by tenants.

Artemis uses MQTT communication protocol and the connection to it can be made via WebSocket session which is authenticated with regular Cloud Director authentication token mechanism.

Note that external RabbitMQ is still supported and still needed for vCloud API extensibility use case.

Networking

NSX-T integration has been enhanced with routed Org VDC networks (previous release supported NAT-routed) with BGP routing protocol. This feature currently requires dedicated Tier-0 Gateway for the tenant.
IPSec VPN is now in the UI (pre-shared key authentication and policy based are supported). IP Sets and Security Groups have been split with support for network objects that dynamically refer to all connected VMs.

The service provider can configure multiple NSX-T Edge Clusters and select which one will be used for a particular Tier-1 (Org VDC) Gateway. This enables separation of Tier-0 and Tier-1 Gateways to different Edge Clusters.

The NSX-V side of networking has also one new feature – you can now use Cross VDC networking within the same vCenter Server. This for example enables multi egress networks within single Org VDC for stretched cluster use case.

Storage

vSphere VM Encryption is now supported within Cloud Director. The encryption happens in the hypervisor which means the data is encrypted both in rest as well in flight. The encryption is set up via vCenter Server storage policies by enabling host based rules. A compatible external key management server must be deployed and connected to vCenter Server. It means the feature is fully in realm of the service provider and key management is not exposed to tenants.

Other Features

• Proxying of dedicated vCenter Servers (so called Centralized Point of Management – CPoM feature) was improved with extra stats, more proxies and browser extension to simplify the usage
• Support for VM (UI) and vApp (API only) live migration between Provider VDCs
• Due to UI upgrade to Clarity 2+ custom themes will have to be recompiled
• The provider can enable promiscuous mode and forged transmits on VXLAN backed Org VDC network (API only)
• Blocking tasks for OpenAPI tasks.
• Cloud Director 10.1 is the first release that enables automated NSX-V to NSX-T migration. More on that in a later blog post.

This is an updated blog post of the original vCloud Director 10: NSX-T Integration to include all VMware Cloud Director 10.1 related updates.

Intro

VMware Cloud Director relies on NSX network virtualization platform to provide on-demand creation and management of networks and networking services. NSX for vSphere has been supported for long time and vCloud Director allows most of its feature to be used by its tenants. However as VMware slowly shifts away from NSX for vSphere and pushes forward modern, fully rewritten NSX-T networking platform, I want to focus in this article on its integration with vCloud Director.

History

Let me start with highlighting that NSX-T is evolving very quickly. It means each release (now at version 3.0) adds major new functionality. Contrast that with NSX-V which is essentially feature complete in a sense that no major functionality change is happening there. The fast pace of NSX-T development is a challenge for any cloud management platforms as they have to play the catch up game.

The first release of vCloud Director that supported NSX-T was 9.5. It supported only NSX-T version 2.3 and the integration was very basic. All vCloud Director could do was to import NSX-T overlay logical segments (virtual networks) created manually by system administrator. These networks were imported into a specific tenant Org VDC as Org VDC networks.

The next version of vCloud Director – 9.7 supported only NSX-T 2.4 and from the feature perspective not much had changed. You could still only import networks. Under the hood the integration however used completely new set of NSX-T policy based APIs and there were some minor UI improvements in registering NSX-T Manager.

vCloud Director version 10 for the first time introduced on-demand creation of NSX-T based networks and network services. NSX-T version 2.5 was required.

The latest Cloud Director version 10.1 is extending NSX-T support with new features.

Note: Cloud Director 10.1.0 does not support NSX-T 3.0. That support will come in the next patch release (10.1.1).

NSX-T Primer

While I do not want to go too deep into the actual NSX-T architecture I fully expect that not all readers of this blog are fully familiar with NSX-T and how it differs from NSX-V. Let me quickly highlight major points that are relevant for topic of this blog post.

• NSX-T is vCenter Server independent, which means it scales independently from vCenter domain. NSX-T essentially communicates with ESXi hosts directly (they are called host transport nodes). The hosts must be prepared with NSX-T vibs that are incompatible with NSX-V which means a particular host cannot be used by NSX-V and NSX-T at the same time.
• Overlay virtual networks use Geneve encapsulation protocol which is incompatible with VXLAN. The concept of Controller cluster that keeps state and transport zone is very similar to NSX-V. The independence from VC mentioned in the previous point means vSphere distributed switch cannot be used, instead NSX-T brings its own N-VDS switch. It also means that there is concept of underlay (VLAN) networks managed by NSX-T. All overlay and underlay networks managed by NSX-T are called logical segments.
• Networking services (such as routing, NATing, firewalling, DNS, DHCP, VPN, load balancing) are provided by Tier-0 or Tier-1 Gateways that are functionally similar to NSX-V ESGs but are not instantiated in dedicated VMs. Instead they are services running on shared Edge Cluster. The meaning of Edge Cluster is very different from the usage in NSX-V context. Edge Cluster is not a vSphere cluster, instead it is cluster of Edge Transport Nodes where each Edge Node is VM or bare metal host.
• While T0 and T1 Gateways are similar they are not identical, and each has specific purpose or set of services it can offer. Distributed routing is implicitly provided by the platform unless a stateful networking service requires routing through single point. T1 GWs are usually connected to single T0 GW and that connection is managed automatically by NSX-T.
• Typically you would have one or small number of T0 GWs in ECMP mode providing North-south routing (concept of Provider Edge) and large number of T1 GWs connected to T0 GW, each for a different tenant to provide tenant networking (concept of Tenant Edge).

VMware Cloud Director Integration

As mentioned above since NSX-T is not vCenter Server dependent, it is attached to Cloud Director independently from VC.

(Geneve) network pool creation is the same as with VXLAN – you provide mapping to an existing NSX-T overlay transport zone.

Now you can create Provider VDC (PVDC) which is as usual mapped to a vSphere cluster or resource pool. A particular cluster used by PVDC must be prepared for NSX-V or NSX-T and all clusters must share the same NSX flavor. It means you cannot mix NSX-V clusters with NSX-T in the same PVDC. However you can easily share NSX-V and NSX-T in the same vCenter Server, you will then just have to create multiple PVDCs. Although NSX-T can span VCs, PVDC cannot – that limitation still remains. When creating NSX-T backed PVDC you will have to specify the Geneve Network Pool created in the previous step.

Within PVDC you can start creating Org VDCs for your tenants – no difference there.

Org VDCs without routable networks are not very useful. To remedy this we must create external networks and Org VDC Edge Gateways. Here the concept quite differs from NSX-V. Although you could deploy provider ECMP Edges with NSX-V as well (and I described here how to do so), it is mandatory with NSX-T. You will have to pre-create T0 GW in NSX-T Manager (ECMP active – active is recommended). This T0 GW will provide external networking access for your tenants and should be routable from the internet. Instead of just importing external network port group how you would do with NSX-V you will import the whole T0 GW in Cloud Director.

During the import you will also have to specify IP subnets and pools that the T0 GW can use for IP sub-allocation to tenants.

Once the external network exist you can create tenant Org VDC Edge Gateways. The service provider can pick specific existing NSX-T Edge Cluster for their placement.

T1 GWs are always deployed in Active x Standby configuration, the placement of active node is automated by NSX-T. The router interlink between T0 and T1 GWs is also created automatically by NSX-T. It is possible to disconnect Org VDC Edge GW from Tier-0 GW (this is for example used in NSX-V to NSX-T migration scenario).

During the Org VDC Edge Gateway the service providers also allocates range of IPs from the external network. Whereas with NSX-V these would actually be assigned to the Org VDC Edge Gateway uplink, this is not the case with NSX-T. Once they are actually used in a specific T1 NAT rule, NSX-T will automatically create static route on the T0 GW and start routing to the correct T1 GW.

Tenant Networks

There are four major types of NSX-T based Org VDC networks and three of them are available to be created via UI:

• Isolated: Layer 2 segment not connected to T1 GW. DHCP service is not available on this network (contrary to NSX-V implementation).
• Routed: Network that is connected to T1 GW. The default is NAT-routed which means its subnet is not announced to upstream T0 GW and only way to route to reach it from outside is to use DNAT rule on T1 GW from a allocated external IP address.
  Cloud Director version 10.1 introduces fully routed network more on it below.
• Imported: Existing NSX-T overlay logical segment can be imported (same as in VCD 9.7 or 9.5). Its routing/external connectivity must be managed outside of vCloud Director.
• In OpenAPI (POST /1.0.0/OrgVdcNetwork) you will find one more network type:  DIRECT_UPLINK. This is for a specific NFV use case. Such network is connected directly to T0 GW with external interface. Note this feature is not officially supported!

Note that only Isolated and routed networks can be created by tenants.

In direct connect use case it is desirable to announce routed Org VDC networks upstream so workloads are reachable directly without any NAT. This is possible in Cloud Director version 10.1, but requires dedicated Tier-0 GW for the particular tenant. The provider must create new Tier-0, connect it to tenant’s particular direct connect transit VLAN and then when deploying Org VDC Edge GW select Dedicate External Network switch.

Cloud Director will make sure that dedicated External Network Tier-0 GW is not accessible to any other Org VDC Edge Gateway.

Tenant can then configure on its Org VDC Edge GW BGP routing, which is in fact set by Cloud Director on the dedicated Tier-0 GW (while Tier-0 to Tier-1 routes are auto-plumbed by NSX).

Tenant Networking Services

Currently the following T1 GW networking services are available to tenants:

• Firewall (with IP Sets and Security Groups based on network objects)
• NAT
• DHCP (without binding and relay)
• DNS forwarding
• IPSec VPN: Policy based with pre shared key is supported.

All other services are currently not supported. This might be due to NSX-T not having them implemented yet, or Cloud Director not catching up yet. Expect big progress here with each new Cloud Director and NSX-T release.

Networking API

All NSX-T related features are available in the Cloud Director OpenAPI (CloudAPI). The pass through API approach that you might be familiar with from the Advanced Networking NSX-V implementation is not used!

Feature Comparison

I have summarized all Cloud Director networking features in the following table for quick comparison between NSX-V and NSX-T.

VMware Cloud Director as a cloud management solution is built on top of the underlying compute and networking platforms that virtualize the physical infrastructure. For the compute and storage part VMware vSphere was always used. However, the networking platform is more interesting. It all started with vShield Edge which was later rebranded to vCloud Networking and Security, Cisco Nexus 1000V was briefly an option, but currently NSX for vSphere (NSX-V) and NSX-T Data Center are supported.

VMware has announced the sunsetting of NSX-V (current end of general support is planned for (January 2022) and is fully committed going forward to the NSX-T Data Center flavor. The two NSX releases while similar are completely different products and there is no direct upgrade path from one to the other. So it is natural that all existing NSX-V users are asking how to migrate their environments to the NSX-T?

NSX-T Data Center Migration Coordinator has been available for some time but the way it works is quite destructive for Cloud Director and cannot be used in such environments.

Therefore with VMware Cloud Director 10.1 VMware is releasing compatible migration tool called VMware NSX Migration for VMware Cloud Director.

The philosophy of the tool is following:

• Enable granular migration of tenant workloads and networking at Org VDC granularity with minimum downtime from NSX-V backed Provider VDC (PVDC) to NSX-T backed PVDC.
• Check and allow migration of only supported networking features
• Evolve with new releases of NSX-T and Cloud Director

In other words, it is not in-place migration. The provider will need to stand up new NSX-T backed cluster(s) next to the NSX-V backed ones in the same vCenter Server. Also the current NSX-T feature set in Cloud Director is not equivalent to the NSX-V. Therefore there are networking features that cannot in principle be migrated. To see comparison of the NSX-V and NSX-T Cloud Director feature set see the table at the end of this blog post.

The service provider will thus need to evaluate what Org VDCs can be migrated today based on existing limitations and functionality. Start with the simple Org VDCs and as new releases are provided migrate the rest.

How does the tool work?

• It is Python based CLI tool that is installed and run by the system administrator. It uses public APIs to communicate with Cloud Director, NSX-T and vCenter Server to perform the migrations.
• The environment must be prepared is such way that there exists NSX-T backed PVDC in the same vCenter Server as the source NSX-V PVDC and that their external networks are at the infrastructure level equivalent as existing external IP addresses are seamlessly migrated as well.
• The service provider defines which source Org VDC (NSX-V backed) is going to be migrated and what is the target Provider VDC (NSX-T backed)
• The service provider must prepare dedicated NSX-T Edge Cluster whose sole purpose is to perform Layer-2 bridging of source and destination Org VDC networks. This Edge Cluster needs one node for each migrated network and must be deployed in the NSX-V prepared cluster as it will perform VXLAN port group to NSX-T Overlay (Geneve) Logical Segment bridging.
• When the tool is started, it will first discover the source Org VDC feature set and assess if there are any incompatible (unsupported) features. If so, the migration will be halted.
• Then it will create the target Org VDC and start cloning the network topology, establish bridging, disconnect target networks and run basic network checks to see if the bridges work properly. If not then roll-back is performed (bridges and target Org VDC are destroyed).
• In the next step the north / south networking flow will be reconfigured to flow through the target Org VDC. This is done by disconnecting the source networks from the gateway and reconnecting the target ones. During this step brief N/S network disruption is expected. Also notice that the source Org VDC Edge GW needs to be connected temporarily to a dummy network as NSX-V requires at least one connected interface on the Edge at all times.
• Each vApp is then vMotioned from the source Org VDC to the target one. As this is live vMotion no significant network/compute disruption is expected.
  
• Once the provider verifies the correct functionality of the target VDC she can manually trigger the cleanup step that migrates source catalogs, destroys bridges and the source Org VDC and renames the target Org VDC.
• Rinse and repeat for the other Org VDCs.

Please make sure you read the release notes and user guide for the list of supported solutions and features. The tool will rapidly evolve – short roadmap already includes pre-validation and roll-back features. You are also encouraged to provide early feedback to help VMware decide how the tool should evolve.

In VMware Cloud Director 10.1 (VCD) the organization users can subscribe to event and task push notifications which might be useful if the tenant needs to keep track of the activity in the cloud, connect CMDB or any other custom solution and does not want to permanently poll audit log via API.

Access to notifications was in the past only in the realm of service providers who needed to deploy RabbitMQ and connect their Cloud Director cells to it. They can still do so and in fact have to, if they need blocking taks or use VCD API extension (for example Container Service Extension, App Launch Pad or vRealize Operations Tenant App).

The new functionality is enabled by internal Artemis ActiveMQ bus that runs on every VCD cell. The MQTT client connects to the public https endpoint and uses WebSocket connnection to the bus. Authentication is provided via the JWT authentication token. The official documentation provides some detail here, but not enough to actually set this up.

Therefore I want to demonstrate here with very simple Python script how to set up connection and start utilizing this feature.

The Python 3 script leverages the Pyvcloud module (22.0 or newer is required) and Paho MQTT Python Client. Both can be installed simply with pip.

pip install pyvcloud paho-mqtt

In the example org admin credentials are used, which allows to subscription to all organization messages via publish/<org UUID>/* subscription string. It can also be used by system administrator while changing the subscription string to publish/*/*.

#!/usr/bin/python3

import paho.mqtt.client as mqtt
import json
import datetime
import pyvcloud.vcd.client
import pyvcloud.vcd.vdc

vcdHost = 'vcloud.fojta.com'
vcdPort = 443
path = "/messaging/mqtt"
logFile = 'vcd_log.log'

#org admin credentials
user = 'acmeadmin'
password = 'VMware1!'
org = 'acme'

credentials = pyvcloud.vcd.client.BasicLoginCredentials(user, org, password)
vcdClient = pyvcloud.vcd.client.Client(vcdHost+":"+str(vcdPort),None,True,logFile)
vcdClient.set_credentials(credentials)
accessToken = vcdClient.get_access_token()
headers = {"Authorization": "Bearer "+ accessToken}

if max(vcdClient.get_supported_versions_list()) < "34.0":
    exit('VMware Cloud Director 10.1 or newer is required')

org = vcdClient.get_org_list()
orgId = (org[0].attrib).get('id').split('org:',1)[1]

def on_message(client, userdata, message):
    event = message.payload.decode('utf-8')
    event = event.replace('\\','')
    eventPayload = event.split('payload":"',1)[1]
    eventPayload = eventPayload[:-2]
    event_json = json.loads(eventPayload)
    print(datetime.datetime.now())
    print (event_json)

# Enable for logging
# def on_log(client, userdata, level, buf):
#     print("log: ",buf)

client = mqtt.Client(client_id = "PythonMQTT",transport = "websockets")
client.ws_set_options(path = path, headers = headers)
client.tls_set_context(context=None)
# client.tls_insecure_set(True)
client.on_message=on_message
# client.on_log=on_log  #enable for logging
client.connect(host = vcdHost, port = vcdPort , keepalive = 60)
print('Connected')
client.subscribe("publish/"+orgId+"/*")
client.loop_forever()

Notice that the client needs to connect to the /messaging/mqtt path on the VCD endpoint and must provide valid JWT authentication token in the header. That rules some MQTT WebSocket clients that do not support custom headers (JavaScript).

The actual event is in JSON format with nested payload JSON providing the details. The code example prints the time when the event was received and just the nested payload JSON. The script runs forever until interrupted with Ctrl+C.

Note: The actual RabbitMQ extensibility configuration in VCD and the Non-blocking AMQP Notifications setting in the UI have no impact on this functionality and can be disabled if not used by the service provider.

Are you still on vCloud Director 9.7 (VCD) in multi-cell configuration? Then you are susceptible to Java Message Service (JMS) certificate expiration issue. Read on.

Background

In multi-cell set up VCD cells need to communicate between themselves. They use shared database but for much faster and efficient communication they also use internal ActiveMQ message bus. It is used for activity sharing and vCenter Server events notifications. If the message bus is dysfunctional it slows any operations almost to halt. For this particular certificate issue you will see in the logs similar message:

Could not accept connection from tcp://<primary-cell-IP:port> : javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown

In vCloud Director 9.7 the bus communication become encrypted in preparation for other use cases (read here). On upgrade or new deployment of each cell new certificate was issued by internal VCD_CA with 365 day duration. In vCloud Director 10.0 or VMware Cloud Director 10.1 the certificate is regenerated upon upgrade and its duration is extended to 3 years.

To find out the certificates expiry date, run the following command from any cell:


/opt/vmware/vcloud-director/bin/cell-management-tool jms-certificates -status


It will for every cell print out its JMS certificate details:

…

Cell with UUID fd0d2ca0-e357-4aae-9a3b-1c1c5d143d05 and IP 192.168.3.12 has jms certificate: [
[
Version: V3
Subject: CN=vcd-node2.vmware.local
Signature Algorithm: SHA256withRSA, OID = 1.2.840.113549.1.1.11

Key: Sun RSA public key, 2048 bits
modulus: 25783371233977292378120920630797680107189459997753528550984552091043966882929535251723319624716621953887829800012972122297123129787471789561707788386606748136996502359020718390547612728910865287660771295203595369999641232667311679842803649012857218342116800466886995068595408721695568494955018577009303299305701023577567636203030094089380853265887935886100721105614590849657516345704143859471289058449674598267919118170969835480316562172830266483561584406559147911791471716017535443255297063175552471807941727578887748405832530327303427058308095740913857590061680873666005329704917078933424596015255346720758701070463
public exponent: 65537
Validity: [From: Wed Jun 12 15:38:11 UTC 2019,
To: Thu Jun 11 15:38:11 UTC 2020]

…

Yes, this particular cell’s certificate will expire Jun 12 2020 – in less than two months!

The Fix

Set a calendar reminder and when the certificate expiration day is approaching run the following command.

/opt/vmware/vcloud-director/bin/cell-management-tool jms-certificates --certgen

Or upgrade to vCloud Director 10.0 or newer.

This short PowerCLI script will change the vSphere Distributed Switch default port group configuration to enable MAC learning policy. This means every port group on such switch inherits this configuration and will have MAC learning enabled unless specifically disabled.

For more information why would you need that read William’s Lam blog.

$vds = get-vdswitch 'DSwitch1'
$spec = New-Object VMware.Vim.VMwareDVSConfigSpec
$spec.DefaultPortConfig = New-Object VMware.Vim.VMwareDVSPortSetting
$spec.DefaultPortConfig.MacManagementPolicy = New-Object VMware.Vim.DVSMacManagementPolicy
$spec.DefaultPortConfig.MacManagementPolicy.MacLearningPolicy = New-Object VMware.Vim.DVSMacLearningPolicy

$spec.DefaultPortConfig.MacManagementPolicy.MacLearningPolicy.Enabled = $True
$spec.DefaultPortConfig.MacManagementPolicy.MacLearningPolicy.AllowUnicastFlooding = $True
$spec.DefaultPortConfig.MacManagementPolicy.MacLearningPolicy.Limit = 4000
$spec.DefaultPortConfig.MacManagementPolicy.MacLearningPolicy.LimitPolicy = "DROP"
$vds.ExtensionData.ReconfigureDvs_Task($spec)

Update 08/07/2020

In case you are using this approach for nested vSphere lab instead of the old promiscuous mode, make sure the vmk0 vmkernel port has a different MAC address than the MAC address of the vmnic of the nested ESXi host. This is because when the vmk0 is migrated to a different ESXi host uplink the vDS will not learn the MAC address on the new switch port as it conflicts with the assigned MAC on the first uplink port (same MAC cannot be learnt on two ports).

The vmkernel port MAC can be easily changed by editing /etc/vmware/esx.conf file.

There has been more and more interest lately among service providers in usage of VMware Cloud Foundation (VCF) as the underlying virtualization platform in their datacenter. VCF is getting more and more mature and offers automated lifecycle capabilities that service providers appreciate when operating infrastructure at scale.

I want to focus on the topic how would you design and deploy VMware Cloud Director (VCD) on top of VCF with a specific example. While there are whitepaper on this topic written they do not go into the nitty gritty detail. This should not be considered as prescribed architecture – just one way to skin a cat that should inspire you for your own design.

VCF 4.0 consists of a management domain – smaller infrastructure with usually one vSphere 7 cluster , NSX-T 3 and vRealize components (vRealize Suite Lifecycle Manager, vRealize Operations Manager, vRealize Log Insight). It is also used for deployment of management components for workload domains, which are separate vSphere 7+NSX-T 3 environments.

VCF has prescribed architecture based on VMware Validated Designs (VVD) how all the management components are deployed. Some are on VLAN backed networks but some are on overlay logical segments created in NSX-T (VVD calls them application virtual networks – AVN) and routed via NSX-T Edge Gateways. The following picture shows typical logical architecture of the management cluster which we will start with:

Reg-MGT and X-Reg-MGMT are overlay segments, rest are VLAN networks.
VC Mgmt … Management vCenter Server
VC Res … Workload domain (resource) vCenter Server
NSX Mgmt … Management NSX-T Managers (3x)
Res Mgmt … Workload domain (resource) NSX-T Managers (3x)
SDDC Mgr … SDDC Manager
Edge Nodes … NSX-T Edge Nodes VMs (2x) that provide resources for Tier-0, Tier-1 gateways and Load Balancer
vRLCM … vRealize Suite Lifecycle Manager
vROps … vRealize Operation Managers (two or more nodes)
vROps RC … vRealize Operation Remote Collectors (optional)
vRLI … vRealize Log Insight (two or more nodes)
WS1A … Workspace ONE Access (former VIDM, one or more nodes)

Now we are going to add VMware Cloud Director solution. I will focus on the following components:

• VCD cells
• RabbitMQ (needed for extensibility such as vROps Tenant App or Container Service Extension)
• vRealize Operations Tenant App (provides multitenant vROps view in VCD and Chargeback functionality)
• Usage Meter

I have followed these design principles:

• VCD solution will utilize overlay (AVN) networks
• leverage existing VCF infrastructure when it makes sense
• consider future scalability
• separate internet traffic from the management one

And here is the proposed design:

New overlay segment (AVN) called VCD DMZ has been added to separate the internet traffic. It is routed via separate Tier-1 GW but connected to the existing Tier-0. VCD cells (3 or more) have their primary (eth0) interface on this network with NSX-T Load balancer (running in its own Tier-1 similar to the vROps one). And finally vRealize Operations Tenant App VM.

Existing Reg-Mgmt is used for the secondary interface of VCD cells, Usage Meter VM and for vSAN File Services NFS share that VCD cell require.

And finally the cross region X-Reg-MGMT is utilized for RabbitMQ nodes (2 or more) in order to leverage existing vROps Load Balancer and get away with deploying additional one just for RabbitMQ.

Additional notes:

• VCF deploys two NSX-T Edge nodes in 2-node NSX-T Edge Cluster. These currently cannot easily be scaled out. Therefore I would recommend deploying additional Edge nodes in separate NSX-T Edge cluster (directly in NSX-T) for the DMZ Tier-1 gateway and VCD load balancer. This guarantees compute and networking resources especially for the load balancer that will perform SSL termination (might not apply if you chose to use different load balancer e.g. Avi). This will also add possibility to deploy separate Tier-0 for more N/S bandwidth.
• vSAN FS NFS deployment is described here. Do not forget to enable MAC learning on the Reg-MGMT NSX-T logical segment (via segment profile).
• Both Tier-1 gateways can provide north-south firewalling for additional security
• As all the incoming internet traffic to VCD goes over the VCD load balancer which provides Source NAT I have opted to have default route on the VCD cells on the management interface to get away with any need for static routes necessary to separate tenant and management traffic

Let me know in the comments if you plan VCD on VCF and if you are facing any challenges.

VMware Cloud Director 10.2 is adding network load balancing (LB) functionality in NSX-T backed Organization VDCs. It is not using the native NSX-T load balancer capabilities but instead relies on Avi Networks technology that was acquired by VMware about a year ago and since then rebranded to VMware NSX Advanced Load Balancer. I will call it Avi for short in this article.

The way Avi works is quite different from the way load balancing worked in NSX-V or NSX-T. Understanding the differences and Avi architecture is essential to properly use it in multitenant VCD environments.

I will focus only on the comparison with NSX-V LB as this is relevant for VCD (NSX-T legacy LB was never viable option for VCD environments).

In VCD in an NSX-V backed Org VDC the LB is running on Org VDC Edge Gateway (VM) that can have four different sizes (compact, large, quad large and extra large) and be in standalone or active / standby configuration. That Edge VM also needs to perform routing, NATing, firewalling, VPN, DHCP and DNS relay. Load balancer on a stick is not an option with NSX-V in VCD. The LB VIP must be an IP assigned to one of external or internally attached network interfaces of the Org VDC Edge GW.

Enabling load balancing on an Org VDC Edge GW in such case is easy as the resource is already there. 

In the case of Avi LB the load balancing is performed by external (dedicated to load balancing) components which adds more flexibility, scalability and features but also means more complexity. Let’s dive into it.

You can look at Avi as another separate platform solution similar to vSphere or NSX – where vSphere is responsible for compute and storage, NSX for routing, switching and security, Avi is now responsible for load balancing.

Picture is worth thousand words, so let me put this diagram here first and then dig deeper (click for larger version).

Control Path

You start by deploying Avi controller cluster (highly available 3 nodes) which integrates with vSphere (to use for compute/storage) and NSX-T (for routing LB data and control plane traffic). The controllers would sit somewhere in your management infrastructure next to all other management solutions.

The integration is done by setting up so called NSX-T Cloud in Avi where you define vCenter Server (only one is supported per NSX-T Cloud) and NSX-T Manager endpoints, NSX-T overlay transport zone (with 1:1 relationship between TZ and NSX-T Cloud definition). Those would be your tenant/workload VC/NSX-T.

You must also point to pre-created management network segment that will be used to connect all load balancing engines (more on them later) so they can communicate with the controllers for management and control traffic. To do so, in NSX-T you would set up dedicated Tier-1 (Avi Management) GW with the Avi Management segment connected and DHCP enabled. The expectation is the Tier-1 GW would be able through Tier-0 to reach the Avi Controllers.

Data Path

Avi Service Engines (SE) are VM resources to perform the load balancing. They are similar to NSX-T Edge Nodes in a sense that the load balancing virtual services can be placed on any SE node based on capacity or reservations (as Tier-1 GW can be placed on any Edge Node). Per se there is no strict relationship between tenant’s LB and SE node. SE node can be shared across Org VDC Edge GWs or even tenants. SE node is a VM with up to 10 network interfaces (NICs). One NIC is always needed for the management/control traffic (blue network). The rest (9) are used to connect to the Org VDC Edge GW (Tier-1 GW) via a Service Network logical segment (yellow and orange). The service networks are created by VCD when you enable load balancing service on the Org VDC Edge GW together with DHCP service to provide IP addresses for the attached SEs. It will by default get 192.168.255.0/25 subnet, but the system admin can change it, if it clashes with existing Org VDC networks. Service Engines run each service interface in a different VRF context so there is no worry about IP conflicts or even cross tenant communication.

When a load balancing pool and virtual service is configured by the tenant Avi will automatically pick a Service Engine to instantiate the LB service. It might even need to first deploy (automatically) an SE node if there is no existing capacity. When SE is assigned Avi will configure static route (/32) on the Org VDC Edge GW pointing the virtual service VIP (virtual IP) to the service engine IP address (from the tenant’s LB service network).

Note: The VIP contrary to NSX-V LB can be almost any arbitrary IP address. It can be routable external IP address allocated to the Org VDC Edge GW or any non-externally routed address but it cannot clash with any existing Org VDC networks. or with the LB service network. If you use an external Org VDC Edge GW allocated IP address you cannot use the address for anything else (e.g. SNAT or DNAT). That’s the way NSX-T works (no NAT and static routing at the same time). So for example if you want to use public address 1.2.3.4 for LB on port 80 but at the same time use it for SNAT, use an internal IP for the LB (e.g. 172.31.255.100) and create DNAT port forwarding rule to it (1.2.3.4:80 to 172.31.255.100:80).

Service Engine Groups

With the basics out of the way let’s discuss how can service provider manage the load balancing quality of service – performance, capacity and availability. This is done via Service Engine Groups (SEG).

SEGs are (today) configured directly in Avi Controller and imported into VCD. They specify SE node sizing (CPU, RAM, storage), bandwidth restrictions, virtual services maximums per node and availability mode.

The availability mode needs more explanation. Avi supports four availability modes:
A/S … legacy (only two nodes are deployed), service is active only on one node at a time and stand by on the other, no scale out support (service across nodes), very fast failover

A/A … elastic, service is active on at least two SEs, session info is proactively replicated, very fast failover

N+M … elastic, N is number of SE nodes service is scaled over, M is a buffer in number of failures the group can sustain, slow failover (due to controller need to re-assign services), but efficient SE utilization

N+0 … same as N+M but no buffer, the controller will deploy new SE nodes when failure occurs. The most efficient use of resources but the slowest failover time.

The base Avi licensing supports only legacy A/S high availability mode. For best availability and performance usage of elastic A/A is recommended.

As mentioned Service Engine Groups are imported into VCD where the system administrator makes a decision whether SEG is going to be dedicated (SE nodes from that group will be attached to only one Org VDC Edge GW) or shared.

Then when load balancing is enabled on a particular Org VDC Edge GW, the service provider assigns one or more SEGs to it together with capacity reservation and maximum in terms of virtual services for the particular Org VDC Edge GW.

Use case examples:

• A/S dedicated SEG for each tenant / Org VDC Edge GW. Avi will create two SE nodes for each LB enabled Org VDC Edge GW and will provide similar service as LB on NSX-V backed Org VDC Edge GW did. Does not require additional licensing but SEG must be pre-created for each tenant / Org VDC Edge GW.
• A/A elastic shared across all tenants. Avi will create pool of SE nodes that are going to be shared. Only one SEG is created. Capacity allocation is managed in VCD, Avi elastically deploys and undeploys SE nodes based on actual usage (the usage is measured in number of virtual services, not actual throughput or request per seconds).

Service Engine Node Placement

The service engine nodes are deployed by Avi into the (single) vCenter Server associated with the NSX Cloud and they live outside of VMware Cloud Director management. The placement is defined in the service engine group definition (you must use Avi 20.1.2 or newer). You can select vCenter Server folder and limit the scope of deployment to list of ESXi hosts and datastores. Avi has no understanding of vSphere host, and datastore clusters or resource pools. Avi will also not configure any DRS anti-affinity for the deployed nodes (but you can do so post-deployment).

Conclusion

The whole Avi deployment process for the system admin is described in detail here. The guide in the link refers to general Avi deployment of NSX-T Cloud, however for VCD deployment you would just stop before the step Creating Virtual Service as that would be done from VCD by the tenant.

Avi licensing is basic or enterprise and is set at Avi Controller cluster level. So it is possible to mix both licenses for two tier LB service by deploying two Avi Controller cluster instances and associating each with a different NSX-T transport zone (two vSphere clusters or Provider VDCs).

The feature differences between basic and enterprise editions are quite extensive and complex. Besides Service Engine high availability modes the other important difference is access to metrics, amount of application types, health monitors and pool selection algorithms.

The Avi usage metering for licensing purposes is currently done via Python script that is ran at the Avi Controller to measure Service Engine total  high mark vCPU usage during a given period and must be reported manually. The basic license is included for free with VCPP NSX usage and is capped to 1 vCPU per 640 GB reported vRAM of NSX base usage.