vCloud Network Isolation (VCDNI or VCNI) is legacy mechanism to create overlay logical networks independently from physical networking underlay. It was originally used in VMware vCenter Lab Manager (where it was known as Cross Host Fencing). vCloud Director offers it as one of many mechanisms for creation of logical networks (next to VXLAN, VLAN and port group backings). VCDNI uses VMware proprietary MAC-in-MAC encapsulation done by vCloud Agent running in ESXi host vmkernel.

It has been for some time superseded by VXLAN technology which is much more scalable, provides better performance and is industry standard technology. VXLAN network pools have been available in vCloud Director since version 5.1.

VCDNI is consumed by manual creation of a vCloud Network Isolation backed Network Pool that is mapped to an underlay VLAN network with up to 1000 logical networks for each pool (VLAN).

As a deprecated and obsolete technology it is no longer supported in vSphere 6.5 and vCloud Director 8.20 is the last release that will support such network pools. vCloud Director 8.20 also provides simple mechanism to perform low-disruption migrations for Org VDC and vApp networks to VXLAN backed networks. Such migration must be done before upgrade to vSphere 6.5 (see more in KB 2148381).

The migration can be performed via UI or API by system administrator with Org VDC granularity.

Migration via UI

1. For an Org VDC using VCDNI network pool open in the System tab – Manager & Monitor, Org VDC properties (note that doing the same from Org tab will not work).
   
2. Go to Network Pool & Services tab and change VCDNI backed network pool to VXLAN backed one and click OK.
   
3. Again open Network Pool & Services tab of the Org VDC. Migrate to VXLAN button will now appear.
   
4. Click the button, confirm the message and start the migration.
   
5. After while the Org VDC status will change from busy to ready and the migration is finished. Details (and possible errors) can be reviewed in the Recent Tasks of the Audit Log.
   

Migration with vCloud API

Org VDC network migration is triggered by single API POST call at the Org VDC level.

POST /api/admin/vdc/<org VDC UUID>/migrateVcdniToVxlan
Content Type: application/vnd.vmware.admin.vdcnitovxlanmigration+xml

The Process

The following happens in the background when migration is triggered for each VCDNI backed network in an Org VDC:

1. ‘Dummy’ VXLAN logical switch is created
2. All VMs connected to VCDNI network are reconnected to the new VXLAN logical switch
3. Edge Gateways connected to VCDNI network are connected to the new VXLAN logical switch
4. Org VDC/vApp network backing is changed in vCloud DB to use the new VXLAN logical switch
5. Original VCDNI port group is deleted

Small network disruption is expected during VM and Edge Gateway reconnections. The following Recent Tasks picture from vSphere Client shows what is happening at vCenter Server level and how much time each task could take. In the example there was one Org VDC network and one vApp network migrated with VM1 and Edge Gateway ACME-GW2 involved.

vCloud Director architecture consist of multiple cells that share common database. The upgrade process involves shutting down services on all cells, upgrading them, upgrading the database and starting the cells. In large environments where there are three or more cells this can be quite labor intensive.

vCloud Director 8.20 brings new feature – an orchestrated upgrade. All cells and vCloud database can be upgraded with a single command from the primary cell VM. This brings two advantages. Simplicity – it is no longer needed to login to each cell VM, upload binaries and execute upgrade process manually. Availability – downtime during the upgrade maintenance window is reduced.

Prerequisites

Set up ssh private key login from the primary cell to all other cells in the vCloud Director instance for user vcloud.

1. On the primary cell generate private/public key (with no passphrase):

   ssh-keygen -t rsa -f $VCLOUD_HOME/etc/id_rsa
   chown vcloud:vcloud $VCLOUD_HOME/etc/id_rsa
   chmod 600 /opt/vmware/vcloud-director/etc/id_rsa 

2. Copy public key to each additional cell in the instance to authorized_keys file. This can be done with one line command ran from the primary cell or with this ssh-copy-id. Use IP/FQDN it is registered with in VCD

   cat $VCLOUD_HOME/etc/id_rsa.pub | ssh root@<cell-IP> “mkdir -p ~/.ssh && cat >> ~/.ssh/authorized_keys” 

3. Verify that login with private key works for each secondary cell in the environment

   sudo -u vcloud ssh -i $VCLOUD_HOME/etc/id_rsa root@<cell IP/FQDN>

Multi-cell Installation

Upload vCloud Director binary to the primary cell and make it executable. Execute the file with ––private-key-path option pointing to the private key.

/root/vmware-vcloud-director-distribution-8.20.0-5070903.bin –private-key-path $VCLOUD_HOME/etc/id_rsa

Optionally a maintenance cell can be specified with –maintenance-cell option.

For troubleshooting, the upgrade log is located on the primary cell in  $VCLOUD_HOME/logs/upgrade-<date and time>.log

For no-prompt execution you can add –unattended-upgrade option.

Workflow

This is the workflow that is automatically executed:

1. Quiesce, shutdown and upgrade of the primary cell. Does not start the cell.
2. If maintenance cell was specified, it is put into maintenance mode.
3. Quiescing and shut down of all the other cells.
4. Upgrade of the vCloud Database (a prompt for backup)
5. Upgrade and start of all other cells (except the maintenance cell)
6. If maintenance cell was specified, it is upgraded and started.
7. Start of the primary cell

What is the difference between a quiesced cell and a cell in the maintenance mode?

Quiesced cell:

• finishes existing long running operations
• answers to new requests and queues them
• does not dequeue any operations (they will stay in the queue)
• VC lister keeps running
• Console proxy keeps running

Cell in maintenance mode

• waits for finish of long running but fails all queued operations
• answer to most requests with HTTP Error code 504 (unavailable)
• still issues auth token for /api/sessions login requests
• No VC listener
• No Console proxy

Interoperability with vCloud Availability

vCloud Availability uses Cloud Proxies to terminate replication tunnels from the internet. Cloud Proxies are essentially stripped down vCloud Director cells and are therefore treated as regular cells during the orchestrated upgrade.

Quiesced Cloud Proxy has no impact on replication operations and traffic. Cloud Proxy in the maintenance mode still preserves existing replications however new replications cannot be established.

2/27/2017: Multiple edits based on feedback from engineering. Thank you Matthew Frost!

NSX Distributed Firewall (DFW) is the most popular feature of NSX which enables microsegmenation of networks with vNIC level firewalls in hypervisor. For real technical deep dive into the feature I recommend reading Wade Holmes free e-book available here.

vCloud Director 8.20 provides this feature to tenants with brand new HTML5 UI and API. It is managed at Org VDC level from Manage Firewall link. This opens new tab with the new user interface.

Firewall Comparison

vCloud Director now offers three different firewalls types for tenants, which might be confusing. So let me quickly compare them.

The picture above shows two Org VDCs each with different network topologies. Org VDC 1 is using Org VDC Edge Gateway that provides firewalling as well as other networking services (load balancing, VPNs, NAT, routing, etc.). It has also brand new UI and Network API. Firewalling at this level is enforced only on packets routed through the Edge Gateway.

One level below we see vApps with vApp Edges. These provide routing, firewalling and NAT between routed vApp Network and Org VDC network. There is no change in firewall capability of vApp Edge in vCloud Director 8.20 and old flash UI and vCloud API can be used for its configuration. Firewalling at vApp Edge level is enforced only on packets routed between Org VDC and vApp networks.

Distributed firewall is applied at the vNIC level of virtual machines. It means it can inspect every packet and frame coming and leaving VM and is therefore completely independent from the network topology and can be used for microsegmentation of layer 2 network. Both layer 3 and layer 2 rules can be created.

Obviously all three firewall types can be combined and used together.

Managing Access to Distributed Firewall

There are four new access rights related to DFW in vCloud Director.

• Manage Firewall
• Configure Distributed Firewall Rules
• View Distributed Firewall Rules
• Enable / Disable Distributed Firewall

The last right is by default available only to system administrators, therefore the provider can control which tenant can and cannot use DFW and it can thus be offered as a value added service. The provider can either enable DFW selectively for specific Org VDCs or alternatively grant Enable/Disable Distributed Firewall right to a specific organization via API. The tenant can enable DFW by himself.

Distributed Firewall under the Hood

Each tenant is given a section in the NSX firewall table and can only apply rules to VMs and Edge Gateways in his domain. There is one section for each Org VDC that has DFW enabled and it is created always on top. As tenants could have overlapping IPs all rules in the section are scoped to a security group with dynamic membership of tenant Org VDC resource pools and thus will be applied only to VMs in the Org VDC.

Tenants can create layer 3 (IP based) or layer 2 (MAC based) rules while using the following objects when defining them:

•  IP address, IP/MAC sets
• Virtual Machine
• Org VDC Network
• Org VDC

Note that using L3 non-IP based rules requires NSX to learn IP address(es) of the guest VM. One of the following mechanism must be enabled:

• VMware Tools installed in guest VM
• DHCP Snooping IP Detection Type
• ARP Snooping IP Detection Type

IP Detection Type is configured in NSX at Cluster Level in Host Preparation tab.

Scope for each rule can be defined in Applied To column. As mentioned before by default it is set to the Org VDC, however tenant can further limit the scope of the rule to a particular VM, or Org VDC network (note that vApp network cannot be used). It is also possible to apply the rule to Org VDC Edge Gateway, in such case the rule is actually created and enforced on the Edge Gateway as pre-rule which has precedence over all other firewall rules defined at that Edge Gateway.

Tenant can enable logging of a specific firewall rule with API by editing <rule … logged=”true|false”> element. NSX then logs the first session packet matching the rule to ESXi host log with tenant specific tag (Org VDC UUID subset string). The provider can then filter such logs and forward them to tenants with its own syslog solution.

vCloud Director 8.20 introduces the possibility to create granular roles at tenant and system level. This is important for service providers who want to differentiate which tenants have access to specific features (for example advanced networking services). This also gives opportunity to tenants to create their own roles that correspond to their team structure (e.g. network administrator). And lastly, system administrator can create additional roles in system context with access to subset of features.

A role is a set of rights which can be assigned to a user or a group. There are many new rights in vCloud Director 8.20. A few examples:

Access to Distributed Firewall:

• Enable / Disable Distributed Firewall

Gateway Advanced Services

• Configure IPSEC VPN
• Configure Load Balancer
• Configure BGP Routing
• Configure OSPF Routing
• Configure SSL VPN
• Configure Firewall
• Configure DHCP
• Configure NAT
• Configure L2 VPN
• Configure Static Routing

Or system level rights like:

Host

• Upgrade Host
• Repair Host
• Migrate Host VMs
• Open a Host in vSphere
• Enable / Disable a Host
• Prepare / Unprepare a Host
• View Host

Prior vCloud Director 8.20

• Only global roles could be created by system administrator next to handful of predefined roles (vApp Author, Organization Administrator, …).
• Every organization would have access to the global and predefined roles.
• Organization administrator could assign the roles to organization users.
• Service provider could not differentiate access to features among different tenants.
• There was only one system administrator role with access to everything.

vCloud Director 8.20

• Roles are no longer global, but instead are organization specific.
• Former global and predefined roles become role templates.
• Service provider can create new role templates.
• Role templates are used to instantiate organization specific roles.
• Service provider can selectively grant rights to specific organizations.
• Organization administrator can create own organization specific roles from subset of granted rights.
• New roles can be created in the system context from subset of system administrator rights.

The transition from pre-vCloud Director 8.20 role management happens during upgrade to 8.20. Existing roles are transferred to role templates and each organization gets its own roles instantiation based on the role templates. The UI has changed and now includes Organization column and filter. A new System organization is added with default System Administrator role.

Tenant Rights and Role Management

When a new organization is created it will have access to all rights that are used in role templates. System administrator can grant additional rights to the organization with vCloud API only:

GET /api/admin … get references to all rights in VCD instance

GET /api/org/<org-id>/rights … get references to all rights in the organization

PUT /api/org/<org-id>/rights … edit rights in the organization

System administrator or Organization Administrator can create new roles in its organization with vCloud API only:

POST /admin/org/<org-id>/roles

Note: While system administrator can edit tenant roles in the UI, editing of a role based on role template would change the role template and thus change it for all organizations (more below).

How to Create Global Role

The UI no longer allows creation of global roles, only organization specific roles can be created that way.

However, there is a way to create global role (actually role template) with the legacy API (e.g. version 9.0, 20.0 but not 27.0). Here is an example:

POST /api/admin/roles
Header:
Accept: application/*;version=9.0
Content-Type: application/vnd.vmware.admin.role+xml

Body:

<?xml version="1.0" encoding="UTF-8"?>
<Role xmlns="http://www.vmware.com/vcloud/v1.5" name="New Global Role">
	<Description>My new global role</Description>
	<RightReferences>
		<RightReference href="https://vcloud.fojta.com/api/admin/right/0b8c8cd2-5af9-32ad-a0bd-dc356503a552" name="General: Administrator View" type="application/vnd.vmware.admin.right+xml"/>
		<RightReference href="https://vcloud.fojta.com/api/admin/right/5e579955-fe9d-3f0b-bc6b-a3da4db328f1" name="Group / User: View" type="application/vnd.vmware.admin.right+xml"/>
		<RightReference href="https://vcloud.fojta.com/api/admin/right/2cd2d9d7-262c-34f8-8bee-fd92f422cc2c" name="General: Administrator Control" type="application/vnd.vmware.admin.right+xml"/>
	</RightReferences>
</Role>

Note: Using above API call with API version 27.0 would create the role in the system organization.

How to Edit Global Roles?

Again with legacy vCloud API we can list all global (template) and system organization roles:

GET /api/admin
Header:
Accept: application/*;version=9.0
Response:
<RoleReferences>
	...
	<RoleReference href="https://vcloud.fojta.com/api/admin/role/75717adf-8700-419e-afe1-d5e2ea3b0bd6" name="User Admin" type="application/vnd.vmware.admin.role+xml"/>
	...
</RoleReferences>

After finding the right role reference we can delete the role template with the following call:

DELETE /api/admin/role/<role-id>
Header:
Accept: application/*;version=9.0

Addition and removal of rights from a role template:

• In UI add/remove the right from the role which is based on role template from any organization.
• To add a new right, the organization needs to have access to the right. If it does not have, add it first with the API calls mentioned above.
• Adding or removing rights to a role based on role template will affect all other organizations.
  • Adding right: other organizations will see the new right if their instance of role template has been granted the right. If the organization did not have access to the right, the right will not be added!
  • Removing right: in other organizations the right will be removed from the role based on the role template

The post was written with kind support of John Kilroy.

Another vCloud Architecture Toolkit whitepaper that I authored was published on the vCAT SP website – it discusses how to architect vCloud Availability solution in large production scenarios.

It is based on real live deployments and includes the following chapters:

• Introduction
• Use Cases
  • Disaster Recovery
  • Migration
• vCloud Availability Architecture Design Overview
  • vCloud Availability Architecture
  • Network Flows
  • Conceptual Architecture
• vCloud Availability Management Components
  • Logical Architecture
  • vCloud Availability Portal
  • Cloud Proxy
  • RabbitMQ
  • Cassandra Database
  • VMware Platform Services Controller
  • vSphere Replication Cloud Service
  • vSphere Replication Manager
  • vSphere Replication Servers
  • ESXi Hosts
  • vCloud Availability Metering
  • vRealize Orchestrator
  • Management Component Resiliency Considerations
• vCloud Director Configuration
  • User Roles
  • Tenant Limits and Leases
  • Organization Virtual Data Center
  • Network Management
  • Storage Management
  • vApps and Virtual Machines
• Billing
• vRealize Orchestrator Configuration
  • On-Premises Deployment
  • In-the-Cloud Deployment
  • Provider Deployment
  • Failover Orchestration
• Monitoring
  • Component Monitoring
  • VM Replication Monitoring
  • Backup Strategy
• Appendix A – Port Requirements / Firewall Rules
• Appendix B – Glossary
• Appendix C – Maximums
• Appendix D – Reference Documents
• Appendix E – Tenant API Structure
• Appendix F – Undocumented HybridSettings vCloud API
• Appendix G – Monitoring

Download from the vCAT-SP website: https://www.vmware.com/solutions/cloud-computing/vcat-sp.html or direct link to pdf.

Just a quick post to recap my experience with usage of Client Integration Plugin in vCloud Director and its compatibility with current browsers.

Client Integration Plugin (CIP) is needed in vCloud Director 8.20 only for import and export of templates and media images. It is not bundled in vCloud Director binaries anymore, instead the user is redirected to KB article with downloads for particular browser/OS combination.

If you downloaded CIP and you still cannot get through to the catalog download/upload dialog, here are some steps to try:

• Cleanup your installed Client Integration plugins. Reboot your PC and install the one you need.
• Google Chrome seems to be working the best. It requires CIP version 5.6 as opposed to the other browser that need version 6.2.
• The newest Mozilla Firefox needs a tweak. Open URL: about:config and accept loss of warranty ;-). Create new boolean type row by right-clicking in an empty space, name it  plugin.load_flash_only and assign value false. Reopen Firefox.
• Alternatively, you can download ESR release of Firefox which works out of the box from here.
• For Internet Explorer 11 you need to enable compatibility mode and add vCloud Director URL to the trusted sites.
• While Edge browser is supported for vCloud Director, it does not support any plugins and will not work with CIP.

Note: VMware is working hard on removing these limitations in the future releases.

In March 2017 VMware released vCenter Chargeback Manager 2.7.3. The main reason why to upgrade to the new release is that it adds full support for vCloud Director 8.20.

So what is new? From the security standpoint TLS 1.2 is now supported and also Java and Tomcat are updated with the latest security patches. There is a new way of collection of network resources consumption. It the pre-8.20 releases of vCloud Director, the configuration of network services was stored in vCloud Director audit table and available via vCloud API used by Chargeback Cloud collector. Chargeback vShield collector was used only for collection of external network transfer using vShield API against vCNS/NSX Manager.

When advanced networking services are configured in vCloud Director 8.20, these changes are no longer tracked by vCloud Director audit table. Chargeback thus relies on NSX API to collect the status of the services. This also means, that if these services are enabled directly in NSX, Chargeback will track them.

The following networking services are tracked by Chargeback:

New networking services:

• Dynamic routing (OSPF/BGP)
• L2 VPN
• SSL VPN
• Distributed firewall (enabled at Org VDC level)

Legacy networking services:

• DHCP
• Edge Firewall
• Edge Gateway High Availability
• NAT
• Static routing
• Enabled IPSec VPN Tunnel Count
• Load Balancer

Metering of external networks is unchanged.

In the past I have wrote how to import running VM to vCloud Director: here and here. Today, I will describe how to export running VM.

Let us first discuss why you would do it. There are several use cases:

• you want to manage the VM by a different Cloud Management Platform (for example vRealize Automation)
• you want to migrate (utilizing cross VC vMotion) to a different vCloud Director
• you want to move the VM to a different Org VDC

Currently (as of vCloud Director 8.20) there is no direct way to export running VM. So the procedure I will describe below is kind of a workaround. But let me first describe, what it means for a VM to be managed by vCloud Director.

• VM is ‘marked’ at vSphere level with Custom Attributes and ManagedBy extension.
• VM has its cloud_uuid stored in its configuration parameters
• VM has assigned CPU/RAM reservations and limits depending on its Org VDC allocation model
• VM is running in resource pool and folder based on its Org VDC/vApp
• VM has a name that includes its UUID
• vCloud Director is tracking the VM in vSphere inventory even if it changes its name, location and MoRef ID.
• vCloud Director is reserving VMs IP and MAC addresses in its IPAM.
• vCloud Director is counting VMs resources to its Org VDC allocation.

In order to remove the VM from vCloud Director management we must take care of all these points.

Here comes the procedure I came up with. It can obviously be automated with vSphere and vCloud APIs if needed for export of large amount of VMs.

First we clean up VM at vSphere level in vCenter Server:

1. Move VM outside the vCloud Director managed Resource Pool (this is to avoid auto-import of the VM)
2. If necessary reconnect VM to network not managed by vCloud Director (this is for situations where the vApp or Org VDC network is going to be deleted later). Obviously as the VM is running, the new network should provide equivalent connectivity as the original one (either with L2 bridging or routing).
3. Remove cloud-uuid from the VM configuration parameters. This can be done on running VM with PowerCLI:
   (Get-AdvancedSetting -entity $vm -Name cloud.uuid)|Remove-AdvancedSetting
   
4. Remove vCloud Director related Custom Attribute values (in my case VCD_fojta_01). Do not remove the whole Custom Attribute, just the value.
   
5. Remove ManagedBy extension from the VM. The easiest to do is leveraging PowerCLI script attached to KB 2032366. You will see the VM icon has changed after the extension has been removed.
   \ManagedBy.ps1 -Cmd Clear -VMs $vm
6. Remove VM resource reservation and limits (if applicable).
   
7. Rename VM (to get rid of UUID in the name).
   Now we need to take care of removing VM in vCloud Director, however even though we removed its cloud-uuid vCloud Director still sees the VM through its vCenter MoRef ID. And we cannot change MoRef ID of running VM. So here comes the workaround:
8. Temporarily remove access to the VM for vCloud Director service account (account configured in vCloud Director for the particular vCenter Server). To do so, you assign No Access role permission on the VM object for the service account user.
   
9. Now the VM has become invisible for vCloud Director and we can clean it up in vCloud Director. First Force-Stop its vApp.
   
10. Now Delete the vApp. Ignore the error.
    
11. Purge the vApp from stranded items with Force Delete.
12. Now we can remove the temporary no access role permission from step #8.
13. Clean up the vApp folder in vCenter Server if it was not removed.

Quite lengthy and not pretty procedure, however we did not do anything unsupported (like editing vCloud database) and the VM has been properly removed from vCloud Director inventory.

Last comment is about VM MAC addresses. If the VM was created in vCloud Director it will have MAC addresses from the vCloud Director range based on VCD installation ID. Have that in mind when moving VM around as duplicate MACs could be generated.

vCloud Director 8.20 allows deployment of Org VDC Edge Gateways in 4 different form factors from Compact to X-Large where each provides different level of performance and consumes different amount of resources.

As these Edge Gateways are deployed by NSX Manager which allows setting custom reservations for CPU and RAM via an API call PUT https://<NSXManager>/api/4.0/edgePublish/tuningConfiguration, it is also possible in vCloud Director to set custom reservations.

Why would you change the default reservations? Reservations at VM (Edge) level reserve the resources for itself which means no other VM can utilize them in case they are unused. They basically guarantee certain level of service that the VM (Edge) from performance perspective will always deliver. In service provider environments oversubscription provides ROI benefits and if the service provider can guarantee enough resources at cluster scale, than the VM level reservations can be set lower if at all.

This can be accomplished by tuning the networking.gatewayMemoryReservationMultiplier and networking.gatewayCpuReservationMultiplier settings via cell-management-tool from vCloud Director cell. By default the CPU multiplier is set to 64 MHz per vCPU and the Memory multiplier to 0.5.

By default Edge Gateways will be deployed with the following reservation settings:

The following command will change memory multiplier to 10%:

/opt/vmware/vcloud-director/bin/cell-management-tool manage-config -n networking.gatewayMemoryReservationMultiplier -v 0.1

Note: The new reservation settings are applicable only for newly deployed Org VDC Edge Gateways. Redeploying existing edges will not change their reservation settings. You must either use NSX API to do so, or modify Org VDC Edge Gateway form factor (e.g. change Large to Compact and then back to Large) which is not so elegant as it will basically redeploy the Edge twice.

Also note that NSX 6.2 and NSX 6.3 have different sizing of Quad Large Edge. vCloud Director 8.20 is by default set for the NSX 6.3 size which is 2 GB RAM (as opposed to NSX 6.2 value of 1 GB RAM). It is possible to change the default for the reservation calculation by editing networking.full4GatewayMemoryMb setting to value ‘1024’

Besides traditional roles (Org Administrator, vApp Author, …) there is already for some time (as of vCloud Director 8.0) the possibility to assign to vCloud Director organization users a role called ‘Defer to Identity Provider’.

I am going to show how this role can be used to manage assignments of organization roles centrally from within the Identity Provider (IdP) and not from vCloud Director at the Organization level. Central management might be beneficial in cases where there are many Organizations (and vCloud Director instances) associated with single IdP and one user might have access to multiple Organizations. With the traditional approach the user (or his group) would have to be imported into each Organization where he/she should have access and assigned a role.

By deferring to identity provider we rely on the IdP to provide the user’s role just in time when the user is logging in. The feature works both with SAML and OAuth identity providers. In my example I am going to be using the SAML IdP provided by Active Director Federation Services federated with vCloud Director as described in my older blog post.

The set up:

• Active Directory is used to manage all the users (with exception of local user for onboarding or troubleshooting purposes). The AD can be owned by the Service Provider or the tenant, it depends on the use case.
• AD FS has been deployed and integrated with vCloud Director Organizations. Note that each Organization must be federated with AD FS specifically. And the federation must be refreshed every year by regeneration of new certificate. In SP use case some level of automation is essential. For details refer to the blog article linked above.
• AD users who should have access to vCloud Director organization will be part of AD group <Tenant_X>.
• Role association will be achieved by assigning the AD users to specific AD groups: Organization Administrator, vApp Author, …
• For each role we will need to add new Transform Claim Rule in AD FS.
  1. On the existing relaying party trust click Edit Claim Rules
  2. Click Add Rule
  3. Select Send Group Membership as a Claim template
  4. Name the rule (e.g. vApp Author)
  5. Browse to the User’s group that represents the role membership (vApp Author group)
  6. In Outgoing claim type select Role.
  7. In Outgoing claim value type the vCloud Director role name (vApp Author).
• In vCloud Director Organization we need to do following:
  1. As already mentioned each Org needs to be federated with AD FS
  2. In Members > Groups import the <Tenant X> group from Source: SAML with Role: Defer to Identity Provider.
  3. With vCloud API specify RoleAttributeName in OrgFederationSettings. The name for AD FS should be:http://schemas.microsoft.com/ws/2008/06/identity/claims/role

Now the users can start logging in with their AD accounts and they will be automatically imported as users with Defer to IdP role. If needed, you can still directly import SAML users and specifically give them role which will take precedence over the IdP role.

Service providers who are offering additional services beyond vanilla vCloud Director IaaS were asking how to add links to them in the existing (Flex) vCloud Director UI.

vCloud Director 8.20 provides very simple way to extend the right column of the Home screen with additional sections and static links. It is really simple extensibility and should be used as interim solution until the new HTML 5 UI will fully replace the existing UI and which will be more extensible.

In the screenshot below you can see that the right section has been extended with vCloud Availability, Backup and Other sections.

The configuration of these links is very simple and is done from cell-management-tool on any vCloud cell.

In my example I used:

./cell-management-tool manage-config -n ui.tenant.customOrgLinks -v “
# vCloud Availability
[Monitor Replications](https://vcloud.fojta.com:8443)
# Backup
[Configure Backup](https://backup.fojta.com)
# Other
[Request Support](https://help.fojta.com)
[Impressum](https://www.fojta.com/impressum)”

Where # denotes the section header, [] the link name and () the link.

It is also possible to pass vCloud session ID as parameter in the URL by including {vcdSession} string.

The CMT manage-config command creates/modifies database entry in the config table tenant-customOrgLinks with the provided value in the quotes. Re-running it will replace the previous entry. The change is immediate, no need to run this on other cells or restart vcd services.

One last note, the right column on the home screen is not visible to all user roles. The role needs to have General > Administrator Control right.

This is a quick followup on my yesterday’s blog post that discussed how to customize vCloud Director UI with additional links. vCloud Availability has separate Portal UI where the users can monitor status of their replications and optionally trigger failover operations. Wouldn’t it be nice if the link from vCloud Director UI would automatically sign in the user into the vCloud Availability Portal UI?

Quick chat with the engineers showed that indeed it is possible by leveraging the {vcdSession} variable that provides the vCloud Director session token. The URL provided in the link then must look like this:

https://<vCloud_Availability_Portal_UI_FQDN >:8443/login?token={vcdSession}

In my case the CMT command for the whole link would look like this:

./cell-management-tool manage-config -n ui.tenant.customOrgLinks -v "
# vCloud Availability
[Monitor Replications](https://vcloud.fojta.com:8443/login?token={vcdSession})"

And this is the end result:

Click on the Monitor Replications link above (red box) opens vCloud Availability Portal screen with the tenant signed, in the next browser tab (below).

Quick post about an issue I discovered in my lab during upgrade to NSX 6.3.3. This particular NSX version has a silent new feature that verifies if syslog configuration on Edges is correct. If the syslog entry is incorrect (it is not an IP address or FQDN with at least one dot character or does not have TCP/UDP protocol specified) it will not let you save it. This however also means that older Edges (with version 6.3.2 or older) that have incorrect syslog setting will fail to be upgraded as the incorrect config will not be accepted.

So how does it relate to the title of the article? If you have vROps in your environment with NSX-V management pack and you have enabled Log Insight integration, the Management Pack will configure syslog on all NSX components. Unfortunately in my case it configures them incorrectly with only hostname and no protocol. This reconfiguration happens roughly every hour. This might be especially annoying in vCloud Director environment where all the Edges are initially deployed with syslog setting specified by VCD, but then are changed within an hour by vROps to something different.

Anyway, the remediation is simple. Disable the Log Insight integration of the vROps NSX Management Pack as shown on the picture below.

The vCloud Architecture Toolkit for Service Provider website has been updated with new set of documents. All documents were re-branded with the new VMware Cloud Provider Program logos that replace the old vCloud Air Network brand.

My Architecting a VMware vCloud Director Solution for VMware Cloud Providers whitepaper has been refreshed to include vCloud Director 8.10 and 8.20 additions that were missing in the previous version. The current version of the document is 2.8 with August 2017 release date.

Here is summary of the new or updated topics:

• Cell sizing
• vCloud DB performance tips
• New vCenter Chargeback Manager network metrics
• vRealize Business for Cloud
• vRealize Log Insight
• vRealize Operations Manager
• NSX Networking updates
• Storage support
• vCloud RBAC
• Org VDC vSphere Resource Settings
• VCDNI deprecation
• New Org VDC Edge GW features
• Distributed Firewall
• VM Auto import
• vCloud API for NSX
• vCloud Director orchestrated upgrade

The document can be downloaded in PDF format or viewed online.

VMware just released new version of vCloud Director – and it is a major release with version number 9.0.

Its documentation has now been moved to VMware Docs website: https://docs.vmware.com/en/vCloud-Director/index.html

I plan in the following days to blog about the major new features, so for now I will just provide list of all of the features, categorized by their exposure to tenants  or provider.

New features in tenant context:

• HTML 5 user interface which provides simplified VM deployment workflows (no vApp needed), is customizable and also provides VM Metrics (those coming from Cassandra DB)
• Networking enhancements: NSX Distributed routing with up to 1000 east-west routing optimized Org VDC networks per Org VDC Edge Gateway, security groups and tags for distributed firewall policies, VLAN tagging for VXLAN based Org VDC networks for Virtual Guest Tagging
• vMotion of VM between vApps in the same Org VDC

New features in tenant context with provider support:

• Multisite – multiple vCloud Director instances can be federated with association of individual organizations
• VLAN trunk support for vCloud Director external networks

New features in provider context:

• Manual VXLAN network pool creation
• Designating specific Provider VDC resource pool/cluster for dedicated Edge GW placement
• PostgreSQL DB support for vCloud Director database including migration from MS SQL/Oracle and high availability
• Securing vCloud Director database communication with encryption (only PostgreSQL)
• Per tenant storage migration workflows for storage array lifecycle
• VM metric database simplification (no KairosDB needed anymore, just pure Cassandra)
• vVOL support
• Enhanced support for vCenter Server HW/Guest operating systems
• SAML2 federation for System Administrators
• vSphere 5.5 is no longer supported

Note that announced migration tool called vCloud Director Extender has not yet been released.

vCloud Director uses Network Pools to create programmatically on-demand L2 networking segments for Org VDC and vApp networks. Network pools can be based on VLANs, VXLAN, port groups and legacy (deprecated) vCloud Network isolation (VCDNI) technology.

VXLAN Network Pool is recommended to be used as it scales the best. Until version 9, vCloud Director would create new VXLAN Network Pool automatically for each Provider VDC backed by NSX Transport Zone (again created automatically) scoped to cluster that belong to the particular Provider VDC. This would create multiple VXLAN network pools and potentially confusion which to use for a particular Org VDC.

In vCloud Director 9 we have the option to create our own VXLAN network pool backed by a NSX Transport Zone manually created and scoped to clusters we want to (and using any control plane mode).

During creation of Provider VDC we then have a choice to create a new VXLAN Network Pool (the legacy behavior) or use an existing one.

Advantages of the new feature are:

• No more clutter of large amount of VXLAN network pools (if there are many Provider VDCs)
• Simpler way to use hybrid or unicast control plane modes (vCloud Director would always default to multicast before)
• Control over scope of VXLAN networks – especially useful for sharing Org VDC networks between Org VDCs from different Provider VDCs.
• Adhering to best practice of scoping transport zone to whole vDS (more here)

vCloud Director version 9 introduces support for the last major missing NSX feature – the distributed logical router (DLR). DLR provides optimized router which in distributed fashion performs routing between different logical switches in the hypervisor. The routing always happens in the hypervisor running the source VM which means that the traffic goes between maximum two ESXi hosts (source and destination) and no tromboning through third host running router VM is necessary. Read here for technical deep dive into how this works. This not only provides much better performance than traditional Edge GW routing, but also scales up to 1000 routed logical networks (as opposed to 10 on Edge GW or up to 209 if trunk port is enabled).

Generally, DLR should be used for routing only between VXLAN based logical switches, although NSX supports VLANs networks with certain caveats as well. Additionally dynamic routing protocols are supported as well and managed by Control VM of the DLR.

Now let’s look how vCloud Director implements DLR. The main focus was making DLR very simple to use and seamlessly integrate with the existing networking Org VDC concepts.

• DLR is enabled on Org VDC Edge Gateway which must be already converted to advanced networking. You cannot use DLR without Org VDC Edge Gateway! There must be one free interface on the Edge (you will see later on why).
• Once DLR is enabled, a logical DLR instance is created in NSX in headless mode without DLR Control VM (the instance is named in NSX vse-dlr-<GW name) (<UUID>)). vCloud Director can get away without Control VM as dynamic routing is not necessary – see later below.
  
• The DLR instance uplink interface is connected to the Org VDC Edge GW with P2P connection using 10.255.255.248/30 subnet. The DLR uses .250 IP address and the Org VDC Edge GW uses .249. This subnet is hardcoded and cannot overlap with existing Org VDC Edge GW subnets. Obviously the Org VDC Edge GW needs at least one free interface.
• DLR has default gateway set to the Org VDC Edge GW interface (10.255.255.249)
• New Org VDC networks now can be created in the Org VDC with the choice to attach them to the Edge Gateway (as regular or subinterface in a trunk) or to attach them to the DLR instance.
  For each distributed Org VDC network a static route will be created on the Org VDC Edge Gateway to point to the DLR uplink interface. This means there is no need for dynamic routing protocols on the DLR instance.

In the diagram below is the networking topology of such setup.

In the example you can see three Org VDC networks. One (blue) traditional (10.10.10.0/24) attached directly to the Org VDC Edge GW and two (purple and orange) distributed (192.168.0.0/24 and 192.168.1.0/24) connected through the DLR instance. The P2P connection between Org VDC Edge GW and DLR instance is green.

• DHCP relay agents are automatically configured on DLR instance for each distributed Org VDC network and point to DHCP Relay Server – the Org VDC Edge GW interface (10.255.255.249). To enable DHCP service for particular distributed Org VDC network, the DHCP Pool with proper IP Range just needs to be manually created on the Org VDC Edge Gateway. If Auto Configure DNS is enabled, DHCP will provide IP address of the Org VDC Edge P2P interface to the DLR instance.

Considerations

• Up to 1000 distributed Org VDC networks can be connected to one Org VDC Edge GW (one DLR instance per Org VDC Edge GW).
• Some networking features (such as L2 VPN) are not supported on the distributed Org VDC networks.
• VLAN based Org VDC networks cannot be distributed. The Org VDC must use VXLAN network pool.
• IPv6 is not supported by DLR
• vApp routed networks cannot be distributed
• The tenant can override the automatic DHCP and static route configurations done by vCloud Director for distributed networks on the Org VDC Edge GW. The tenant cannot modify the P2P connection between the Edge and DLR instance.
• Disabling DLR on Org VDC Edge Gateways is possible but all distributed networks must be removed before.
• Both enabling and disabling DLR on Org VDC Edge Gateway are by default system administrator only operations. It is possible to grant these rights to a tenant with the granular RBAC introduced in vCloud Director 8.20.
• DLR feature is in the base NSX license in the VMware Cloud Provider Program.

Edit 02/10/2017: Engineering (Abhinav Mishra) provided a way how to change P2P subnet between the Edge and DLR. Add the following property value with CMT:

$VCLOUD_HOME/bin/cell-management-tool manage-config -n gateway.dlr.default.subnet.cidr -v <subnet CIDR>

Example: $VCLOUD_HOME/bin/cell-management-tool manage-config -n gateway.dlr.default.subnet.cidr -v 169.254.255.248/30

No need for cell reboot.

Edit 03/10/2017: Existing Org VDC networks can be migrated from traditional to DLR or sub-interface based networks in all directions in non-disruptive way with running VMs attached.

In the past in vCloud Director 8.20 (and older versions) system admins (the provider context) could use local, LDAP and vSphere SSO accounts. vCloud Director 9.0 now replaces vSphere SSO accounts with more generic SAML2 accounts which means you can have the same IdP mechanism in the tenant and system context.

This change however breaks the previous vSphere SSO federation which was as simple as entering the vSphere Lookup Service URL and enabling the vSphere Single Sign-On with a check box (which in vCloud Director 9.0 is no longer there).

Here is the procedure how to enable vSphere Single Sign-On in vCloud Director 9.0.

1. Login to vCloud Director as system admin and from administration>System Settings/Federation download the metadata document (spring_saml_metadata.xml) from the link provided (../cloud/org/System/saml/metadata/alias/vcd). Make sure the certificate (below) is valid.
2. Login to vSphere Web Client as SSO admin and go to Administration/Single Sign-On/Configuration/SAML Service Providers
3. Import the metadata from step #1
4. Download the vsphere.local.xml metadata from the link below.
   
5. Go back to VCD, check use SAML Identity Provider and upload metadata from #4.

Note that Import Users/Group source now changes from vSphere SSO to SAML.

vCloud Director 9 introduces brand new HTML5 tenant user interface. It currently provides only limited set of operations:

• simplified workflow to create VMs without the need to create vApps
• view and modify (but not create) vApps
• create and configure networks and networking services
• configure multisite organization associations

The legacy (flex) UI is still available and the user can jump to it if needed which is useful especially for Org Administrators.

The new UI can be modified – in the initial release with the following elements:

• Portal name
• Portal color (in RGB HEX format e.g. #EE00EE)
• Portal theme (default/dark)
• Logo (png file of max size 512 KB)

This will expand in the future with tenant granularity, custom themes and custom links, but for now we have only these four elements.

The configuration must be done with new CloudAPI which is JSON based. At the time of writing its documentation has not been yet published.

• Retrieve authentication token with vCloud API or UI
• Use the following call to change logo:

PUT /cloudapi/branding/logo

Content-Type: image/png

x-vcloud-authorization: …

Body: <binary image file>

• Use the following call to change other elements:

PUT /cloudapi/branding

Content-Type: application/json

x-vcloud-authorization: …

Body:
{
    “portalName”: “vCloud Director 9.0”,
    “portalColor”: “#EE00EE”,
    “selectedTheme”: {
        “themeType”: “BUILT_IN”,
        “name”: “Dark”    },
    “customLinks”: []
}

For some time there has been a hidden security feature in vCloud Director that allows disabling local system administrator accounts. During vCloud Director installation a default local system administrator account is created. The user credentials are stored encrypted in the vCloud Director database but there is no way to enforce complex password policies other than Account Lockout Policy.

It is possible to configure external identity sources such us generic LDAP for basic authentication and SAML2 IdP (such as vCenter SSO). The authentication and thus also the password policies are than managed externally. However, when you try to delete or disable all local system administrator accounts you will get the following error:

Cannot delete or deactivate the last system administrator.

This is a built in protection against completely locking yourself out when the external identity sources are not available.

Some customers can have the need to enforce strict security rules on all vCloud Director system administrator logins. There is a non-documented way to disable all local system administrator accounts with a single command. The system administrator can run the following cell-management-tool  command to enable config property local.sysadmin.disabled.

$VCLOUD_HOME/bin/cell-management-tool manage-config -n local.sysadmin.disabled -v true

Immediately after the property is enabled, authentication with local accounts will stop working. It means authentication for all local system administrator accounts that exist in vCloud Director (not just the default account created during installation) will be rejected. Organization local accounts will not be affected.

In case access to external IdPs is lost, the system admin can again disable the property to regain access to vCloud Director:

$VCLOUD_HOME/bin/cell-management-tool manage-config -n local.sysadmin.disabled -v false