I just learned from engineering neat trick related to how cell management tool can be invoked without specifying administrator credentials.

The issue is that currently you cannot use LDAP account to trigger cell management tool commands which are mostly used for quiescing and shutting down cells for maintenance. Using vCloud Director local administrator account is discouraged as it poses a security issue. However what is possible is to trigger the cell management tool as root (or with sudo) and supply via hidden flag -i the process ID of the java process.

Here is an example:

First I query the java PID with ps aux command. Then I use the standard cell-management-tool command without specifying the user with the -i flag at the end.

So you can force the administrator to log in to the cell guest OS via a LDAP account and then run the command with sudo.

Thank you Zachary Shepherd for the tip.

Offline Data Transfer (ODT) is a feature of vCloud Connector that allows migration of VMs from customer own datacenter to vCloud Air with NAS appliance which is shipped via regular mail. The point is to avoid slow wide area network connectivity and leverage awesome bandwidth but slow latency of sneakernet.

Have you ever wondered why it is supported only with vCloud Air and not with any public or private cloud based on vCloud Director? Well I am going to lay down the whole process here in this blog post so nothing is stopping anyone testing this feature on your own.

Let me first paste picture from the manual which describes in high level how the process works:

vCloud Connector (vCC) is leveraged to manage the whole process. The customer (on left) deploys his own vCloud Connector Server and Node which he attaches to his on premises infrastructure (vSphere based). He then requests the ODT service. The provider will deploy ODT node in the public cloud (on right) and also its own vCC Server to manage it. Regular NAS appliance is prepared – its only purpose is to provide storage capacity which is fast and reliable enough via NFS protocol and can be easily packaged and shipped.

Customer mounts it to his vCC Node (to a directory via NFS mount). Both the ODT and vCC Nodes are registered in his vCC Server. Then via the traditional vSphere Client and vCC Plugin only the local vSphere environment (here it differs from the traditional vCC transfer).

The actual export is done by selecting the objects to export (templates, vApps or VMs) and clicking the small Offline Data Transfer icon: Mount path is entered and links and credentials to the target Cloud and ODT node. There is also option to select if a particular VM should be deployed and connected to a network. These steps above are all described in the manual here.

But what about the provider side of the whole process?

ODT Node

ODT Node is actually regular vCloud Connector Node tweaked by running configureSneakernetNode.sh script which can be found in /opt/vmware/hcagent/scripts folder on the Node VM itself.

The ODT needs to have network access to the vCenter Server (and ESXi hosts) of the target vCloud VDC environment.

Import

The actual import is done via provider vCloud Connector server which is again the regular vCloud Connector server with no tweaks this time. The ODT Node is registered there which enables import menu in the vCC plugin GUi in vSphere Client. The shipped NAS appliance must be mounted to the ODT Node and the ODT URL and mount path is entered in the Import Wizzard. The actual physical connection of the NAS appliance can be done using dedicated VLAN with point to point connection of the second ODT network interface.

Next we need to pick the target vCenter Server and a credentials for it. ODT Node will import offline VMs which are stored as encrypted OVFs on the NAS appliance into the target vCenter Server. To do that it needs a big enough datastore and a dummy network in order to connect the imported VMs temporarily to it. Once that is done VMs are imported by vCloud Director to the target VDCs, catalogs and networks. The provider needs to have big enough datastore and create dummy standard switch port group on every host with name ‘VM Network’. This network does not need to have external access.

As you can see contrary to the regular internet vCloud Connector transfer where the VM is transfer from the original environment via on-prem node to public node to vCloud Director (through its API and transfer storage – see here for more detail) the transfer does not go through the vCloud Director cells and its transfer storage at all. This is possible thanks to handling the final step of the process by the provider himself (he has vCenter Server access) and makes also the transfer faster (potentially one less step). On the other hand this brings some security and operational process challenges (physical access to management network, vCenter credentials) which must be properly addressed.

One of my customers had deployed true hybrid vRealize Automation with multiple cloud endpoints: vCloud Air and internal vCloud Director and AWS. I was called in to troubleshoot strange issue where sometimes deployment of a cloud multimachine blueprint (vApp) would work but most often it would fail with the following message:

VCloud Clone VM failed for machine: XXX100 [Workflow Instance Id=19026]
System.InvalidOperationException: Error occurred while getting vApp template with ID: urn:vcloud:vapptemplate:a21de50d-8b5e-41a6-81d1-acfd8ab8364b
…
INNER EXCEPTION: com.vmware.vcloud.sdk.utility.VCloudException: [ 8ae6fbca-e0d2-43e7-bc94-5bc9d776bf8d ] No access to entity “com.vmware.vcloud.entity.vapptemplate:a21de50d-8b5e-41a6-81d1-acfd8ab8364b”

Endpoint was properly configured, template existed, so what could be wrong? Why were we denied the access to the template?

It turns out that by design vRealize Automation does not match a template to a particular endpoint. It identifies it just by name. So in our case sometimes it would try to deploy the blueprint to wrong endpoint where the template of the particular name did not exist.

The fix is simple:

• Define reservation policies which would identify each endpoint.
• Assign them to the proper reservations
• Assign reservation policies to the Cloud vApp blueprint. This way there will never be confusion from which template to provision to which endpoint.

A customer asked me how to track deployments of certain catalog templates. This is mainly for licensing or charging purposes where the provider creates and maintains (patches) OS templates which the tenants can consume via deployment from public catalog.

There are multiple ways how to achieve this. One involves tracking the deployment tasks from the catalog via Audit Event logs or RabbitMQ blocking tasks. The other I find very elegant is leveraging vCloud metadata which can be assigned to many vCloud Director objects. The actual feature has been present in vCloud Director for a long time (1.5) but is has had a few improvements since.

The owner of an object can create, update and delete up to 1024 name=value pairs on each particular object. An additional group of name=value pairs is under control of the system administrator who can chose visibility of such pair: hidden, read only or read/write. The metadata can be accessed both via GUI and API.

So back to our use case:

1. As System Administrator create hidden metadata entry on the catalog VM. For example: “LicensingID = AAABBB“. This must be done before adding the VM to the public catalog.
2. From now on every deployed VM from this catalog template will have this hidden metadata. The tenant user will not see it, however system admin will:

   Tenant View

   
   

   Admin View

3. Querying all VMs which contain this metadata can be done with one API call
   GET https://vcloud.fojta.com/api/query?type=adminVM&filter=(metadata@SYSTEM:LicensingID==STRING:AAABBB)

   API Response

   The response contains Org ID (highlighted in yellow) which can be used for cost allocation to each tenant.

Quick PowerCLI one-liner which displays all VMs running in vCloud Director together with their VM Tools version.

Get-CIVM |Select Name, {$_.Org.Name}, {$_.OrgVdc.Name}, {$_.ExtensionData.Section.VMWareTools.Version}

I see more and more that new and existing vCloud Director deployments leverage NSX as the networking component instead of the legacy vShield / vCloud Network and Security (vCNS). The main reasons are the announced end-of-life for vCNS and the additional features that NSX brings to the table (although most of them are not yet tenant consumable in vCloud Director – as of version 5.6.4).

When deploying NSX with vCloud Director what new considerations should be included when designing the architecture? In this post I want to concentrate on the concept of the Edge Cluster.

What is Edge Cluster?

VMware has published very good NSX-V Network Virtualization Design Guide. This is very detailed document describing all NSX concepts as well as how they should be properly architected. The concept of Edge Cluster is discussed in quite a detail as well so let me just summarize here.

NSX overlay networks allow the creation of logical networks over an existing IP network fabric. This enables highly scalable network design using Leaf / Spine architecture where the boundary between L2 and L3 networks is at the rack level (leafs) and all communication between racks is L3 only going through a set of spine routers.

NSX spans logical network across all racks however in the end we need to connect virtual workloads from the logical networks to the outside physical world (WAN, Internet, colocated physical servers, etc.). These networks are represented by a set of VLAN networks and because we are not stretching L2 across the racks we cannot trunk them everywhere – so they are connected only to one (or two for redundancy) rack which thus become the Edge Cluster.

So the purpose of the Edge Cluster is to host virtual routers – Edge Service Gateways that provide the connectivity between the physical world (VLANs) and virtual world (VXLAN logical switchites). Note that it does not mean that every Edge Gateway needs to be deployed there. If an Edge Gateway provides connectivity between two VXLAN logical switches – it can be deployed anywhere as logical switches span all clusters.

vCloud Director Edges

vCloud Director deploys Edge VMs in order to provide Organization VDC or vApp connectivity. The actual deployment is done through vCNS or NSX Manager but it is vCloud Director who makes decision about placement and configuration of the Edges. vCloud Director Edge Gateway provides connectivity between one or more vCloud Director External Network and one or more Organization VDC Network. It is deployed inside Provider VDC in a special System VDC Resource Pool on a datastore belonging to the Org VDC default storage policy. vCloud Director placement engine selects the most appropriate cluster where the Edge Gateway VM will be deployed – based on which clusters belong to Provider VDC, what is their available capacity and most importantly their access to the right storage and external networks.

vApp Edges provide connectivity between an Organization VDC network and a vApp network. They always have only one external and one internal interface. They are also deployed by vCloud Director to the Provider VDC System VDC Resource Pool and exist only when the vApp is in deployed mode (Powered On).

Transport Zone

Transport Zone defines the scope of a VXLAN logical switch. It consists of one or more vSphere clusters. Transport Zone can be created manually, however vCloud Director automatically creates for each Provider VDC one Transport Zone which matches the clusters that are added to the Provider VDC and associates it with a VXLAN Network Pool. When Organization VDC is created by vCloud System Administrator a Network Pool must be assigned – all Organization VDC and vApp Networks will then have its scope.

Design Option I – Traditional

In the traditional network architecture Access/Aggregation/Core the L2/L3 boundary is at the aggregation switches. This means all racks connected to the same set of aggregation switches have access to the same VLANs and thus there is no need for an Edge Cluster as the Edge connecting VLAN with VXLAN based networks can run on any given rack. In vCloud Director it means that as long as the external networks (VLANs) are trunked to aggregation switches we do not need to worry about Edge placement. The set of racks (clusters) connected to the same aggregation domain usually map to a vCloud Director Provider VDC. The transport zone is then identical to the aggregation domain.

Traditional Access/Aggregation/Core architecture

The drawback of such design is that Provider VDCs cannot span multiple aggregation domains.

Design Option II – Combined Edge/Compute Cluster

In case spine/leaf network architecture is used, VLANs backing vCloud Director external networks are trunked only to one cluster. In this design option we will call it Edge/Compute Cluster. As explained above vCloud Director placement engine will deploy Edge VMs to a cluster based on VLAN connectivity – therefore it will automatically place all Edge Gateways into the Edge/Compute cluster as this is the only cluster where the external connectivity (VLANs) exists. vCloud Director will however also opportunistically place regular tenant VMs into this cluster (hence its name Edge/Compute).

Spine/leaf with Edge/Compute Cluster

This design option has all the scale advantages of Spine/Leaf architecture however the possibility of tenant workloads taking limited space of Edge/Compute cluster is the drawback. There are two potential options how to remediate this:

1. vCloud Director Edge Gateways are always deployed by vCloud System Administrator. He/she could make sure that prior Edge Gateway deployment there is enough capacity in the Edge/Compute cluster. If not some tenant workloads can be migrated away to another cluster – this must be done from within vCloud Director (Resource Pool / Migrate to option). Live migration is however possible only if the Edge/Compute Cluster shares the same VXLAN prepared vSphere Distributed Switch (vDS) with the other clusters and this requires at least four network uplinks on the Edge/Compute Cluster hosts (two uplinks for vDS with external VLANs and two uplinks for VXLAN vDS).
2. Artificially limit the size of Edge/Compute Cluster so the placement engine does not choose it for regular tenant workloads. This can be done by leveraging Resource Pool which is created manually in the Edge/Compute cluster and attached to the Provider VDC instead of the whole cluster. Then an artificial limit is set by System Administrator and is increased only when a new Edge Gateway needs to be deployed.

Both options unfortunately provide significant operational overhead.

Design Option IIb – Combined Edge/Compute Cluster with Non-elastic VDC

While elastic Org VDC types (such are Pay-As-You-Go or Allocation type) can span multiple clusters what would be the impact of non-elastic VDC such as Reservation Pool in this design option?

In non-elastic Org VDC all tenant workloads are deployed into the primary Provider VDC resource pool. However Edge VMs can be deployed into secondary resource pools. This means as long as the Edge/Compute cluster is added as secondary Resource Pool into a Provider VDC this design option can still be used.

Spine/leaf with Edge/Compute Clsuter and non-elastic VDC

Design Option III – Dedicated Edge Cluster

This design option extends the previous one but in this case we will have dedicated Edge Cluster which is not managed by vCloud Director at all. We will also introduce new Edge Gateway type – Provider Edges. These are manually deployed by the service provider totally outside of vCloud Director into the Edge Cluster. Their external uplinks are connected to external VLAN based networks and internal interfaces are connected to transit VXLAN Logical Switch spanning all Compute and the Edge clusters (manually created transport zone with all clusters). The transit network(s) are then consumed by vCloud Director as External Network – note that little workaround is need to do so – read here.

The Provider Edges can provide all NSX functionality (dynamic routing protocols on external uplinks, L2 bridging, L2 VPN, etc.). They can scale as additional vCloud Director External Networks are added (current maximum in VCD 5.6 is 750 External Networks). The Edges deployed by vCloud Director then go into compute clusters as all their interfaces connect to VXLAN logical switches spanned everywhere in the Provider VDC.

Spine/leaf with Dedicated Edge Cluster

Read vCloud Director with NSX: Edge Cluster (Part 2) here.

In my previous article vCloud Director with NSX: Edge Cluster I described various design options of NSX Edge Cluster in vCloud Director environment. In this article I would like to discuss additional option which extends the Design Option III – Dedicated Edge Cluster. Below is the picture showing the scenario from the previous post.

Spine/leaf with Dedicated Edge Cluster

There is one Provider deployed Edge in the Edge Cluster for each Transit vCloud Director External network to which Org VDC Edge Gateways are connected to. The option works quite well for use cases where the Provider Edge is dedicated to single tenant – e.g. it is providing VPN services or L2 bridging. (Note that in L2 bridging use case the Org VDC Edge Gateway is not deployed and Org VDC networks connect directly to tenant dedicated external network).

However when we want to provide access to a shared service (for example internet) where we will deploy multiple Org VDC Edge Gateways of different tenants connected to the same external network they will all have to go through a single Provider Edge which can become a bottleneck.

As of NSX version 6.1 Edge Gateways can however be deployed in ECMP (Equal Cost Multi-Path) configuration where we can aggregate bandwidth of up to 8 Edges (8x10GB = 80 GB througput). High availability of ECMP Edges is then achieved with dynamic routing protocol (BGP or OSPF) with aggressive timing for short failover times (3 seconds) which will quickly remove failed path from the routing tables.

The problem is that (as of vCloud Director 5.6) Organization VDC Edges are deployed in the legacy (vShield/vCNS) mode and do not support ECMP routing nor dynamic routing protocols. The design I propose will get around this limitation by deploying Distributed Logical Router between Provider and Organization VDC Edges.

Spine/leaf with Dedicated Edge Cluster and ECMP Edges

The picture above shows two Provider ECMP Edges (can scale up to 8) with two physical VLAN connections each to upstream physical router and one internal interface to the Transit Edge logical switch. Distributed Logical Router (DLR) then connects the Transit Edge logical switch with the Transit vCloud Director External Network to which all tenant Org VDC Edge Gateways are connected to. The DLR has ECMP routing enabled as well as OSPF or BGP dynamic routing peering with the Provider Edges. The DLR will provide two (or more) equal paths to upstream Provider Edges and will choose one based on hashing algorithm of source and destination IP of the routed packet.

The two shown Org VDC Edge Gateways (which can belong to two different tenants) then will take advantage of all the bandwidth provided by the Edge Cluster (indicated with the orange arrows).

The picture also depicts the DLR Control VM. This is the protocol endpoint which peers with Provider Edges and learns and announces routes. These are then distributed to ESXi host vmkernel routing process by the NSX Controller Cluster (not shown in the picture). The failure of DLR Control VM has impact on routing information learned via OSPF/BGP protocol even if DLR is highly available in active standby configuration due to the protocol aggressive timers (DLR control VM failover takes more than 3 seconds). Therefore we will create static route on all ECMP Provider Edges for the Transit vCloud Director External network subnet. That is enough for north – south routing as Org VDC subnets are always NATed by the tenant Org VDC Edge Gateway. South – north routing is static as the Org VDC Edge Gateways are configured with default gateway defined in the External Network properties.

The other consideration is placement of DLR Control VM. If it fails together with one of ECMP Provider Edges the ESXi host vmkernel routes are not updated until DLR Control VM functionality  fails over to the passive instance and meanwhile route to the dead Provider Edge is black holing traffic. If we have enough hosts in the Edge Cluster we should deploy DLR Control VMs with anti-affinity to all ECMP Edges. Most likely we will not have enough hosts therefore we would deployed DLR Control VMs to one of the compute clusters. The VMs are very small (512 MB, 1 vCPU) therefore the cluster capacity impact is negligible

My colleague George Kobar recently wrote blog post about running Nested ESXi on vCloud Air. The gist of the article is the description how to solve networking issues around nested virtualization that usually result in the need to have promiscuous mode enabled on the virtual distributed switch which is not possible to do in public cloud environment. The trick is to use nested vMACs identical to those assigned by vCloud Air to virtual ESXi hosts.

I had the opportunity to test the practical viability of the approach when I needed to test quite complex architecture which involved NSX, multiple vSphere 6 racks with spine and leaf network architecture and vCloud Director version 8 (currently in public beta). If you read my previous article about vCloud Director and NSX you will recognize the topology.

I have 3 racks each with two ESXi hosts running in different subnets and all communication between racks is routed over the spine router. For nested storage I am using HP VSA which provides iSCSI storage and then there are some management components (Domain Controller, Windows based vCenter Server with MS SQL database, NSX Manager and vCloud Director cell). On the virtual ESXi hosts are running nested VMs – NSX Controller, Edge Gateways, DLR control VM and some linux test VMs.

Here is how I set the whole nested lab:

• One vCloud Air Edge Gateway is simulating the spine router. It has 8 7 Org VDC networks attached. For each rack there is ESXi management network (used for ESXi management, vMotion and iSCSI traffic) and transport network used for VXLAN fabric. Then there is one management network for management workloads and one for Internet access which is also routed to Internet to allow out of band RDP access to vCenter Server which acts as jump host so I do not need to use vCloud Air console.
• Another vCloud Air Edge Gateway is used to simulate WAN router with external access for virtual workloads with Internet Org VDC network. As my vCloud Air VDC is in Advanced Networking Services beta I could even leverage OSPF peering between the Edge and my nested Edges.
• Edge rack virtual ESXi hosts have plenty of NICs in order to have enough vMACs that can be used by nested objects (VMkernel ports and VMs). I used the following strategy:
  • One NIC is used for ESXi management. ESXi uses the ‘physical’ MAC address for vmk0 so no worries and management/vMotion and iSCSI networking works out of the box (and if not check this KB). Standard Switch is used.
  • Second NIC is used for VXLAN VTEP. When you prepare VXLAN fabric on nested hosts random MACs are assigned to VTEP vmk1. These need to be changed after the preparation so they are identical to MACs assigned to virtual ESXi hosts. This is done by editing /etc/vmware/esx.conf on each host and rebooting. Do VXLAN pings to check if your fabric is set up properly. Any VM connected to VXLAN network will have its traffic encapsulated by VTEPs and thus properly work in the nested environment.
  • ECMP Edges have uplink interface connected to VLAN network. I used the 3rd ESXi NIC on each Edge Rack host for one Edge. When you deploy the Edge via NSX Manager you specify the corresponding MAC of the virtual ESXi host. As the Edge cannot be vMotioned (its MAC wouldn’t match ESXi host MAC) I deployed it to local storage of the virtual ESXi host and the VLAN was presented on local standard switch. BTW tying ECMP Edges to particular host is recommended practice anyway.
  • The last challenge is the deployment of the NSX Controller (I deployed only one which is enough for testing needs). The NSX Controller needs to be deployed to regular non-VXLAN portgroup and it is not possible to set its MAC address as it is deployed by NSX Manager in automated fashion. So I deployed another Edge (called Controller router) on one of the Edge rack ESXi hosts again with uplink MAC set to MAC of the 4th NIC of the ESXi host. Then I created portgroup for the NSX Controller deployment and deployed the Controller. As long as the NSX Controller VM is on the same host as the Controller router (both deployed to local storage) the connectivity is local and then routed to the outside world with the Controller VM MAC.
  • EDIT 9/23/2015 Do not forget to add custom static route on the spine router (vCA Edge Gateway) to the NSX Controller subnet via its nested Controller router.

Here are some screenshots showing different views of the nested lab:

vCloud Air vApp View

Nested vCenter View

Nested Networking View

Once the base of the nested lab is set up it can be used for many different things – vCloud Director or vRealize Automation testing or testing upgrades (I personally upgraded the lab from vSphere 5.5 to 6).

When VMware NSX 6.0 came out about more than one year ago, one of the new great features it had on top of the its predecessor VMware vCloud Network and Security (vCNS) was L2VPN service on Edge Service Gateway which allows stretching layer 2 network segments between distant sites in different management domains. NSX 6.1 further enhanced the functionality by introducing Standalone Edge which can be deployed on top of vSphere without an NSX license and acts as L2VPN client.

Many vCloud Service Providers are now deploying their public cloud with vCloud Director and NSX instead of vCNS so I am often asked how could they leverage NSX in order to provide L2VPN service to their tenants.

As of today neither vCloud Director 5.6 nor 8.0 (in beta at the moment) can deploy NSX Edges and manage the L2VPN service. However it is still possible for the SP to provide L2VPN as a managed service for his tenants.

Let me demonstrate how would it work on the following artificial example.

The customer has an application that resides on 3 different VLAN based networks (subnet A, B and C) routed with existing physical router. He would like to extend subnets A and B into the cloud and deploy VMs there. The VMs in the cloud should access the internet in order to connect to external SaaS services through the provider connection (egress optimization) but should still be able to reach database running on subnet C which is hosted on premises.

The diagram below shows the whole architecture (click for larger version):

On the left is the customer on premises datacenter with physical router and three VLAN based networks. On the right is the public cloud with NSX design I proposed in one of my previous blog articles. While the unimportant parts are grayed out what is important is how the customer Org VDC and the NSX Edge Gateways is deployed.

• The provider deploys tenant dedicated NSX Edge Service Gateway outside of vCloud Director manually and configures it based on customer requirements. The provider creates two logical switches (VXLAN 5005 and 5006) which will be used for extending customer subnets A and B. The switches are trunked to the NSX Edge and the Edge interface IP addresses are assigned identical to the IP addresses of the physical router on-premises (a.a.a.1 and b.b.b.1).
• The two logical switches are configured in vCloud Director as External Networks with the proper subnets A and B and pool of unused IPs.
• Two Org VDC networks are created inside tenant’s Org VDC as directly connected to the two External Networks.
• L2VPN server is configured on the NSX Edge (with encryption algorithm, secret, certificate and stretched interfaces). Also Egress Optimization Gateway Address is configured (both physical gateway IPs are entered – a.a.a.1 and b.b.b.1). This will filter ARP replies of the two gateways sharing the same IPs in the tunnel and allow NSX Edge to act as the gateway to the internet.
• The tenant will install Standalone Edge which is distributed as OVF inside his datacenter and set it up: he must map VLANs to tunnel IDs supplied by the provider, configure Edge Server public IP and port and encryption details.

Now what about the subnet C? How can VMs deployed in the cloud get to it if the physical router is unreachable due to the enabled egress optimization? Following trick is used:

• Another subnet z.z.z.0/30 is used for P2P connection between the NSX Edge in the cloud and the physical router.
• IP address z.z.z.1/30 is configured on the physical router on one of the stretched subnets (e.g. A).
• The second IP z.z.z.2/30 is configured on the NSX Edge on the same subnet.
• Finally static route is created on the NSX Edge pointing subnet C to the next hop address z.z.z.1.

Some additional considerations:

• In case the tenant has licensed NSX on-premises he can obviously use ‘full’ NSX Edge Service Gateway. The advantages are that it is much easier to deploy and configure. It can also stretch VXLAN based networks as opposed to only VLANs which are supported by Standalone Edge.
• Standalone Edge can be connected either to Standard Switch or Distributed Switch. When Standard Switch is used promiscuous mode and forged transmits must be enabled on the trunk port group. VLANs ID 4095 (all) must be configured to pass multiple VLANs.
• When Distributed Switch is used it is recommended to use Sink Port instead of promiscuous mode. Sink Port receives traffic with MAC addresses unknown to vDS.
• Sink Port creation is described here. It requires vDS edit via vCenter Managed Object UI. While Sink Port can be also created with net-dvs –EnableSink command directly on the ESXi host with Standalone Edge VM running it is not recommended as the host configuration can be overridden by vCenter Server.
• RC4-MD4 encryption cipher should not be used as it is insecure and has been deprecated in NSX 6.2.

One of the reasons I have been quiet on my blog lately was today released for public: vCloud Architecture Toolkit for Service Providers.

If you are designing vCloud Director based public cloud I hope you will find design considerations and recommendation included helpful. Let me highlight a few in my opinion interesting topics from the Architecting a VMware vCloud Director Solution document:

• Virtual Machine metric Cassandra database
• RabbitMQ messaging infrastructure
• Considerations around NSX Edge Cluster and NSX Controllers
• Consumption of NSX services
• VSAN impact on vCloud Director catalogs
• OAuth authentication

While vCloud Director offers impressive scalability with possibility to manage up to 20 vCenter Servers there are valid reasons why service providers deploy multiple vCloud Director instances due to fault domains (availability zones) or latency (multiple geographical regions) requirements.

In such case a single tenant (end-customer) has multiple organization accounts in multiple vCloud Director instance each with access to subset of VDCs, catalogs, VMs, etc. The service provider would usually hide this under one custom overarching portal however when the tenant wants to discover and manage his resources in programmable fashion he has to access multiple vCloud API endpoints of the vCloud Director instances.

vCloud Director version 8.0 offers a new feature: Organization Associations. If you search through vCloud API v 9.0 you will see new Admin Types:  OrgAssociationType  and OrgAssociationsType and related new Admin Element OrgAssociationMember. So what it is and how does it work?

The service provider can associate multiple Organizations (belonging to the same customer) together and simplify discovery of all cloud resources through single API endpoint.

Example

In my lab I have two different vCloud Director instances with URLs vcloud.fojta.com and vcloud2.fojta.com. In the first instance I have created two organizations belonging to the same customer: ACME and ACME2. In the second instance I have created organization ACME. All three organizations use the same identity source (in my case LDAP) and the same user exists in all of them with Organization Administrator credentials.

1. The system administrator creates the organization associations. In this example I have associated org: ACME in the second instance with ACME and ACME2 in the first instance. This is done with the following vCloud API call:

   PUT https://vcloud2.fojta.com/api/admin/org/ca5295f0-a521-4d4c-8b2e-322f154fbbea/associations
   Content-Type: application/vnd.vmware.admin.organizationAssociations+xml
   
   Body:
   <?xml version="1.0" encoding="UTF-8"?>
   <OrgAssociations xmlns="http://www.vmware.com/vcloud/v1.5">
       <OrgAssociationMember href="https://vcloud2.fojta.com/api/admin/org/ca5295f0-a521-4d4c-8b2e-322f154fbbea/associations/02b433db-0b37-4304-b07b-0717255ec297" type="application/vnd.vmware.admin.organizationAssociation+xml">
           <MemberUrl>https://vcloud.fojta.com/api/org/02b433db-0b37-4304-b07b-0717255ec297</MemberUrl>
           <MemberName>ACME</MemberName>
           <MemberEndpointCertificate>-----BEGIN CERTIFICATE-----
   MIIFfzCCBGegAwIBAgITTgAAARuwZOW3iRv9KQABAAABGzANBgkqhkiG9w0BAQsF
   ADBCMRMwEQYKCZImiZPyLGQBGRYDY29tMRUwEwYKCZImiZPyLGQBGRYFZm9qdGEx
   FDASBgNVBAMTC2ZvanRhLURDLUNBMB4XDTE1MDExNTE2NTAwNFoXDTE3MDExNDE2
   NTAwNFowZzELMAkGA1UEBhMCQ1oxDzANBgNVBAcTBlByYWd1ZTESMBAGA1UEChMJ
   Zm9qdGEuY29tMRgwFgYDVQQLEw92Q2xvdWQgRGlyZWN0b3IxGTAXBgNVBAMTEHZj
   bG91ZC5mb2p0YS5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC1
   UeaPIsHjRQb8PkybTv7tfCe6oyq8UUxc7tiwX+nWvHWKJ9X6ASis1v/gT0CCa4cG
   fP+tezXayXMrwFSRu6OvanBSTVYvaAbUQl5CsVnQaaeCC5bTAMfGlDsl/q+LnqzW
   i0eG4hTpWG78v88AZkaHjIZdr5CQuDaPGJUqzOgrHjpYTLDJs+oK+S7ScpMyKhia
   hgeRKDfrEeQGGvSEMdHhg3Bg8RK8eyLQLjwCSCVkhYTrM5wyt4fow65beoMnOBbx
   LO+QpB6/amy5mJuVLVx+WJivVvuId2hmELorm2iJMjUAabybLmbMPmqHTTGyZaYw
   vxaDRDr0DbTMUYFyOh6LAgMBAAGjggJHMIICQzAdBgNVHQ4EFgQUi1Dhxpbkz9Dh
   tYOljP+MW/9GF+AwHwYDVR0jBBgwFoAUs0GCJG1KfknG9couJQXq4CZq4SQwgfoG
   A1UdHwSB8jCB7zCB7KCB6aCB5oaBr2xkYXA6Ly8vQ049Zm9qdGEtREMtQ0EoMSks
   Q049REMyLENOPUNEUCxDTj1QdWJsaWMlMjBLZXklMjBTZXJ2aWNlcyxDTj1TZXJ2
   aWNlcyxDTj1Db25maWd1cmF0aW9uLERDPWZvanRhLERDPWNvbT9jZXJ0aWZpY2F0
   ZVJldm9jYXRpb25MaXN0P2Jhc2U/b2JqZWN0Q2xhc3M9Y1JMRGlzdHJpYnV0aW9u
   UG9pbnSGMmh0dHA6Ly9EQzIuZm9qdGEuY29tL0NlcnRFbnJvbGwvZm9qdGEtREMt
   Q0EoMSkuY3JsMIG7BggrBgEFBQcBAQSBrjCBqzCBqAYIKwYBBQUHMAKGgZtsZGFw
   Oi8vL0NOPWZvanRhLURDLUNBLENOPUFJQSxDTj1QdWJsaWMlMjBLZXklMjBTZXJ2
   aWNlcyxDTj1TZXJ2aWNlcyxDTj1Db25maWd1cmF0aW9uLERDPWZvanRhLERDPWNv
   bT9jQUNlcnRpZmljYXRlP2Jhc2U/b2JqZWN0Q2xhc3M9Y2VydGlmaWNhdGlvbkF1
   dGhvcml0eTAhBgkrBgEEAYI3FAIEFB4SAFcAZQBiAFMAZQByAHYAZQByMA4GA1Ud
   DwEB/wQEAwIFoDATBgNVHSUEDDAKBggrBgEFBQcDATANBgkqhkiG9w0BAQsFAAOC
   AQEArNmceh3GkHx5Qf9s/NUlG7lh+i5nPUzlbnlhTJQeOJXSKG3DzU8ocb1xWguT
   x1ICyLZTQq11q3D3/3xgi9KJJaWzo8X5O/Mj81x4V8Jp9d8OgERc7lrVyrAJPbJA
   k7q/4tY41VOu8P5i+A21Pxuo3xELkOt5rcb2qt6QH1QizSA8Dzjq8uwCpdDo8eCP
   ZWUwc2lNUOyCmhFD7boNecHRJZN92i3W0YKfV+BC0cIXnqU2Y+4YEKAHWwH/gRm0
   ZI41oyatyoHpTjCGFtKzrSo/mxitIoj5ZTY9wwSNPlcmziw29hOTM1fOx//rqgrW
   17CREB/BoFj12Hd9bVXFgMGUSg==
   -----END CERTIFICATE-----</MemberEndpointCertificate>
       </OrgAssociationMember>
       <OrgAssociationMember href="https://vcloud2.fojta.com/api/admin/org/ca5295f0-a521-4d4c-8b2e-322f154fbbea/associations/13e52807-3d0a-4c0f-abdb-62d8fccb36ea" type="application/vnd.vmware.admin.organizationAssociation+xml">
           <MemberUrl>https://vcloud.fojta.com/api/org/13e52807-3d0a-4c0f-abdb-62d8fccb36ea</MemberUrl>
           <MemberName>ACME2</MemberName>
           <MemberEndpointCertificate>-----BEGIN CERTIFICATE-----
   MIIFfzCCBGegAwIBAgITTgAAARuwZOW3iRv9KQABAAABGzANBgkqhkiG9w0BAQsF
   ADBCMRMwEQYKCZImiZPyLGQBGRYDY29tMRUwEwYKCZImiZPyLGQBGRYFZm9qdGEx
   FDASBgNVBAMTC2ZvanRhLURDLUNBMB4XDTE1MDExNTE2NTAwNFoXDTE3MDExNDE2
   NTAwNFowZzELMAkGA1UEBhMCQ1oxDzANBgNVBAcTBlByYWd1ZTESMBAGA1UEChMJ
   Zm9qdGEuY29tMRgwFgYDVQQLEw92Q2xvdWQgRGlyZWN0b3IxGTAXBgNVBAMTEHZj
   bG91ZC5mb2p0YS5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC1
   UeaPIsHjRQb8PkybTv7tfCe6oyq8UUxc7tiwX+nWvHWKJ9X6ASis1v/gT0CCa4cG
   fP+tezXayXMrwFSRu6OvanBSTVYvaAbUQl5CsVnQaaeCC5bTAMfGlDsl/q+LnqzW
   i0eG4hTpWG78v88AZkaHjIZdr5CQuDaPGJUqzOgrHjpYTLDJs+oK+S7ScpMyKhia
   hgeRKDfrEeQGGvSEMdHhg3Bg8RK8eyLQLjwCSCVkhYTrM5wyt4fow65beoMnOBbx
   LO+QpB6/amy5mJuVLVx+WJivVvuId2hmELorm2iJMjUAabybLmbMPmqHTTGyZaYw
   vxaDRDr0DbTMUYFyOh6LAgMBAAGjggJHMIICQzAdBgNVHQ4EFgQUi1Dhxpbkz9Dh
   tYOljP+MW/9GF+AwHwYDVR0jBBgwFoAUs0GCJG1KfknG9couJQXq4CZq4SQwgfoG
   A1UdHwSB8jCB7zCB7KCB6aCB5oaBr2xkYXA6Ly8vQ049Zm9qdGEtREMtQ0EoMSks
   Q049REMyLENOPUNEUCxDTj1QdWJsaWMlMjBLZXklMjBTZXJ2aWNlcyxDTj1TZXJ2
   aWNlcyxDTj1Db25maWd1cmF0aW9uLERDPWZvanRhLERDPWNvbT9jZXJ0aWZpY2F0
   ZVJldm9jYXRpb25MaXN0P2Jhc2U/b2JqZWN0Q2xhc3M9Y1JMRGlzdHJpYnV0aW9u
   UG9pbnSGMmh0dHA6Ly9EQzIuZm9qdGEuY29tL0NlcnRFbnJvbGwvZm9qdGEtREMt
   Q0EoMSkuY3JsMIG7BggrBgEFBQcBAQSBrjCBqzCBqAYIKwYBBQUHMAKGgZtsZGFw
   Oi8vL0NOPWZvanRhLURDLUNBLENOPUFJQSxDTj1QdWJsaWMlMjBLZXklMjBTZXJ2
   aWNlcyxDTj1TZXJ2aWNlcyxDTj1Db25maWd1cmF0aW9uLERDPWZvanRhLERDPWNv
   bT9jQUNlcnRpZmljYXRlP2Jhc2U/b2JqZWN0Q2xhc3M9Y2VydGlmaWNhdGlvbkF1
   dGhvcml0eTAhBgkrBgEEAYI3FAIEFB4SAFcAZQBiAFMAZQByAHYAZQByMA4GA1Ud
   DwEB/wQEAwIFoDATBgNVHSUEDDAKBggrBgEFBQcDATANBgkqhkiG9w0BAQsFAAOC
   AQEArNmceh3GkHx5Qf9s/NUlG7lh+i5nPUzlbnlhTJQeOJXSKG3DzU8ocb1xWguT
   x1ICyLZTQq11q3D3/3xgi9KJJaWzo8X5O/Mj81x4V8Jp9d8OgERc7lrVyrAJPbJA
   k7q/4tY41VOu8P5i+A21Pxuo3xELkOt5rcb2qt6QH1QizSA8Dzjq8uwCpdDo8eCP
   ZWUwc2lNUOyCmhFD7boNecHRJZN92i3W0YKfV+BC0cIXnqU2Y+4YEKAHWwH/gRm0
   ZI41oyatyoHpTjCGFtKzrSo/mxitIoj5ZTY9wwSNPlcmziw29hOTM1fOx//rqgrW
   17CREB/BoFj12Hd9bVXFgMGUSg==
   -----END CERTIFICATE-----</MemberEndpointCertificate>
       </OrgAssociationMember>
   </OrgAssociations>
   

   As can be seen I am supplying URI for each Organization, its name and endpoint certificate.

2. I can review organization associations with GET /admin/org/{id}/associations call or I can add or remove single association with similar POST / DELETE calls.
3. Now when the end-user authenticates against the organization that has these associations he can run query calls that will run against all associated organizations. To make the query federated he must add federated=global string to the Accept header.

   GET https://vcloud2.fojta.com/api/query?type=organization
   Accept: application/*;version=9.0;federated=global
   x-vcloud-authorization: 4a1f241b371b46f5a36abac85f5962c7
   

   The reply lists all three organizations:

   <?xml version="1.0" encoding="UTF-8"?>
   <QueryResultRecords xmlns="http://www.vmware.com/vcloud/v1.5" name="organization" page="1" pageSize="128" total="3" href="https://vcloud2.fojta.com/api/query?type=organization&amp;page=1&amp;pageSize=128&amp;format=records" type="application/vnd.vmware.vcloud.query.records+xml" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.vmware.com/vcloud/v1.5 http://vcloud2.fojta.com/api/v1.5/schema/master.xsd">
       <Link rel="alternate" href="https://vcloud2.fojta.com/api/query?type=organization&amp;page=1&amp;pageSize=128&amp;format=references" type="application/vnd.vmware.vcloud.query.references+xml"/>
       <Link rel="alternate" href="https://vcloud2.fojta.com/api/query?type=organization&amp;page=1&amp;pageSize=128&amp;format=idrecords" type="application/vnd.vmware.vcloud.query.idrecords+xml"/>
       <OrgRecord canPublishCatalogs="false" deployedVMQuota="0" displayName="ACME Corporation" isEnabled="true" isReadOnly="false" name="ACME" numberOfCatalogs="1" numberOfDisks="0" numberOfGroups="7" numberOfVApps="1" numberOfVdcs="2" storedVMQuota="0" href="https://vcloud.fojta.com/api/org/02b433db-0b37-4304-b07b-0717255ec297" numberOfRunningVMs="1"/>
       <OrgRecord canPublishCatalogs="false" deployedVMQuota="0" displayName="ACME Inc." isEnabled="true" isReadOnly="false" name="ACME2" numberOfCatalogs="0" numberOfDisks="0" numberOfGroups="1" numberOfVApps="1" numberOfVdcs="1" storedVMQuota="0" href="https://vcloud.fojta.com/api/org/13e52807-3d0a-4c0f-abdb-62d8fccb36ea" numberOfRunningVMs="0"/>
       <OrgRecord canPublishCatalogs="false" deployedVMQuota="0" displayName="Acme Inc." isEnabled="true" isReadOnly="false" name="ACME" numberOfCatalogs="0" numberOfDisks="0" numberOfGroups="1" numberOfVApps="0" numberOfVdcs="0" storedVMQuota="0" href="https://vcloud2.fojta.com/api/org/ca5295f0-a521-4d4c-8b2e-322f154fbbea" numberOfRunningVMs="0"/>
   </QueryResultRecords>
   

Design Considerations

• Only GET requests for the top level Org Level data (org, sessions, tasks, templates/catalogs) and all Query Service requests by Org users can be federated.
• All System Admin query request span single vCloud Director instance and cannot be federated.
• All non-GET requests cannot be federated.
• All associated organizations must use the same identity source as the user is authenticated against all organizations and vCloud Director stores the session tokens for subsequent requests. Local accounts are not supported.
• OAuth identity source offers the lowest administration overhead  when managing multiple organizations belonging to the same end-user as the user/role management can be performed centrally (see vCAT-SP for more information).
• Association works only in one direction: when organization A is associated with organization B it does not mean that organization B is associated with A. Bidirectional association must be explicitly configured for both organization.

Just a short post about an issue with combination of vCloud Director 5.6 and NSX.

In case you are upgrading vCNS to NSX and are using vCloud Director 5.6 you are still affected by the Edge Redeploy bug described in the following two KBs:

• vCloud Director 5.x and VMware NSX for vSphere 6.x interoperability guidelines
• Interoperability considerations for vCloud Director 5.5 with VMware vSphere 6.0, VMware NSX for vSphere 6.1, and vSAN 6.0

The legacy vCNS Edges (version 5.5) are during vCloud Director redeploy operations incorrectly upgraded to NSX Edges (version 6). This breaks compatibility with vCloud Director.

The fix involves adding following entry into vCloud Director database:

Oracle:

INSERT INTO config (config_id, cat, name, value, sortorder) VALUES (seq_config.NextVal, 'vcloud', 'networking.edge_version_for_vsm6.1', '5.5', 0);
commit;

Microsoft SQL Server:

INSERT INTO config (cat, name, value, sortorder) VALUES ('vcloud', 'networking.edge_version_for_vsm6.1', '5.5', 0);


For NSX 6.1 use …vsm6.1, For NSX 6.0 or 6.2 use …vsm6.0 or …vsm6.2 in the string above.

The bug is fixed in builds of vCloud Director 5.5.3 and later and 8.0 and later.

With depletion of public IPv4 addresses service providers are starting to consider offering IPv6 addresses for their tenants workloads. Let me describe what are the options related to IPv6 support for service providers that use vCloud Director.

In the new vCloud Architecture Toolkit (vCAT) Document for Service Providers I have proposed a design how to provide IPv6 services to tenants. So let me summarize the constraints:

• Currently in vCloud Director the tenants cannot assign IPv6 subnets to Org VDC or vApp networks
• In consequence this means that the tenants cannot use vCloud Director IP Address Management (IPAM) to assign IPv6 addresses to their VMs. However, IPv6 addresses can still be assigned from within the guest operating system.
• vCloud Director deployed Edge Gateways do not support IPv6. It means internal or external interfaces of the Edges need to have IPv4 addresses.
• vCloud Director relies for network services on vCloud Networking and Security (vCNS) or NSX components. vCNS does not support IPv6 however NSX does. vCNS will soon go out of support anyway.

The proposed design that works around the above limitations is following. Let me paste the picture from the linked vCAT document:

Provider managed tenant Edge GW

The provider deploys NSX Edge Service Gateway outside of vCloud Director (directly from NSX GUI/API) and connects it to a VXLAN or VLAN based network which is then imported to vCloud Director as an external network. Both the Edge Gateway and the external networks are dedicated to a particular tenant and managed by the provider.

The tenant can attach his workloads to an OrgVDC network which is directly connected to the external network. As this tenant NSX Edge is managed externally outside of vCloud Director scope it can offer full set of services NSX provides – and among them are IPv6 services.

There is one undocumented cool feature that I recently discovered which enables even more IPv6 functionality to the tenant.

There is in fact the possibility for service provider to assign IPv6 subnet to the external network and thus the tenant can use vCloud Director IPAM in a limited way. He can manually assign IPv6 address (IP Mode Static – Manual) to a VM network interface from vCloud Director UI/API and let vCloud Director to configure the VM networking through guest customization. vCloud DIrector even makes sure the IP address is unique.

Note: IP Mode Static – IP Pool is not supported as it is not possible to define IPv6 IP pool.

Here is how to configure IPv6 subnet on external network:

1. Create vCloud DIrector external network (with IPv4 subnet)
2. Find vCloud UUID of the external network. For example use the following API call: GET /api/admin/extension/externalNetworkReferences
3. Insert into vCloud Director database gateway, prefix length, nameservers and dns suffix information. You must create new entries in config table with the following values:
   
   cat = network
   name = ipv6.<ext network UUID>.gateway | subnetprefixlength | nameserver1 | nameserver2 | dnssuffix
   value = <value of the network property>
   
   The following example is valid for MS SQL database:
   
   external network UUID: 85f22674-7419-4e44-b48d-9210723a8e64
   subnet: fd6a:32b6:ab90::/64
   gateway IPv6 address: fd6a:32b6:ab90::1
   DNS 1: fd13:5905:f858:e502::208
   DNS 2: fd13:5905:f858:e502::209
   dns suffix: acme.fojta.com
   
   
INSERT into config values ('network', 'ipv6.85f22674-7419-4e44-b48d-9210723a8e64.dnssuffix', 'acme.fojta.com', 0);
INSERT into config values ('network', 'ipv6.85f22674-7419-4e44-b48d-9210723a8e64.nameserver1', 'fd13:5905:f858:e502::208', 0);
INSERT into config values ('network', 'ipv6.85f22674-7419-4e44-b48d-9210723a8e64.nameserver2', 'fd13:5905:f858:e502::209', 0);
INSERT into config values ('network', 'ipv6.85f22674-7419-4e44-b48d-9210723a8e64.subnetprefixlength', '64', 0);
INSERT into config values ('network', 'ipv6.85f22674-7419-4e44-b48d-9210723a8e64.gateway', 'fd6a:32b6:ab90::1', 0);
4. In the tenant Org VDC create Org VDC network directly connected to the external network.
5. The tenant can now connect VMs to the Org VDC network and assign IPv6 addresses directly from UI (or API).

   Deploy template with IPv6

   VMs with IPv6 Addresses

Note that when using this provider managed Edge Gateway concept, the external network is dedicated to a particular tenant. For scalability reasons it is recommended to use VXLAN based external networks created directly in NSX. vCloud Director supports maximum of 750 external networks.

The tenant cannot directly manage Edge Gateway services and must rely on the provider to configure them.

In vCloud Director 8.0 many enhancements were made to enable unattended installation. This is useful to eliminate manual steps to speed up installation process as well as ensure identical configuration among multiple vCloud Director instances.

Let’s say the provider needs to deploy multiple vCloud Director instances each consisting of multiple cells. Here is the process in high level steps.

Preparation of base template

• Create Linux VM with supported RHEL/CentOS distribution.
• Upload vCloud Director binaries to the VM (e.g. vmware-vcloud-director-8.0.0-3017494.bin)
• Execute the installation file without running the configure script

Prerequisites for each vCloud Director Instance

The following must be prepared for each vCloud Director instance <N>:

• Create database:
  • DB name: vcloudN
  • DB user: vcloudN
  • DB password: VMware1!
• Prepare NFS transfer share
• Create DNS entries, load balancer and corresponding signed certificates for http and consoleproxy and save them to a keystore file certificates.ks. In my example I am using keystore password passwd.

Unattended Installation of the First Cell

• Deploy base template and assign 2 front-end IP addresses. These must match load balancer configuration. e.g. 10.0.2.98, 10.0.2.99
• Mount NFS transfer share to /opt/vmware/vcloud-director/data/transfer
• Upload certificates to /opt/vmware/vcloud-director/etc/certificates.ks
• Run configure script – notice the piping of “Yes” answer to start VCD service after the configuration:
  echo "Y" | /opt/vmware/vcloud-director/bin/configure -cons 10.0.2.98 -ip 10.0.2.99 -dbhost 10.0.4.195 -dbport 1433 -dbtype sqlserver -dbinstance MSSQLSERVER -dbname vcloudN -dbuser vcloudN -dbpassword 'VMware1!' -k /opt/vmware/vcloud-director/etc/certificates.ks -w passwd -loghost 10.0.4.211 -logport 514 -g -unattended

  where 10.0.4.195 is IP address of my MS SQL DB server and 10.0.4.211 syslog server.

• Store /opt/vmware/vcloud-director/etc/responses.properties file created by the initial configuration in a safe place.
• Run initial configuration to create instance ID and system administrator credentials:
  /opt/vmware/vcloud-director/bin/cell-management-tool initial-config --email vcloudN@vmware.com --fullname Administrator --installationid N --password VMware1! --systemname vCloudN --unattended --user administrator
  where N is the installation ID.

Unattended Installation of Additional Cells

vCloud cells are stateless, all necessary information is in vCloud database. All we need is responses.properties file from the first cell that contains necessary encrypted information how to connect to the database.

• Deploy base template and assign 2 front-end IP addresses. These must match load balancer configuration. e.g. 10.0.2.96, 10.0.2.97
• Mount NFS transfer share to /opt/vmware/vcloud-director/data/transfer
• Upload certificates to /opt/vmware/vcloud-director/etc/certificates.ks
• Upload responses.properties file to /opt/vmware/vcloud-director/etc/responses.properties
• Run configure script – notice the piping of “Yes” answer to start VCD service after the configuration:
  echo "Y" | /opt/vmware/vcloud-director/bin/configure -r /opt/vmware/vcloud-director/etc/responses.properties -cons 10.0.2.96 -ip 10.0.2.97 -k /opt/vmware/vcloud-director/etc/certificates.ks -w passwd -unattended

Additional configurations from now on can be done via vCloud API.

vCloud Director based clouds support non-disruptive maintenance of the underlying physical hosts. They can be patched, upgraded or completely exchanged without any impact on the customer workloads all that thanks to vMotion and DRS Maintenance Mode which can evacuate all running, suspended or powered-off workloads from an ESXi host.

Many service providers are going to be upgrading their networking platform from vCloud Network and Security (vCNS) to NSX. This upgrade besides upgrading the Manager and deploying new NSX Controllers requires upgrade of all hosts with new NSX VIBs. This host upgrade results in the need to reboot every host in the service provider environment.

Depending on number of hosts, their size and vMotion network throughput evacuating each host can take 5-10 minutes and reboot can add additional 5 minutes. So for example sequential reboot of 200 hosts could result in full weekend long maintenance window. However, as I mentioned, these reboots can be done non-disruptively without any impact on customers – so no maintenance windows is necessary and no SLA is breached.

So how do you properly reboot all hosts in vCloud Director environment?

While vSphere maintenance mode helps, it is important to properly coordinate it with vCloud Director.

• Before a host is put into a vSphere maintenance mode it should be disabled in vCloud Director which will make sure it does not try to communicate with the host for example for image uploads.
• All workloads (not just running VMs) must be evacuated during the maintenance mode. A customer who decides to power on VM or clone a VM which is registered to a rebooting (and temporarily unavailable) host would be otherwise impacted.

So here is the correct process (omitting the parts that actually lead to the need to reboot the hosts):

1. Make sure that cluster has enough capacity to temporarily run without 1 host (it is very common to have atleast N+1 HA redundancy)
2. Disable host in vCloud Director
3. Put host into vSphere maintenance mode while evacuating all running, suspended and powered-off VMs
4. Reboot host
5. When hosts comes up exit the maintenance mode
6. Enable host
7. Repeat with other hosts

As a quick proof of concept I am attaching a PowerCLI script that automates this. It needs to talk to both vCloud Director and vCenter Server therefore replace Connect strings at the beginning to match your environment.

## Connect to vCloud Director and all vCenter Servers it manages
Connect-CIServer -Server vcloud.gcp.local -User Administrator -Password VMware1!
Connect-VIServer -Server vcenter.gcp.local -User Administrator -Password VMware1!

$ESXiHosts = Search-cloud -QueryType Host
foreach ($ESXiHost in $ESXiHosts) {
	$CloudHost = Get-CIView -SearchResult $ESXiHost
	Write-Host
	Write-Host "Working on host" $CloudHost.Name
	Write-Host "Disabling host in vCloud Director"
	$CloudHost.Disable()
	Write-Host "Evacuating host"
	Set-VMHost $CloudHost.Name -State Maintenance -Evacuate | Out-Null
	Write-Host "Rebooting host"
	Restart-VMHost $CloudHost.Name -Confirm:$false | Out-Null
    Write-Host -NoNewline "Waiting for host to come online "
    do {
		sleep 15
		$HostState = (get-vmhost $CloudHost.Name).ConnectionState
		Write-Host -NoNewline "."
    }
    while ($HostState -ne "NotResponding")
    do {
		sleep 15
		$HostState = (get-vmhost $CloudHost.Name).ConnectionState
		Write-Host -NoNewline "."
    }
	while ($HostState -ne "Maintenance")
	Write-Host
	Write-Host "Host rebooted"
	Set-VMHost $CloudHost.Name -State Connected | Out-Null
	Write-Host "Enabling Host in vCloud Director"
	$CloudHost.Enable()
}

Just a short post to link a new whitepaper I wrote about upgrade of vCloud Networking and Security to NSX in vCloud Director Environment.

It discusses:

• interoperability and upgrade path
• impact of network virtualization technologies (Nexus 1000V,VCDNI)
• migration considerations
• migrations scenario with minimal production impact

VMware vCloud Director® relies on VMware vCloud® Networking and Security or VMware NSX® for vSphere® to provide abstraction of the networking services. Until now, both platforms could be used interchangeably because they both provide the same APIs that vCloud Director uses to provide networks and networking services.
The vCloud Networking and Security platform end-of-support (EOS) date is 19 September 2016. Only NSX for vSphere will be supported with vCloud Director after the vCloud Networking and Security end-of-support date.
To secure the highest level of support and compatibility going forward, all service providers should migrate from vCloud Networking and Security to NSX for vSphere. This document provides guidance and considerations to simplify the process and to understand the impact of changes to the environment.
NSX for vSphere provides a smooth, in-place upgrade from vCloud Networking and Security. The upgrade process is documented in the corresponding VMware NSX Upgrade Guides (versions 6.0 , v6.1 , 6.2 ). This document is not meant to replace these guides. Instead, it augments them with specific information that applies to the usage of vCloud Director in service provider environments.

read more

I had recently multiple discussions about NSX and its Layer 2 bridging capabilities with various service providers. Let me summarize some important points and considerations when you would use which.

Why?

Let’s start with simple question – why would you need layer 2 bridging? Here are some use cases:

• The end-user wants to burst their application to the cloud but wants to keep certain components on-site and because its legacy application it cannot be re-IP’d or requires single subnet communication.
• The service provider is building new cloud offering next to legacy business (collocation, managed services) and wants to enable existing customers to migrate or extend their workloads to the cloud seamlessly (no IP address changes)

How?

NSX offers three ways how to bridge layer two networks.

Layer 2 VPN

This is proprietary VPN solution which enables to create encrypted tunnel across IP networks between Edge Service Gateways that stitches one or more L2 networks. These Edge Gateways can be deployed in different management domains and there is also option of deploying standalone Edge which does not require NSX license. This is great for the cloud bursting use case. I have blogged about L2VPN already in the past here.

While this option is very flexible it is also quite CPU intensive for both the L2 VPN Client and Server Edge VMs. This option provides up to 2Gb throughput.

NSX Native L2 Bridging

L2 bridge is created in the ESXi VMkernel hypervisor by deploying a Logical router control VM. The control VM is used only for the bridge configuration and its pinning to a particular ESXi host. As the bridging happens in the VMkernel it is possible to achieve impressive line rate (10 Gb) throughput.

It is possible to bridge only VXLAN based logical switch with VLAN based port group. The same physical uplink must be utilized so this means that the VLAN port group must be on the same vSphere Distributed Switch (vDS) that is prepared with the VXLAN VTEP and where the VXLAN logical switch portgroups are created.

VLAN and VXLAN portgroups are on the same vDS

The above fact prohibits a scenario where you would have Edge Cluster with multiple pairs of uplinks connected to separate vDS switches. One for VLAN based traffic and the other for VXLAN traffic. You cannot create NSX native L2 bridge instance between two vDS switches.

This important especially for the collocation use case mentioned at the beginning. In order to use the L2 bridge the customer VLAN must be connected to the Edge Cluster Top of the Rack pair of switches.

If this is not possible, as a workaround the service provider can use the L2 VPN option – it is even possible to run both L2 VPN Server and Client Edges on the same host connected through a transit VXLAN network where one Edge is connected to trunk with VLAN networks from one vDS and the other to trunk with VXLAN networks on another vDS. Unfortunately this has performance impact (even if NULL-MD5 encryption is used) and should be used only for temporary migration use cases.

Hardware VTEP

The last bridging option discussed is a new feature of NSX  6.2. It is possible to extend the VXLAN logical switch all the way to a compatible hardware device (switch from a VMware partner) that acts as Layer 2 gateway and bridges the logical switch with a VLAN network. The device performing the function of hardware VTEP is managed from NSX via OVSDB protocol while the control plane is still managed by NSX Controllers. More details are in the following white paper.

As this option requires new dedicated and NSX compatible switching hardware it is more useful for the permanent use cases.

New vCloud DIrector 8.10 (read eight dot ten) is out and with it some little neat features. Let me quickly talk about one of them – the ability to run vCloud Director cell with just 1 IP address.

In the past you always had to configure vCloud Director cell at least with two IP addresses. One for the web interface (providing UI and API) and the other for remote console proxy. The reason was that both services shared the same port 443. In vCloud Director 8.10 there is possibility to specify ports for each service and thus use just one IP address. This helps if your DMZ subnet is too small and you need to deploy more VMs into that network (more cells, databases, etc.).

Note that the configure script will not ask you for ports, instead you need to use unattended installation option or add port entries afterward in global.config file.

Unattended Installation

Here is the example of configure parameters that sets console proxy to the same IP address as http (10.0.1.60) and uses port 8443 instead of the standard 443:


/opt/vmware/vcloud-director/bin/configure" -cons 10.0.1.60 --console-proxy-port-https 8443 -ip 10.0.1.60 --primary-port-http 80 –-primary-port-https 443 -dbhost 10.0.4.195 -dbport 1433 -dbtype sqlserver -dbinstance MSSQLSERVER -dbname vcloud -dbuser vcloud -dbpassword 'VMware1!' -k /opt/vmware/vcloud-director/etc/certificates.ks -w 'passwd' -loghost 10.0.4.211 -logport 514 -g --enable-ceip true -unattended


Global Properties

An alternative option is to edit the /opt/vmware/vcloud-director/etc/global.properties file and add new port entries:

Before:


...
product.version = 8.10.0.3879706
product.build_date = 2016-05-12T20:32:07-0700
vcloud.cell.ip.primary = 10.0.1.60
consoleproxy.host.https = 10.0.1.61
...


After


...
product.version = 8.10.0.3879706
product.build_date = 2016-05-12T20:32:07-0700
vcloud.cell.ip.primary = 10.0.1.60
consoleproxy.host.https = 10.0.1.60
consoleproxy.port.https = 8443
vcloud.http.port.standard = 80
vcloud.http.port.ssl = 443
...


Do not forget to reconfigure your loadbalancer remote console pool to point to the new IP-port combination.

Another nice little feature in the recently released vCloud Director 8.10 (read eight dot ten) is the ability to import running VM under vCloud Director management.

In the past the vCloud system administrator could import virtual machine running in vCloud Director managed vSphere environment, however that VM had to be turned off.

Now in vCloud DIrector 8.10 the VM can be running which enables new use cases:

• You can take existing vSphere environment under management of vCloud Director without impacting the workloads. Example would be going from vSphere only managed services to self service with vSphere + vCloud Director
• Migrations of running VMs from vSphere to vCloud Director or between vCloud Directors. Cross vCenter Server vMotion nicely complements this feature. Cross-Cloud vMotion.

There are currently some limitations and considerations:

• It is API only feature. The GUI button (above) can be used to import only powered-off VMs.
• VM must be running on resources that are available to the Org VDC where it is being imported. That includes compute (cluster), storage policies and network port group. Obviously the networking is the most complex to get it right during the migration and requires some thought.
• Import of VM with IDE controller is not supported.
• A new vApp is created for the imported VM. You cannot import VM to an existing vApp.
• As an input to the API call MoRef ID of the VM must be provided. The existing vCloud API call to list vSphere VMs outside of vCloud Director management however does not list running VMs. Therefore the MoRef ID must be acquired with vSphere API.

Example:

POST https://vcd-01a.corp.local/api/admin/extension/vimServer/3702cc8f-c23f-4a66-b9f3-73fc4b58ba82/importVmAsVApp

Accept: application/*+xml;version=20.0
x-vcloud-authorization: 3e131f9e3bc240269a7758fdb6c1bf7f
Content-type: application/vnd.vmware.admin.importVmAsVAppParams+xml


<?xml version="1.0" encoding="UTF-8"?>
<ImportVmAsVAppParams xmlns="http://www.vmware.com/vcloud/extension/v1.5" name="ImportedTestVM" sourceMove="true">
   <VmMoRef>vm-86</VmMoRef>
   <Vdc href="https://vcd-01a.corp.local/api/vdc/a10bdf92-18dc-474b-aafc-42d31ba83207" />
</ImportVmAsVAppParams>

Very useful new feature of vCloud Director 8.10 is the possibility to query with vCloud API guest OS customization status. Typical use case is when the tenant runs custom orchestration to deploy VM and install and configure application in it. When the VM is powered-on for the first time, the operating system boots up and vCloud Director runs customization scripts to set identity (hostname, SID), networking, administrator password, etc. Read Massimo’s blog for deep dive into guest cutomization.

Tenant’s custom orchestration then needs to wait for the customization to finish and then finally log into the VM and proceed with the application installation and configuration. The problem in the past was that there was no easy way to find out if the guest customization was finished. Not anymore.

With vCloud API you can easily query the vApp VM to get its customization status.

Example:

GET https://vcd-01a.corp.local/api/vApp/vm-22e51563-52a6-4a13-961a-d9dffa6aabf5/guestcustomizationstatus

<?xml version="1.0" encoding="UTF-8"?>
<GuestCustomizationStatusSection xmlns="http://www.vmware.com/vcloud/v1.5" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.vmware.com/vcloud/v1.5 http://vcd-01a.corp.local/api/v1.5/schema/master.xsd">
	    <GuestCustStatus>GC_PENDING</GuestCustStatus>
</GuestCustomizationStatusSection>

The possible guest customization states are:

• GC_PENDING: Guest Customization is either not started or in progress.
• REBOOT_PENDING: Guest Customization is successful, but reboot is pending.
• GC_FAILED: Guest Customization failed, error is logged.
• POST_GC_PENDING: Reboot has happened, waiting for post customization script to complete
• GC_COMPLETE: Guest customization is complete