PowerCLI: Find VMs with xHCI controller

The ESXi vulnerability found at the 2020 Tianfu Cup was a Critical one, with a CVSSv3 base score of 9.3.

VMware lists an article with the fixes and workarounds here:
https://www.vmware.com/security/advisories/VMSA-2020-0026.html
The fix is to apply the latest patch, and the workaround is to remove the xHCI (USB 3.0) controller from any VMs that have it.

To quickly determine whether you have an exposure you can run the following PowerCLI against your environment and it will list the VMs which have that particular controller type attached.

Get-VM | ?{$_.ExtensionData.Config.Hardware.Device.DeviceInfo.Label -match "xhci"}

vRealize Orchestrator Name/IP lookups

I’ve started looking at upgrading our standalone VRO instances from 7.x to 8.x, and one thing that has changed significantly is that we can no longer use the appliance linux environment to run dig or nslookup

There are a couple of System calls:

System.resolveHostName(hostname);
System.resolveIpAddress(ip);

These allow the usual forward and reverse lookups, but have significant limitations

System.resolveHostName :

  • Only returns one record at a time, so if there are multiple records you would have to write a loop to collect them all
  • Only returns the IP address, no ability to return record type, TTL, SOA record

System.resolveIpAddress :

  • Only returns one record at a time, so if there are multiple records you would have to write a loop to collect them all
  • Only returns the host name, no ability to return record type, TTL, SOA record
  • ONLY WORKS IF A FORWARD RECORD EXISTS THAT MATCHES

This final point took some significant figuring out, and in combination with the rest of the points resulted in me having to change some of the workflows to do an SSH to a linux server to do normal dig and nslookup commands, rather than using the System calls.

Windows: Converting PEM and PKey to PFX file

I’ve just been working on our Chef recipe that installs a CA signed cert for the Windows RDP service. Originally I wrote the recipe to interact with Windows PKI, but it was a bit clunky and I was never really happy with it.

With Hashicorp Vault now being available in our environment I started looking at migrating over to using this, as the integration with Chef was far superior. However, the one stumbling block I came across was that Vault would export the certificate as a PEM file and a Private Key file, whereas Windows could only install and use this pair as a PFX file. A couple of utilities have been available for a while for doing the conversion, either openssl or Pvk2Pfx but I’ve always shied away from installing new software if at all possible to simplify maintenance.

Fortunately I’ve discovered that certutil now has an option to do this conversion ‘-MergePFX’

Simply put the certificate and key in the same folder with the same name (but different extensions), such as rdpcert.cer and rdpcert.key and run:
certutil -MergePFX rdpcert.cer rdpcert.pfx
and this will combine the files into a PFX file for import.

OT: Migrating Windows from SATA to NVMe

Totally off topic from what I normally blog, but thought it might be useful to some people.

I’ve just migrated my home PC from a SATA SSD to an NVMe drive, and had some fun as there were some things I didn’t know about when I started.
Mostly this was because I was going from an old style MBR partition table, to a GPT/EFI setup

  1. Make sure your PC is capable of booting NVMe
    Mine wasn’t (although I had done some research on this before I started. Running an Asus P8Z77-V LK which doesn’t have an M.2 slot at all, led me to see whether it was possible to add a PCIe adapter. This *is* possible, however the BIOS doesn’t support NVMe boot out of the box, and a modified one is needed. See this forum post for more details
  2. Purchase a PCIe NVMe adapter if you need one. I bought one of these very cheaply
  3. Purchase an NVMe SSD. I bought a Samsung 970 Evo Plus as they have good performance and reputedly better legacy boot support. I’m not sure if that made a difference to the migration.
  4. Create a Windows Recovery/Boot USB and a GParted Boot USB
  5. Install the hardware, making sure the PCIe adapter is in a slot supporting x4 lanes
  6. Make sure your PC is not in the middle of doing some updates. Mine was and this seemed to screw over the registry somehow, requiring a lot of wailing and gnashing of teeth before a partial recovery point restore got things working again
  7. Shut down the PC and boot from the GParted Boot USB
  8. Create a GPT partition table on the new drive
  9. Create a 100MB EFI partition and a 128MB MSFT partition at the start of the drive with FAT32. Mark the EFI partition as ‘Boot’
  10. Clone the Windows partition from the SSD to the new drive
  11. Power down the PC and disconnect the old SATA drive
  12. Switch back on with the Windows Boot USB connected
  13. Use this post to build the EFI boot configuration (I did a lot of this the hard way….)
  14. Remove the USB and go into the BIOS and temporarily turn off Secure Boot, and configure the necessary setting for the PC to boot UEFI from the PCIe storage device
  15. Boot up – it may fail and have to boot into safe mode once to sort out the boot device in windows
  16. Once it is booting into windows successfully, go back into the BIOS and turn Secure Boot back on.

vRealize Orchestrator REST API Inconsistency

I’ve been updating one of our Jenkins jobs (which drives a vRO flow) to call the REST API directly, rather than shelling out to run a script.

As part of this I started looking at retrieving the full workflow log, rather than just the events list, but came up against an error

The API reference shows the method as being:

However when trying this you get a ‘405’ error:

Digging into the server.log on the vRO appliance I found the following message:
WARN {} [PageNotFound] Request method 'GET' not supported
so I tried changing this to a POST on the off-chance.. and voila:

I’ve raised an SR with VMware Support to see if I can get this fixed or the documentation updated

UPDATE – I had a response to say that the syslog call is only supported from 7.6 onwards, so maybe I’m lucky that it works at all in 7.5!

Deploying a VM from a Content Library OVF/OVA template

Following on from my series of posts on uploading OVA files to Content Library, and searching content library, the final step of all of this would be to deploy a VM using one of these templates

Inputs Required

The list of inputs that we need is as follows

NameTypeDescription
datastoreVC:DatastoreThe datastore to deploy to
endpointVAPI:VAPIEndpointThe VAPI endpoint to use
folderVC:VmFolderThe folder to deploy to
hostVC:HostSystemThe target host for deployment
hostnamestringName of the new VM
ovfLibraryItemIdstringid of the content library item
respoolVC:ResourcePoolResource pool to deploy to

Setup

Similar to previous steps, the first thing you need to do is make a connection to the VAPI endpoint
var client = endpoint.client();

Next, we need to create a model for the deployment target – where are we deploying the VM to? We use the deployment target object for this:
var deploymentTarget = new com_vmware_vcenter_ovf_library__item_deployment__target();
deploymentTarget.folder_id = folder.id;
deploymentTarget.host_id = host.id;
deploymentTarget.resource_pool_id = pool.id;

The final step before we deploy the VM is to create a resource pool deployment spec, this include the name, and the datastore to use:
var resourcePoolDeploymentSpec = new com_vmware_vcenter_ovf_library__item_resource__pool__deployment__spec();
resourcePoolDeploymentSpec.accept_all_EULA = true;
resourcePoolDeploymentSpec.name = hostname;
resourcePoolDeploymentSpec.default_datastore_id = datastore.id;

Quite why the datastore is in that one rather than the deployment target is a mystery to me. The only thing I can think of is that the deployment spec includes the storage provisioning profile.

Deploying the VM

To deploy the new VM, we use the com_vmware_vcenter_ovf_library__item call, and pass the ovfLibraryItemId, with the deployment target and deployment specs that we created above
var ovfSvc = new com_vmware_vcenter_ovf_library__item(client);
var result = ovfSvc.deploy(null, ovfLibraryItemId, deploymentTarget, resourcePoolDeploymentSpec);

Returning the vmObject

The deploy method above returns a structure that includes the id of the VM which we can use to find the vmObject, which is an output item of type VC:VirtualMachine
vmObject = VcPlugin.getAllVirtualMachines(null, "xpath:matches(id, '" + result.resource_id.id + "')")[0];
The search returns an array, so I’m using the [0] to strip the object out of the array – searching on an id, there can only be one VM with a particular id unless you’re using linked mode .
Finally we close the connection to the endpoint
client.close();

Full code listing

// Set the VAPI endpoint to the first endpoint returned

if (endpoint == null) {  
  throw "Unable to locate a VAPI endpoint";
}
var client = endpoint.client();  

// create a DeploymentTarget  
var deploymentTarget = new com_vmware_vcenter_ovf_library__item_deployment__target();

deploymentTarget.folder_id = folder.id;
deploymentTarget.host_id = host.id;
deploymentTarget.resource_pool_id = pool.id;

// create a ResourcePoolDeploymentSpec  
var resourcePoolDeploymentSpec = new com_vmware_vcenter_ovf_library__item_resource__pool__deployment__spec();
resourcePoolDeploymentSpec.accept_all_EULA = true;
resourcePoolDeploymentSpec.name = hostname;
resourcePoolDeploymentSpec.default_datastore_id = datastore.id;
  
// deploy the ovf  
var ovfSvc = new com_vmware_vcenter_ovf_library__item(client);
var result = ovfSvc.deploy(null, ovfLibraryItemId, deploymentTarget, resourcePoolDeploymentSpec); 

// return the vmObject
vmObject = VcPlugin.getAllVirtualMachines(null, "xpath:matches(id, '" + result.resource_id.id + "')")[0];
client.close();

VRO – Listing out items from Content Library

Shortly after starting a series of posts on using VRO to upload templates to content library, I was asked how you would retrieve a list of content library items.

It isn’t quite as straightforward as you might think as the List() function only provides a list of IDs rather than full details. However armed with this knowledge, you can get the list of IDs and use that to iterate through and produce a full list of items.

There are four main sources that I’ve been using for the API information
(the links here are direct to the Content Library section)

Overview

The steps we need to go through here are

  • Find and connect to an endpoint
  • Find the content library(s) on the endpoint and connect
  • Get a list of items in the content library
  • For each item in the list, get its details
  • Return the array of items with their full details

Find and connect to an endpoint

The code for this uses the VAPI plugin with the getAllEndpoints() method. An array of all configured endpoints on the VRO instance is returned. We do a quick check to ensure we have something in the array before using it with a for each command

var endpoints = VAPIManager.getAllEndpoints();
var endpoint = endpoints[0]
if (endpoint == null) {
throw "Unable to locate a VAPI endpoint";
}
var ovfLibraryItem = new Array();
for each(var endpoint in endpoints){

System.log("Searching endpoint " + endpoint);
var client = endpoint.client();
...

You can see there that I’ve also initialised the array in which the list of library items will be stored.
The connection to the endpoint is handled by the client() method

Find the content library(s) on the endpoint and connect

Firstly we get a list of the content libraries on the endpoint, with the com_vmware_content_library call. The list() method is used to return an array of their identifiers

var clib = new com_vmware_content_library(client);
System.log("The number of libraries on this endpoint is: " + clib.list().length);
System.log(clib.list());
if(clib.list().length >= 1){

Get a list of items in the content library

To query items in the library, we need to create an object with the com_vmware_content_library_item call

var itemSvc = new com_vmware_content_library_item(client);

Once we’ve created that, we can iterate through the list of content libraries and get a list of items within that library. The list() method is used again here to get an array of items.
for each(var clibrary in clib.list()){
var items = itemSvc.list(clibrary);
System.log(items);

For each item in the list, get its details

This just iterates through the array of items, and uses the get() method on com_vmware_content_library_item to get the detail for each item in turn. The results are pushed into the ovfLibraryItem array that we created near the start.
for each(item in items) {
var results = itemSvc.get(item);
System.log(results.name);
ovfLibraryItem.push(results);
}

Return the array of items with their full details

This is as simple as making the ovfLibraryItem array an output from the scriptable task and the workflow, remembering to close down the endpoint client at the end of the script

Putting it all together

The full code for this comes together like this

// Set the VAPI endpoint to the first endpoint returned
var endpoints = VAPIManager.getAllEndpoints();  
var endpoint = endpoints[0]

if (endpoint == null) {  
  throw "Unable to locate a VAPI endpoint";
}
var ovfLibraryItem = new Array();

for each(var endpoint in endpoints){
  System.log("Searching endpoint " + endpoint);
  var client = endpoint.client();  
  var clib = new com_vmware_content_library(client);  
  System.log("The number of libraries on this endpoint is: " + clib.list().length);
  System.log(clib.list());

  if(clib.list().length >= 1){
    var itemSvc = new com_vmware_content_library_item(client);

    for each(var clibrary in clib.list()){
      var items = itemSvc.list(clibrary); 
      System.log(items);

      for each(item in items) {
        var results = itemSvc.get(item); 
        System.log(results.name);
        ovfLibraryItem.push(results);
      }
    }
  }
  client.close();
}

No inputs are required, and the only output is ovfLibraryItem which is an array of VAPI:com_vmware_content_library_item__model


Uploading VM Template OVAs to Content Library with vRealize Orchestrator – Step 3

This is following on from these posts:
Uploading VM template OVAs to Content Library with vRealize Orchestrator
Uploading VM template OVAs to Content Library with vRealize Orchestrator – Step 2
where we learned how to search for an item in Content Library, and how to create a new item in Content Library if there isn’t already one there.

The final step is to attach some content, or to update the content, for the chosen Content Library item, both of which work in the same way.

Creating an Update Session in Content Library

Again, we will need to set up the connection to the VAPI endpoint :
var client = endpoint.client(); 
The update session is created with
var updSess = new com_vmware_content_library_item_update__session(client);
and the item update model is created with
var updSessModel = new com_vmware_content_library_item_update__session__model();

We then need to configure the item update model and use it to start the update session
updSessModel.library_item_id = ovfLibraryItemId;
var client_token = System.nextUUID();
var updateid = updSess.create(client_token,updSessModel);

The next step is to specify the file resource and get the content service to retrieve it

Creating the File Update Resource

The update file object is created with:
var fileupd = new com_vmware_content_library_item_updatesession_file(client);
Next we need to create a file add specification:
var filespec = new com_vmware_content_library_item_updatesession_file_add__spec();
We then flesh this out with the type of transfer we’re doing. Here it’s a PULL rather than a PUSH as we have a URL to retrieve the OVAs from:
var filetype = "PULL";
filespec.source_type = filetype;

And to specify the URL, we create a transfer endpoint object, and configure it with the constructed URL (the url here is a workflow attribute, as it’s static in our environment)
var source_endpoint = new com_vmware_content_library_item_transfer__endpoint();
source_endpoint.uri = url+templateName+".ova";
filespec.source_endpoint = source_endpoint;
filespec.name = templateName+".ova";

The suffix on filespec.name is important, as it’s how the Content Library import process knows what kind of import it’s dealing with. You could do 2 updates here to load an OVF and a VMDK if you’re importing OVFs rather than an OVA
The completed filespec is then added to the file update session
var addresult = fileupd.add(updateid,filespec);

Monitoring the upload

You can issue a complete at this point, and move on, but it is possible to monitor the upload for completion, which is preferable so we can monitor for failures
A simple loop around a ‘get’ function is used here
var result = "";
var loop=0;
do {
result = fileupd.get(updateid,filespec.name);
System.log("Status: "+result.status);
System.sleep(10*1000);
loop += 1;
if (loop == 60) { break;success=false; errorText="Failed to upload image";}
}
while (result.status != "READY");

Completing the Update Session

The command to commit the update session is:
var result = updSess.complete(updateid);
Again we monitor this with a loop around a ‘get’ function to monitor this until completion before closing down the endpoint client session
client.close();

The full code for the update then looks like this

// Create new update session
System.log("start update session");
var client = endpoint.client(); 
var updSess = new com_vmware_content_library_item_update__session(client);
var updSessModel = new com_vmware_content_library_item_update__session__model();
updSessModel.library_item_id = ovfLibraryItemId;
client_token = System.nextUUID();
var updateid = updSess.create(client_token,updSessModel);

// Create file update resources
var fileupd =  new com_vmware_content_library_item_updatesession_file(client);
var filespec = new com_vmware_content_library_item_updatesession_file_add__spec();
var source_endpoint = new com_vmware_content_library_item_transfer__endpoint();
source_endpoint.uri = url+templateName+".ova";
filespec.source_endpoint = source_endpoint;
var filetype = "PULL";
filespec.source_type = filetype;
filespec.name = templateName+".ova";
var addresult = fileupd.add(updateid,filespec);
System.log("File add: " + addresult);

// loop until transfer is complete
var result = "";
var loop=0;
do {
  result = fileupd.get(updateid,filespec.name);
  System.log("Status: "+result.status);
  System.sleep(10*1000);
  loop += 1;
  if (loop == 60) { break;success=false; errorText="Failed to upload image";}
}
while  (result.status != "READY");
  
// loop until template update is complete
var result = updSess.complete(updateid); 
var loop=0;
do {
  System.sleep(10*1000);
  loop += 1;
  result = updSess.get(updateid);
  if (loop == 60) { break;success=false; errorText="Failed to complete template update"}
}
while  (result.state != "DONE");
System.log("Update session complete: version=" + result.library_item_content_version + ", state=" + result.state);
client.close();

Uploading VM Template OVAs to Content Library with vRealize Orchestrator – Step 2

This is following on from https://richdowling.wordpress.com/2019/07/05/uploading-vm-template-ovas-to-content-library-with-vrealize-orchestrator/ where we checked for the existence of a named OVF template in the Content Library

In the workflow, I check the result of the “Find” step by examining the output attribute ovfLibraryItemId – if it’s null or zero length, we know that we need to create the item.

Creating a new item in Content Library

First, we will need to set up the connections to the VAPI endpoint and content libary – we use the endpoint that was passed through from the previous step:
var client = endpoint.client();
var clib = new com_vmware_content_library(client);

Next we have to create an object with the content library item model:
var itemmodel= new com_vmware_content_library_item__model();

and populate it with the appropriate values:
var type = "ovf";
itemmodel.type = type;
itemmodel.library_id = clib.list()[0];
itemmodel.name = templateName;
itemmodel.description = "Autogenerated template";


It should be possible to look up the valid item type and select from the list, however I couldn’t get this to work through VRO, it seemed as though the appropriate API call hadn’t been implemented through the existing VAPI plugin.
Note that I’ve chosen to just pick the first Content Library here – clib.list()[0] as we only have one per site at present, and I couldn’t think of a sensible way to choose programmatically when there’s only one.

After generating an item model, we have to apply it to the Content Library with a create item call:
var client_token = System.nextUUID();
var newitem = new com_vmware_content_library_item(client);
ovfLibraryItemId = newitem.create(client_token,itemmodel);

The client_token is a unique identifier to distinguish between concurrent updates. We could probably do with a little more validation at this step…

Finally we close down the connection with:
client.close();
and pass out the ovfLibraryItemId as an output attribute

// Create new item in Content Library
var client = endpoint.client(); 
var clib = new com_vmware_content_library(client); 

// Generate an item model for the content library item
var itemmodel= new com_vmware_content_library_item__model()
var type = "ovf";
itemmodel.type = type;
itemmodel.library_id = clib.list()[0];
itemmodel.name = templateName;
itemmodel.description = "Autogenerated template";


// Generate a random token for the update and create the content library item
client_token = System.nextUUID();
var newitem = new com_vmware_content_library_item(client);
var ovfLibraryItemId = newitem.create(client_token,itemmodel);

// Close the connection
client.close();

The next step will be uploading the content to the Library item.

Uploading VM Template OVAs to Content Library with vRealize Orchestrator

VRO doesn’t currently include functionality to interact with Content Library, so to do so we have to invoke the VAPI calls ourselves. I’ve just been through this process, so I’m going to document the process in case anyone else needs to do the same.

There are 3 main steps involved here:

  1. Check to see if the template already exists, and if it does save the ID and move to step 3
  2. Create the template object if it’s not already there
  3. Upload the OVA file to the template object and apply
workflow overview

Step 1: Finding the template object in Content Library

The only thing we pass in to this step is the name of the template templateName – an example here would be centos7
First we have to determine the VAPI Endpoint with
VAPIManager.getAllEndpoints();
This returns an array of endpoints, and in the environment I work in, there’s only one endpoint per VRO server, which does make things simpler, however I then do some checking :
var endpoints = VAPIManager.getAllEndpoints();
var endpoint = endpoints[0]
if (endpoint == null) {
throw "Unable to locate a VAPI endpoint";
} else {
targetendpoint=endpoint;
}

This will bomb the workflow out if it can’t locate a VAPI endpoint to work with.

Next we iterate through the endpoints looking for content libraries:
for each(var endpoint in endpoints){
System.log("Searching endpoint " + endpoint);
var client = endpoint.client();
var clib = new com_vmware_content_library(client);
System.log("The number of libraries on this endpoint is: " + clib.list().length);
...

Once we’ve found an endpoint with at least one content library, we can search the endpoint for content library objects that match the name of the template we want to create or update
if(clib.list().length >= 1){
var itemSvc = new com_vmware_content_library_item(client);
var findItemSpec = new com_vmware_content_library_item_find__spec();
findItemSpec.name = templateName;
var results = itemSvc.find(findItemSpec);
...

If we don’t get a valid result it means it’s not found on that endpoint, and we can move on. If we do, we can get the full details of that item, and store what we need for the next step

if (!Array.isArray(results) || !results.length) {
System.log("Template not found on this endpoint");
} else {
success=true;
var details = itemSvc.get(results[0]);
System.log("Content Library template " + templateName + " found " + (results[0]) + " " + details.type);
System.log(details);
targetendpoint = endpoint;
ovfLibraryItemId = results.shift();
}

targetendpoint and ovfLibraryItemId are passed as an output to the next step

We close the connection to the endpoint with
client.close();

The code in it’s entirety is as follows:

var endpoints = VAPIManager.getAllEndpoints();  
var endpoint = endpoints[0]
if (endpoint == null) {  
   throw "Unable to locate a VAPI endpoint";
 } else {
   targetendpoint=endpoint;
 }

 for each(var endpoint in endpoints){
   System.log("Searching endpoint " + endpoint);
   var client = endpoint.client();  
   var clib = new com_vmware_content_library(client);  
   System.log("The number of libraries on this endpoint is: " + clib.list().length);
   if(clib.list().length >= 1){
     var itemSvc = new com_vmware_content_library_item(client);
     var findItemSpec = new com_vmware_content_library_item_find__spec();
     findItemSpec.name = templateName;
     var results = itemSvc.find(findItemSpec); 

     if (!Array.isArray(results) || !results.length) {
       System.log("Template not found on this endpoint");
     } else {
       var details = itemSvc.get(results[0]);
       System.log("Content Library template " + templateName + " found " + (results[0]) + " " + details.type);
       System.log(details);
       targetendpoint = endpoint;
       ovfLibraryItemId = results.shift();
    }
   }
   client.close();
}

That’s enough for one post, in the next one I’ll show how we create the Content Library item if it’s not already there.