VCSA 7.0U2 Upgrade failure – amcheck_next

We’re currently running through the 7.0U2 upgrades, and have encountered this on one of them. It failed at 92% during the data conversion step, and as part of the diagnostics, support had us run a postgres consistency check script: https://kb.vmware.com/s/article/53062

The script does many checks, but it also creates a postgres extension called amcheck_next.

This caused subsequent upgrade attempts to fail – the message appears in the log /var/log/vmware/applmgmt/PatchRunner.log

2021-05-27T06:56:41.087Z  Running: su -s /bin/bash vpostgres -c cd /var/log/vmware/vpostgres/upgrade && /opt/vmware/vpostgres/12/bin/pg_upgrade -U postgres -B /opt/vmware/vpostgres/12/bin -b /opt/vmware/vpostgres/12/../11/bin -D /storage/db/vpostgres.12 -d /storage/db/vpostgres --check
2021-05-27T06:56:41.087Z  Running command: ['su', '-s', '/bin/bash', 'vpostgres', '-c', 'cd /var/log/vmware/vpostgres/upgrade && /opt/vmware/vpostgres/12/bin/pg_upgrade -U postgres -B /opt/vmware/vpostgres/12/bin -b /opt/vmware/vpostgres/12/../11/bin -D /storage/db/vpostgres.12 -d /storage/db/vpostgres --check']
2021-05-27T06:56:42.140Z  Done running command
, stderr: 2021-05-27T06:56:42.141Z  pg_upgrade --check returned error code 1 with error  out Performing Consistency Checks
-----------------------------
Checking cluster versions                                   ok
Checking database user is the install user                  ok
Checking database connection settings                       ok
Checking for prepared transactions                          ok
Checking for reg* data types in user tables                 ok
Checking for contrib/isn with bigint-passing mismatch       ok
Checking for tables WITH OIDS                               ok
Checking for invalid "sql_identifier" user columns          ok
Checking for presence of required libraries                 fatal
 
Your installation references loadable libraries that are missing from the
new installation.  You can add these libraries to the new installation,
or remove the functions using them from the old installation.  A list of
problem libraries is in the file:
    loadable_libraries.txt
 
Failure, exiting
 
.
2021-05-27 06:56:42,169.169Z vpostgres:Patch ERROR vmware_b2b.patching.executor.hook_executor Patch hook 'vpostgres:Patch' failed.

Checking the “loadable_libraries.txt” in ‘/storage/log/vmware/vpostgres/upgrade’ showed the following:

could not load library "$libdir/amcheck_next": ERROR:  could not access file "$libdir/amcheck_next": No such file or directory
Database: VCDB

To rectify this, after rolling back to the pre-upgrade snapshot, I just connected to the DB with ‘psql -U postgres -h localhost -d VCDB’ and ran ‘DROP EXTENSION amcheck_next;’

I then cleared out the VUM database (which had cause the original failure) and the update then ran through successfully.

‘Data transfer and appliance setup is in progress’ after VCSA upgrade backout

This is as much a reminder for me as for anyone else, having come across this a couple of times and always struggled to find the KB article on how to resolve it

If you perform a migration upgrade on a VCenter Server Appliance, such as from 6.7 to 7.0, and the upgrade fails on the new appliance necessitating a backout, it is common to get a ‘data transfer and appliance setup is in progress’ message on the original appliance

There is a KB article to walk you through resolving this: https://kb.vmware.com/s/article/67179

In particular the final steps where you ‘touch’ three files in the /var/log/vmware/upgrade directory seem to be the main requirement in my experience

Once this has been performed, a simple refresh of the page will return you to a normal login interface

PowerCLI – Disabling ESXi OpenSLP service for VMSA-2021-0002

OpenSLP has cropped up again as an ESXi vulnerability, and if you want to disable the service the KB article given only has details for doing so via the ESXi command line.

Far easier, if you have many hosts, is to use PowerCLI, and while it’s relatively simple I thought I would share this to help anyone else wanting to do so.

Disabling the service
Connect to the environment with ‘connect-viserver’ and then run:

Get-VMHost | %{
	$_ | Get-VMHostFirewallException -Name "CIM SLP" | Set-VMHostFirewallException -Enabled:$false
	Stop-VMHostService -HostService ($_ | Get-VMHostService | ?{$_.Key -eq "slpd"}) -Confirm:$false
	$_ | Get-VMHostService | ?{$_.key -match "slpd"} | Set-VMHostService -Policy "off"
}

Checking the status
Connect to the environment with ‘connect-viserver’ and then run:

Get-VMHost | %{
	$rule = $_ | Get-VMHostFirewallException -Name "CIM SLP"
	$serv = $_ | Get-VMHostService | ?{$_.Key -eq "slpd"}
	$_ | select Name,@{N="Rule";E={$rule.enabled}},@{N="ServiceRunning";E={$serv.Running}},@{N="ServiceEnabled";E={$serv.Policy}}
}

Edit : As per the comment from Zeev, I’d missed disabling the service, I’ve updated the Disabling and Checking scripts above to include the correct information now.

PowerCLI: Find VMs with xHCI controller

The ESXi vulnerability found at the 2020 Tianfu Cup was a Critical one, with a CVSSv3 base score of 9.3.

VMware lists an article with the fixes and workarounds here:
https://www.vmware.com/security/advisories/VMSA-2020-0026.html
The fix is to apply the latest patch, and the workaround is to remove the xHCI (USB 3.0) controller from any VMs that have it.

To quickly determine whether you have an exposure you can run the following PowerCLI against your environment and it will list the VMs which have that particular controller type attached.

Get-VM | ?{$_.ExtensionData.Config.Hardware.Device.DeviceInfo.Label -match "xhci"}

vRealize Orchestrator Name/IP lookups

I’ve started looking at upgrading our standalone VRO instances from 7.x to 8.x, and one thing that has changed significantly is that we can no longer use the appliance linux environment to run dig or nslookup

There are a couple of System calls:

System.resolveHostName(hostname);
System.resolveIpAddress(ip);

These allow the usual forward and reverse lookups, but have significant limitations

System.resolveHostName :

  • Only returns one record at a time, so if there are multiple records you would have to write a loop to collect them all
  • Only returns the IP address, no ability to return record type, TTL, SOA record

System.resolveIpAddress :

  • Only returns one record at a time, so if there are multiple records you would have to write a loop to collect them all
  • Only returns the host name, no ability to return record type, TTL, SOA record
  • ONLY WORKS IF A FORWARD RECORD EXISTS THAT MATCHES

This final point took some significant figuring out, and in combination with the rest of the points resulted in me having to change some of the workflows to do an SSH to a linux server to do normal dig and nslookup commands, rather than using the System calls.

Windows: Converting PEM and PKey to PFX file

I’ve just been working on our Chef recipe that installs a CA signed cert for the Windows RDP service. Originally I wrote the recipe to interact with Windows PKI, but it was a bit clunky and I was never really happy with it.

With Hashicorp Vault now being available in our environment I started looking at migrating over to using this, as the integration with Chef was far superior. However, the one stumbling block I came across was that Vault would export the certificate as a PEM file and a Private Key file, whereas Windows could only install and use this pair as a PFX file. A couple of utilities have been available for a while for doing the conversion, either openssl or Pvk2Pfx┬ábut I’ve always shied away from installing new software if at all possible to simplify maintenance.

Fortunately I’ve discovered that certutil now has an option to do this conversion ‘-MergePFX’

Simply put the certificate and key in the same folder with the same name (but different extensions), such as rdpcert.cer and rdpcert.key and run:
certutil -MergePFX rdpcert.cer rdpcert.pfx
and this will combine the files into a PFX file for import.

OT: Migrating Windows from SATA to NVMe

Totally off topic from what I normally blog, but thought it might be useful to some people.

I’ve just migrated my home PC from a SATA SSD to an NVMe drive, and had some fun as there were some things I didn’t know about when I started.
Mostly this was because I was going from an old style MBR partition table, to a GPT/EFI setup

  1. Make sure your PC is capable of booting NVMe
    Mine wasn’t (although I had done some research on this before I started. Running an Asus P8Z77-V LK which doesn’t have an M.2 slot at all, led me to see whether it was possible to add a PCIe adapter. This *is* possible, however the BIOS doesn’t support NVMe boot out of the box, and a modified one is needed. See this forum post for more details
  2. Purchase a PCIe NVMe adapter if you need one. I bought one of these very cheaply
  3. Purchase an NVMe SSD. I bought a Samsung 970 Evo Plus as they have good performance and reputedly better legacy boot support. I’m not sure if that made a difference to the migration.
  4. Create a Windows Recovery/Boot USB and a GParted Boot USB
  5. Install the hardware, making sure the PCIe adapter is in a slot supporting x4 lanes
  6. Make sure your PC is not in the middle of doing some updates. Mine was and this seemed to screw over the registry somehow, requiring a lot of wailing and gnashing of teeth before a partial recovery point restore got things working again
  7. Shut down the PC and boot from the GParted Boot USB
  8. Create a GPT partition table on the new drive
  9. Create a 100MB EFI partition and a 128MB MSFT partition at the start of the drive with FAT32. Mark the EFI partition as ‘Boot’
  10. Clone the Windows partition from the SSD to the new drive
  11. Power down the PC and disconnect the old SATA drive
  12. Switch back on with the Windows Boot USB connected
  13. Use this post to build the EFI boot configuration (I did a lot of this the hard way….)
  14. Remove the USB and go into the BIOS and temporarily turn off Secure Boot, and configure the necessary setting for the PC to boot UEFI from the PCIe storage device
  15. Boot up – it may fail and have to boot into safe mode once to sort out the boot device in windows
  16. Once it is booting into windows successfully, go back into the BIOS and turn Secure Boot back on.

vRealize Orchestrator REST API Inconsistency

I’ve been updating one of our Jenkins jobs (which drives a vRO flow) to call the REST API directly, rather than shelling out to run a script.

As part of this I started looking at retrieving the full workflow log, rather than just the events list, but came up against an error

The API reference shows the method as being:

However when trying this you get a ‘405’ error:

Digging into the server.log on the vRO appliance I found the following message:
WARN {} [PageNotFound] Request method 'GET' not supported
so I tried changing this to a POST on the off-chance.. and voila:

I’ve raised an SR with VMware Support to see if I can get this fixed or the documentation updated

UPDATE – I had a response to say that the syslog call is only supported from 7.6 onwards, so maybe I’m lucky that it works at all in 7.5!

Deploying a VM from a Content Library OVF/OVA template

Following on from my series of posts on uploading OVA files to Content Library, and searching content library, the final step of all of this would be to deploy a VM using one of these templates

Inputs Required

The list of inputs that we need is as follows

NameTypeDescription
datastoreVC:DatastoreThe datastore to deploy to
endpointVAPI:VAPIEndpointThe VAPI endpoint to use
folderVC:VmFolderThe folder to deploy to
hostVC:HostSystemThe target host for deployment
hostnamestringName of the new VM
ovfLibraryItemIdstringid of the content library item
respoolVC:ResourcePoolResource pool to deploy to

Setup

Similar to previous steps, the first thing you need to do is make a connection to the VAPI endpoint
var client = endpoint.client();

Next, we need to create a model for the deployment target – where are we deploying the VM to? We use the deployment target object for this:
var deploymentTarget = new com_vmware_vcenter_ovf_library__item_deployment__target();
deploymentTarget.folder_id = folder.id;
deploymentTarget.host_id = host.id;
deploymentTarget.resource_pool_id = pool.id;

The final step before we deploy the VM is to create a resource pool deployment spec, this include the name, and the datastore to use:
var resourcePoolDeploymentSpec = new com_vmware_vcenter_ovf_library__item_resource__pool__deployment__spec();
resourcePoolDeploymentSpec.accept_all_EULA = true;
resourcePoolDeploymentSpec.name = hostname;
resourcePoolDeploymentSpec.default_datastore_id = datastore.id;

Quite why the datastore is in that one rather than the deployment target is a mystery to me. The only thing I can think of is that the deployment spec includes the storage provisioning profile.

Deploying the VM

To deploy the new VM, we use the com_vmware_vcenter_ovf_library__item call, and pass the ovfLibraryItemId, with the deployment target and deployment specs that we created above
var ovfSvc = new com_vmware_vcenter_ovf_library__item(client);
var result = ovfSvc.deploy(null, ovfLibraryItemId, deploymentTarget, resourcePoolDeploymentSpec);

Returning the vmObject

The deploy method above returns a structure that includes the id of the VM which we can use to find the vmObject, which is an output item of type VC:VirtualMachine
vmObject = VcPlugin.getAllVirtualMachines(null, "xpath:matches(id, '" + result.resource_id.id + "')")[0];
The search returns an array, so I’m using the [0] to strip the object out of the array – searching on an id, there can only be one VM with a particular id unless you’re using linked mode .
Finally we close the connection to the endpoint
client.close();

Full code listing

// Set the VAPI endpoint to the first endpoint returned

if (endpoint == null) {  
  throw "Unable to locate a VAPI endpoint";
}
var client = endpoint.client();  

// create a DeploymentTarget  
var deploymentTarget = new com_vmware_vcenter_ovf_library__item_deployment__target();

deploymentTarget.folder_id = folder.id;
deploymentTarget.host_id = host.id;
deploymentTarget.resource_pool_id = pool.id;

// create a ResourcePoolDeploymentSpec  
var resourcePoolDeploymentSpec = new com_vmware_vcenter_ovf_library__item_resource__pool__deployment__spec();
resourcePoolDeploymentSpec.accept_all_EULA = true;
resourcePoolDeploymentSpec.name = hostname;
resourcePoolDeploymentSpec.default_datastore_id = datastore.id;
  
// deploy the ovf  
var ovfSvc = new com_vmware_vcenter_ovf_library__item(client);
var result = ovfSvc.deploy(null, ovfLibraryItemId, deploymentTarget, resourcePoolDeploymentSpec); 

// return the vmObject
vmObject = VcPlugin.getAllVirtualMachines(null, "xpath:matches(id, '" + result.resource_id.id + "')")[0];
client.close();

VRO – Listing out items from Content Library

Shortly after starting a series of posts on using VRO to upload templates to content library, I was asked how you would retrieve a list of content library items.

It isn’t quite as straightforward as you might think as the List() function only provides a list of IDs rather than full details. However armed with this knowledge, you can get the list of IDs and use that to iterate through and produce a full list of items.

There are four main sources that I’ve been using for the API information
(the links here are direct to the Content Library section)

Overview

The steps we need to go through here are

  • Find and connect to an endpoint
  • Find the content library(s) on the endpoint and connect
  • Get a list of items in the content library
  • For each item in the list, get its details
  • Return the array of items with their full details

Find and connect to an endpoint

The code for this uses the VAPI plugin with the getAllEndpoints() method. An array of all configured endpoints on the VRO instance is returned. We do a quick check to ensure we have something in the array before using it with a for each command

var endpoints = VAPIManager.getAllEndpoints();
var endpoint = endpoints[0]
if (endpoint == null) {
throw "Unable to locate a VAPI endpoint";
}
var ovfLibraryItem = new Array();
for each(var endpoint in endpoints){

System.log("Searching endpoint " + endpoint);
var client = endpoint.client();
...

You can see there that I’ve also initialised the array in which the list of library items will be stored.
The connection to the endpoint is handled by the client() method

Find the content library(s) on the endpoint and connect

Firstly we get a list of the content libraries on the endpoint, with the com_vmware_content_library call. The list() method is used to return an array of their identifiers

var clib = new com_vmware_content_library(client);
System.log("The number of libraries on this endpoint is: " + clib.list().length);
System.log(clib.list());
if(clib.list().length >= 1){

Get a list of items in the content library

To query items in the library, we need to create an object with the com_vmware_content_library_item call

var itemSvc = new com_vmware_content_library_item(client);

Once we’ve created that, we can iterate through the list of content libraries and get a list of items within that library. The list() method is used again here to get an array of items.
for each(var clibrary in clib.list()){
var items = itemSvc.list(clibrary);
System.log(items);

For each item in the list, get its details

This just iterates through the array of items, and uses the get() method on com_vmware_content_library_item to get the detail for each item in turn. The results are pushed into the ovfLibraryItem array that we created near the start.
for each(item in items) {
var results = itemSvc.get(item);
System.log(results.name);
ovfLibraryItem.push(results);
}

Return the array of items with their full details

This is as simple as making the ovfLibraryItem array an output from the scriptable task and the workflow, remembering to close down the endpoint client at the end of the script

Putting it all together

The full code for this comes together like this

// Set the VAPI endpoint to the first endpoint returned
var endpoints = VAPIManager.getAllEndpoints();  
var endpoint = endpoints[0]

if (endpoint == null) {  
  throw "Unable to locate a VAPI endpoint";
}
var ovfLibraryItem = new Array();

for each(var endpoint in endpoints){
  System.log("Searching endpoint " + endpoint);
  var client = endpoint.client();  
  var clib = new com_vmware_content_library(client);  
  System.log("The number of libraries on this endpoint is: " + clib.list().length);
  System.log(clib.list());

  if(clib.list().length >= 1){
    var itemSvc = new com_vmware_content_library_item(client);

    for each(var clibrary in clib.list()){
      var items = itemSvc.list(clibrary); 
      System.log(items);

      for each(item in items) {
        var results = itemSvc.get(item); 
        System.log(results.name);
        ovfLibraryItem.push(results);
      }
    }
  }
  client.close();
}

No inputs are required, and the only output is ovfLibraryItem which is an array of VAPI:com_vmware_content_library_item__model