We’re currently running through the 7.0U2 upgrades, and have encountered this on one of them. It failed at 92% during the data conversion step, and as part of the diagnostics, support had us run a postgres consistency check script: https://kb.vmware.com/s/article/53062
The script does many checks, but it also creates a postgres extension called amcheck_next.
This caused subsequent upgrade attempts to fail – the message appears in the log /var/log/vmware/applmgmt/PatchRunner.log
2021-05-27T06:56:41.087Z Running: su -s /bin/bash vpostgres -c cd /var/log/vmware/vpostgres/upgrade && /opt/vmware/vpostgres/12/bin/pg_upgrade -U postgres -B /opt/vmware/vpostgres/12/bin -b /opt/vmware/vpostgres/12/../11/bin -D /storage/db/vpostgres.12 -d /storage/db/vpostgres --check 2021-05-27T06:56:41.087Z Running command: ['su', '-s', '/bin/bash', 'vpostgres', '-c', 'cd /var/log/vmware/vpostgres/upgrade && /opt/vmware/vpostgres/12/bin/pg_upgrade -U postgres -B /opt/vmware/vpostgres/12/bin -b /opt/vmware/vpostgres/12/../11/bin -D /storage/db/vpostgres.12 -d /storage/db/vpostgres --check'] 2021-05-27T06:56:42.140Z Done running command , stderr: 2021-05-27T06:56:42.141Z pg_upgrade --check returned error code 1 with error out Performing Consistency Checks ----------------------------- Checking cluster versions ok Checking database user is the install user ok Checking database connection settings ok Checking for prepared transactions ok Checking for reg* data types in user tables ok Checking for contrib/isn with bigint-passing mismatch ok Checking for tables WITH OIDS ok Checking for invalid "sql_identifier" user columns ok Checking for presence of required libraries fatal Your installation references loadable libraries that are missing from the new installation. You can add these libraries to the new installation, or remove the functions using them from the old installation. A list of problem libraries is in the file: loadable_libraries.txt Failure, exiting . 2021-05-27 06:56:42,169.169Z vpostgres:Patch ERROR vmware_b2b.patching.executor.hook_executor Patch hook 'vpostgres:Patch' failed.
Checking the “loadable_libraries.txt” in ‘/storage/log/vmware/vpostgres/upgrade’ showed the following:
could not load library "$libdir/amcheck_next": ERROR: could not access file "$libdir/amcheck_next": No such file or directory Database: VCDB
To rectify this, after rolling back to the pre-upgrade snapshot, I just connected to the DB with ‘psql -U postgres -h localhost -d VCDB’ and ran ‘DROP EXTENSION amcheck_next;’
I then cleared out the VUM database (which had cause the original failure) and the update then ran through successfully.