Clearing old Host Profile answer files

We recently had a problem where the Fault Tolerance logging service seemed to be randomly getting assigned to the VMotion vmknic, instead of it’s dedicated vmknic. This obviously prevented FT state sync from occuring, a fact that I discovered in a 20 minute change window at 4.30AM ūüė¶

I found the cause of the state sync failure by reading through th vmware.log file for the affected VM, and noticing that the sync seemed to be trying to happen between source and destination IPs on different subnets. Looking at the host IP services configuration within the cluster I found a host which was correct (fortunately the host the FT primary was on was correct too), and used that for the secondary VM which enabled sync to occur.

The problem was affecting roughly 50% of the cluster, and had apparently happened a number of times earlier and been corrected. I noticed that these hosts also had remnants of a host profile answer file – just the Hostname and VMotion interface details, whereas the hosts that were still configured correctly didn’t have any answer file settings stored in VCenter.

Easy I though, bit of PowerCLI will sort that, so had a look for cmdlets for viewing/modifying answer file settings. I hit a blank pretty much straightaway. There are cmdlets for host profiles, one of which allows you to include answerfiles as part of applying a host profile, but nothing for viewing/modifying/removing answer files.

So to the Views we go. A bit of searching turned up this which was helpful, and after a bit of testing I came up with:

$hostProfileManagerView = Get-View "HostProfileManager"
$blank = New-Object VMware.Vim.AnswerFileOptionsCreateSpec

foreach ($vmhost in (Get-Cluster <cluster> | Get-VMhost | sort Name)) {
   $file = $hostProfileManagerView.RetrieveAnswerFile($vmhost.ExtensionData.MoRef)
   if ($file.UserInput.length -gt 0) {
     $file = $hostProfileManagerView.UpdateAnswerFile($vmhost.ExtensionData.MoRef,$blank)
     $file = $hostProfileManagerView.RetrieveAnswerFile($vmhost.ExtensionData.MoRef)
     Write-Output "$($vmhost.Name) $([string]$file.UserInput)"

This iterates through each host in the cluster, and if it has an answerfile, it replaces it with a blank one.


ESXi TLS/SSL/Cipher configuration

Anyone that’s had to configure the TLS/SSL settings for their VMware infrastructure will have probably come across William Lam’s posting on the subject. This provided a much needed script for disabling the weaker protocols on ports 443 (rhttpproxy) and 5989 (sfcb), but leaves out the HA agent on port 8182, and doesn’t alter ciphers – we are having to remove the TLS_RSA ciphers to counter TLS ROBOT ¬†warnings.

The vSphere TLS Reconfigurator utility does fix the TLS protocols for port 8182 (HA communications), but can only be used when the ESXi version is the same minor version as the vCenter, and none of the options will amend the ciphers being used. This was a useful posting I came across for amending the cipher list.
I did attempt to use the (new to ESXi 6.5) Advanced Setting –¬†UserVars.ESXiVPsAllowedCiphers but it appears that this isn’t actually implemented yet. Certainly the rhttpproxy ignores the setting when it starts, and I have raised an SR with VMware to investigate this.

So I thought it might be useful to list the ports that tend to crop up on a vulnerability scan and what is required to fix them, in case there are elements that you may need to configure beyond what the usual utilities and scripts are capable of, such as standalone hosts.

I have only tried these on recent ESXi 6.0U3 and 6.5U1 builds

TCP/443 -VMware HTTP Reverse Proxy and Host Daemon

Set Advanced Settings:
UserVars.ESXiVPsDisabledProtocols to¬†“sslv3,tlsv1,tlsv1.1”
If it’s ESXi 6.0 the following two are also needed:
UserVars.ESXiRhttpproxyDisabledProtocols to “sslv3,tlsv1,tlsv1.1”
UserVars.VMAuthdDisabledProtocols to¬†“sslv3,tlsv1,tlsv1.1”
For the removal of TLS_RSA ciphers:
UserVars.ESXiVPsAllowedCiphers to
The ESXiVPsAllowedCiphers setting does not work, instead manually edit /etc/vmware/rhttpproxy/config.xml and add a cipherList entry:


Restart rhttpproxy service or reboot host

TCP/5989 – VMware Small Footprint CIM Broker

Edit /etc/sfcb/sfcb.cfg and add lines
enableTLSv1: false
enableTLSv1_1: false
enableTLSv1_2: true

Restart sfcb / CIM service or reboot

From what I have seen, the default is to have SSLv3/TLSv1/TLSv1.1 disabled anyway.

TCP/8080 –¬†VMware vSAN VASA Vendor Provider

Should be fixed by the TCP/443 settings

TCP/8182 –¬†VMware Fault Domain Manager

Set Advanced Setting on the *Cluster* :
das.config.vmacore.ssl.protocols to “tls1.2”

Go to each host and initiate “Reconfigure for vSphere HA”

TCP/9080 –¬†VMware vSphere API for IO Filters

Should be fixed by the TCP/443 settings


PowerCLI shortcuts

I’ve just set up some shortcuts for connecting to our various VMware environments, as I was sick of typing out the full


every time.

If you want this to apply for just your userid, you can create (or edit if it already exists)  %UserProfile%\Documents\Windows­PowerShell\profile.ps1

And if you want it to apply for all users, you can create (or edit)

I created the latter, and added lines such as:

function ENV1 {connect-viserver}
function ENV2 {connect-viserver}

Now to connect to a VCenter, all I have to type is ENV1
Do you have any favourite powershell/powerCLI shortcuts like this?

PowerCLI prompting for credentials

One of our VCenters has been prompting for credentials when running connect-viserver since it was first set up, rather than passing through the signed in user’s credentials, and I decided to look into this annoyance.

The particular instance of VCenter has an external PSC, and this web page states that only the PSC needs to be joined to the domain. Indeed, you can’t add the VCSA appliance to the domain through the web interface if it has an external PSC, the option simply isn’t there.

One thing that did stand out from that web page was:

If you want to enable an Active Directory user to log in to a vCenter Server instance by using the vSphere Client with SSPI, you must join the vCenter Server instance to the Active Directory domain. For information about joining a vCenter Server Appliance with an external Platform Services Controller to an Active Directory domain, see the VMware knowledge base article at

I then discovered on this web page :

If you run Connect-VIServer or Connect-CIServer without specifying the User, Password, or Credential parameters, the cmdlet searches the credential store for available credentials for the specified server. If only one credential object is found, the cmdlet uses it to authenticate with the server. If none or more than one PSCredential objects are found, the cmdlet tries to perform a SSPI authentication. If the SSPI authentication fails, the cmdlet prompts you to provide credentials.

Putting those two paragraphs together, 1) AD login with SSPI requires the VCSA to be added to the domain, even with an external PSC, and 2) PowerCLI attempts to use SSPI if it has no credential objects.

The KB article in the first paragraph gives details of how to add the VCSA to the domain from command line, so I did the following:

  • Started PowerCLI
    Ran connect-viserver command to test
    Prompts for credentials
  • Ran the likewise command to add the VCSA to the domain
    Ran connect-viserver command to test
    Prompts for credentials
  • Restarted the VCenter services
    Ran connect-viserver command to test
    Prompts for credentials
    Oh &%$&…..
  • Tested from another Windows server – start up PowerCLI
    Ran connect-viserver command to test
    Loads with no prompt for credentials
  • Returned to original Windows server and restarted PowerCLI
    Ran connect-viserver command to test
    Loads with no prompt for credentials

So it would seem that you at least need to restart PowerCLI, and maybe you need to restart VCenter services (I’m not sure if that was needed now), once you’ve added the VCSA to the domain.

Remediating security issues on VRO 6.6

I’ve recently had to fix a bunch of security vulnerabilities on vRealize Operations 6.6, and thought it may be worth documenting for anyone else trying to fix the same issues.

It was mostly around use of weaker protocols, and self-signed certificates, and I think I’ve managed to isolate the minimum work necessary to fix, happy to be corrected if there are better ways of doing it, or if I’ve missed anything.

  1. Appliance interface on TCP/5480
    • SSH on to the appliance as root
    • replace¬† /opt/vmware/etc/lighttpd/server.pem with a signed certificate (including certification chain if it’s a private CA) and private key.
    • edit /opt/vmware/etc/lighttpd/lighttpd.conf and replace
        ssl.cipher-list = "HIGH:!aNULL:!ADH:!EXP:!MD5:!DH:!3DES:!CAMELLIA:!PSK:!SRP:@STRENGTH"
        ssl.honor-cipher-order = "enable"
        ssl.cipher-list = "EECDH+AESGCM:EDH+AESGCM"
        ssl.use-compression = "disable"
        setenv.add-response-header  += ( "Strict-Transport-Security" => "max-age=63072000; includeSubDomains; preload",
            "X-Frame-Options" => "SAMEORIGIN",
            "X-Content-Type-Options" => "nosniff")
  2. Appliance SFCB interface on TCP/5489
    • SSH onto the appliance as root
    • vi /opt/vmware/share/sfcb/ and update the line:
      umask 0277; openssl req -x509 -days 10000 -newkey rsa:2048 \
      umask 0277; openssl req -x509 -days 730 -newkey rsa:2048 \
    • vi /opt/vmware/etc/ssl/openssl.conf and update
      commonName=<appliance FQDN>
      and add lines
      DNS.2 = <appliance FQDN>
      DNS.3 = <appliance hostname>
      at the end
    • cd /opt/vmware/etc/sfcb/
      and issue
      to update the certificates.
  3. Update the VCO service and configuration console
    • Log in to https://vcoserver:8283/vco-controlcenter/#/control-app/certificates
    • Generate a new SSL certificate with the correct common name and organization details
    • from a root bash shell on the appliance, generate a CSR with:
      -certreq -alias dunes -keypass "password" -keystore
      "/etc/vco/app-server/security/jssecacerts" -file "/tmp/cert.csr"
      -storepass "password"

      (the password is found at /var/lib/vco/keystore.password )
    • Sign the CSR with your Certification Authority
    • Copy the cert to the VCO server as /tmp/cert.cer
    • Re-import the signed certificate with:
      -importcert -alias dunes -keypass "password" -file "/tmp/cert.cer
      -keystore "/etc/vco/app-server/security/jssecacerts" -storepass
    • Verify the keystore with:
      keytool -list -keystore "/etc/vco/app-server/security/jssecacerts" -storepass "password" 
    • Edit the following files to remove TLS1.0
      search for sslEnabledProtocols= and change to read sslEnabledProtocols="TLSv1.1, TLSv1.2"
      also change ciphers= line to remove 3DES ciphers.
  4. Reboot the appliance
  5. Test connections with the following statements:
    openssl s_client -connect <servername>:5480 -tls1
    openssl s_client -connect <servername>:5480 -tls1_2openssl s_client -connect <servername>:5489 -tls1
    openssl s_client -connect <servername>:5489 -tls1_2

    openssl s_client -connect <servername>:8281 -tls1
    openssl s_client -connect <servername>:8281 -tls1_2

    openssl s_client -connect <servername>:8283 -tls1
    openssl s_client -connect <servername>:8283 -tls1_2

    The tls1 connections should now fail, and the tls1.2 connections should still work.

If anyone has examples of getting the SFCB to work with a CA signed certificate I’d be interested, as I’ve tried a number of things without success. It may be down to the properties of the certificate, but the above is sufficient for my requirements at the moment.

PowerCLI – Get-Patch only uses date not time

I’ve been putting together some PowerCLI to set ‘point-in-time’ baselines for VUM patch updates. This is mostly to aid in our interactions with our security colleagues, for example so that when they ask “Is everything patched up to date?” we can say, “The hosts are all compliant with the baseline of dd-mm-yyyy”.

However, when I was using Get-Patch -After ‘<date time>’ to generate delta baselines, I found that it was including the patches from the date/time supplied, rather than anything after.

For example:

PowerCLI C:\> (Get-PatchBaseline “ESXi-standard-baseline-*” | Get-Patch -targettype host -vendor VMware* | Measure-Object -Property “ReleaseDate” -Maximum).Maximum

05 October 2017 01:00:00

But then feeding that into Get-Patch didn’t have the desired effect:

PowerCLI C:\> Get-Patch -TargetType Host -Vendor “VMware*” -After “05 October 2017 01:00:00”

Name Product Release Date Severity Vendor Id
—- ——- ———— ——– ———
Updates esx-base,… {embeddedEsx… 05/10/2017 0… Critical ESXi650-201710401-BG
Updates esx-base,… {embeddedEsx… 05/10/2017 0… Critical ESXi600-201710301-BG

It was pulling out the last patches from the previous baseline, for inclusion in the new one.

Fortunately the resolution for this is pretty straightforward:

PowerCLI C:\> Get-Patch -TargetType host -Vendor “VMware*” | where {$_.ReleaseDate -gt “05 October 2017 01:00:00”}

Which returns no values in this instance, as there’s nothing currently in the patch database after that date.

VMware TAM round table, Manchester 

I ventured over the Penines yesterday to attend the VMware TAM round table meeting, being held in the Manchester Piccadilly Hilton. This provided the opportunity to meet both our companies outgoing TAM and our new one, and learn a little more about what our current contract can provide, that we’re not really utilising.

I also made notes on the presentations, and thought I would share them¬†in case it’s useful to someone else. Apologies in advance to the presenters for my paraphrasing – if the decks are made available, I may update these notes!

I arrived pretty early, due to the train times. Tea/Coffee and pastries were provided, then we converged on the meeting room. After introductions, the first topic up was Simon Todd, covering VSAN 6.5

VSAN 6.5 – Simon Todd

Success Stories

  • SKY – one of the UKs largest VSAN implementations at 6 Petabytes
    • Using it to maintain competitiveness, and to provide grow as you go, expand when needed.
    • Used for production workloads, eg SQL, Exchange, On Demand video, Sky Q, Video Transcoding, UHD streaming
  • Water Utility Company
    • 66-74% cost saving on VM storage cost
    • Procurement cycle went from 3-6 months for traditional SAN to 7 days for extra capacity
    • Billing run for 7M customers dropped from 16 hours to 3 hours
  • A380 – runs VSAN to collect data from 300k sensors for data analysis for preventative maintenance. Every hour saved on the ground saves $25k
  • Oil Rigs, Nuclear subs, Aircraft carriers – anywhere that server maintenance is tricky

Performance Testing

  • Have to use the ¬†right tools, and use them in the right way
  • iometer is a legacy tool, if you’re testing All-Flash storage you need to use >1 outstanding IOs per target – the manufacturer IO figures usually state the queue length and block size.
  • For testing, use VSAN proactive tests, HCI bench, you can use iometer but have to understand how SSDs work and set the configuration accordingly
  • Performance stats now available in the vCenter Web Client (since 6.0U2), going back 90 days.

Configuration Considerations

  • Have to make sure RAID controller, firmware versions and driver versions match what is on the HCL
  • Can use ready nodes to ensure they’re correct from out of the box. If they don’t match your requirements, you are able to increase the spec (more memory, storage) the default spec is just a minimum.
  • Can mix multiple vendors in a single cluster – try and keep the specifications the same (CPU/Mem/Storage) to avoid wastage.
  • Ideally have 2 Disk Groups per host (this means minimum of 2 cache devices)
  • Use multiple capacity devices per Disk Group
  • VMware are working on having VSAN managing firmware and driver revisions, to help you with matching the HCL
  • Network
    • Ensure MTU > 1500
    • Use a different multicast address per cluster
    • 10GigE is a *must* for all flash
    • Use Network IO Control if you have shared interfaces (usually the case if you’re using 10GigE)

VSAN 6.5 – what’s new?

  • iSCSI access
    • Provide block storage to servers not in the VSAN cluster
    • Can use for eg Oracle RAC, Physical workloads
    • Max LUN size is 62TB
    • Still enables Dedupe and Compression, RAID0/1/5
  • 2 node direct connect – connect 2 nodes with crossover cables and have a remote witness – this enables a low cost ROBO entry point for VSAN
  • Supports NVMe, 512e storage, 100Gbps networking
  • New PowerCLI support
    • Health check and remediation
    • Lots of new cmdlets
  • VSAN now ready for
    • VMware Integrated Containers
    • Photon
    • Docker Volume Driver
  • All Flash is now supported in all license versions, higher license versions add things like dedupe/compression

A useful website is

Cloud Foundation – Lee Dilworth

Lee provided an overview of the new VMware Cloud Foundation offering. From a personal viewpoint, it seems like a new ‘unified SDDC platform’ seems to be offered each year, but maybe that’s just my perception….

  • High demand for technology that simplifies infrastructure, but hard to integrate the different technologies.
  • SDDC Manager – provides Automated Lifecycle Management, of Compute, Network and Storage
  • This is an ‘Integrated Platform’, vSphere + NSX + VSAN
  • Provides Cross Cloud Architecture, Private and Public Cloud (AWS)
  • Can be used on a limited range of VSAN ready nodes (3 vendors at present, including Dell), or VxRACK
  • Based on full stack vSphere (vSphere + NSX + VSAN) with SDDC manager on top, plus a range of optional components such as LogInsight, VRO, and VRA via external integration.
  • SDDC Manager
    • Single point management (manages Hardware and Software)
    • One management domain
    • One to multiple workload domains
    • Provides full lifecycle management
    • Integrates into the Web Client
  • Hardware management
    • Uses OOB management agents in Top Of Rack switches
    • Provides Discovery, Bootstrap, Monitoring
    • Uses both In Band and Out of Band connections
  • Management¬†Domain
    • One management domain per cloud instance
    • Uses 3 nodes minimum but 4 recommended
    • Dedicated VCenter plus redundant PSCs
    • Both vDS and NSX vSwitch
    • VSAN
  • Workload Domains
    • Either VDI or standard Virtual Infrastructure
    • Carved out by the SDDC manager
    • Dedicated VC in management domain
    • Shared SSO with management PSCs
    • VSAN
    • NSX – dedicated NSX Manager in management domain, controllers in workload domain.
  • Can automatically deploy and patch vSphere, NSX, VSAN
  • Can deploy but not currently patch LogInsight etc
  • You can upgrade workload domains independently
  • Minimum of 8 nodes (4 mgmt, 4 workload)
  • Maximum of 8 racks
  • VSAN All Flash *or* Hybrid, and can even use network attached storage

Training and Certification Update – Ed Wills (I think!)

There was a short session to give an update on the latest training courses.

vSphere 6.5

  • What’s new 5.5->6.5 – 3 days
  • vSphere ICM 6.5 – 5 days
  • vSphere Optimize and Scale 6.5 – 5 days


  • Cloud Automation Design and Deploy 7.1 – 5 days
  • vCD ICM 8.1 – 5 days
  • Cloud Orchestration and Extensibility – 5 days

Fast Track

  • Horizon 7 ICM & App Volumes – 5 days
  • NSX ICM & Troubleshooting and Operations 6.2 – 5 days
  • vSphere ICM & VSAN 6.5 – 5 days

Enterprise Learning Subscription

This was something I’d not heard of before, but you can register people for 75 training credits per person per year and get access to:

  • All on-demand courses
  • Learning Zone
  • Exam prep materials
  • VCP exam voucher

There is a minimum of 5 people per company.

Training Needs Analysis

This is a new offering, where VMware will perform an analysis of what training your staff require.

It considers business needs, current staff competencies, training methods, cost, effectiveness, and produces a benchmark of the current state, what training is required and why, priorities, where the training will be delivered, who should receive it, how the training will be delivered and how much it will cost.

vRealise Automation – Kim Ranyard

Kim gave an overview of the history of vRA

  • It was originally DynamicOps Cloud Automation Center
  • Then bought by VMware
  • Released as vCAC 5.1 -> 5.2 -> 6.0 -> 6.1
  • Then vRA 6.2 -> 7.0 -> 7.0.1 -> 7.1 -> 7.2

vRA 7.0

  • Designed to accelerate time-to-value
  • Simplified Virtual Appliances HA Landscape
    (instead of needing large numbers of VMs to get it up and working, condensed to 1, or 2 for HA)
  • Enhanced Authentication capabilities
  • Per-tenant branding of the portal
  • Unified Service Design
  • Converged Application Authoring
  • Out-of-the-Box blueprints for more apps, such as MS SQL Server, LAMP stack
  • Able to dynamically configure NSX components
  • Blueprints as code – you can export/import blueprints as YAML
  • Event Broker – provides centralised policy management, helps to integrate with vRO workflows

vRA 7.1

  • Now includes a silent install option
  • Can migrate from 6.2 to 7.1
  • Fixes a number of 6.x upgrade blockers
  • Includes a number of provisioning enhancements, eg provision eager-zero disks, change number of vCPU on a VM
  • Data collection improvements
  • Picks up vSphere Infrastructure changes better, in case someone makes a change outside of vRA
  • Has Out-of-the-Box IPAM integrations
  • Includes more Ready-to-Import blueprints

Application-Centric Infrastructure

  • Can now scale out/in a service (blueprints only), eg add additional app servers to a service to cope with increased load, scale back as load decreases
  • AD integration – can create/delete AD objects OOtB
  • New ‘reconfigure states’ to enable triggering other workflows

vRA 7.2

  • Enhanced update API
  • Migration improvements
  • LDAP support
  • Scale in/out for XaaS components
  • Enhanced LoadBalancing capability
  • IPAM framework extended
  • Re-assign managed VMs
  • Azure endpoint support
  • Container management (container host, and containers)
  • ServiceNow integration

vSphere 6.5 – David Burgess

The most interesting section to me as I’ve not really had chance to look at it yet, was this section on what’s new with vSphere 6.5.

vCenter 6.5


  • The VCSA is now the preferred version of vCenter, and new features will be added to it, not to the Windows version.
  • VCSA exclusive features today:
    • Native HA capability
    • Integrated VMware Update Manager
    • Improved Appliance Management
    • Native Backup/Restore
    • Uses PhotonOS rather than SuSE.
  • VCSA Deployment
    • The installer has support for Windows, Mac and Linux
    • Deploys the OVF, then configures as a second step
    • Options to Install/Upgrade/Migrate/Restore
    • Can migrate from Windows, 5.5 or 6.0 to 6.5
  • VCSA has an HTML5 management interface for the appliance itself
  • VCSA HA – Active/Passive with a witness VM (3 VMs in total)
  • HTML5 Web Client
    • Now fully supported by VMware
    • ~90% feature parity with the flash web client
  • Performance is much better – less resource intensive (applies to Windows vCenter too)

ESX Lifecycle

  • Host profiles are much improved
  • Auto-Deploy – there is now a graphical image builder (rather than just the PowerCLI cmdlets), and it supports IPv6 and UEFI

vSphere API & CLI

  • New REST API for VM management
  • Choice of SDKs and automation tools – multiple languages, plus PowerCLI and DCLI


  • Enhanced Logging
  • VM Encryption – both disk and vmotion traffic
    • Uses an external Key Management Server
    • Can have a non-crypto admin user that can do most admin but not access console, read/write data etc
  • Encrypted vMotion – can be set to Disabled/Opportunistic/Required
  • UEFI Secure boot (for the hypervisor) – needs signed drivers
  • VM Secure boot (UEFI secure boot for the VM)

Application Availability and Resource Management

  • Proactive HA – detect hardware degraded conditions, vMotion guests off host. Hardware OEM participation is required, eg Dell OpenManage, HP Insight Manager
  • HA Orchestrated Restart – VM-to-VM dependency checks (this has validation checks to prevent dependency loops for example)
  • 5 Restart Priorities (up from 3 in previous versions)
  • HA Admission Control – this has been updated to simplify
    • Chooseter Failures To Tolerate
    • Based on % of resources reserved
    • Automatic calculations, rather than manual reconfiguration whenever you add/remove a host
    • Overrides are possible
  • New DRS options
    • Even distribution (helps to balance out the cluster even if it’s not required for performance reasons
    • Can base on consumed memory rather than active memory
    • Takes into account CPU overcommitment

Other changes

  • New CPU models and architectures are now supported
  • LUN limit has been increased to 512
  • Supports vRDMA (virtualised Remote Direct Memory Access) via a paravirtual driver.


The day then concluded with a demo of VRA with Codestream.

I felt it was a worthwhile event, and it was great to meet a few new people. Thanks again to the VMware UK TAM team for running it.