RHCSA: Understand and use essential tools

  • Access a shell prompt and issue commands with correct syntax
    bash shell, case sensitivity, pwd, ls, cd
    fundamental stuff!
  • Use input-output redirection (>, >>, |, 2>, etc.)
    redirect to and from files: > < >> << 1> 2>
    pipe between commands: |
    prevent > overwriting an existing file: set –o noclobber
    cat – <<%
    text input
  • Use grep and regular expressions to analyze text
    grep “string” file.txt
    egrep “string1|string2” file.txt
  • Access remote systems using ssh
    ssh user@hostname
  • Log in and switch users in multiuser targets
    su, su – username, sudo
  • Archive, compress, unpack, and uncompress files using tar, star, gzip, and bzip2
    tar cvf xvf file.tar file.*
    (add z for gzip, Z for compress, j for bzip2)
    star -xattr -H=exustar -c -f=test.star file.*
    cpio –iv / -ov
  • Create and edit text files
    vim, touch
  • Create, delete, copy, and move files and directories
    rm mv touch cp mkdir rmdir
  • Create hard and soft links
    hard link (inodes point to same blocks): ln file newfile
    soft link (indirect pointer): ln –s file newfile
    directories have to be soft links
  • List, set, and change standard ugo/rwx permissions
    chmod 777 file, chmod a+rwx file
    7 is made from the sum of 4 (read) 2 (write) and 1 (execute)
    So 5 would be read + execute, 4 would be read only
    3 digits for User, Group, Other
  • Locate, read, and use system documentation including man, info, and files in /usr/share/doc
    man ps, whatis ps, apropos ps, info bash
    Install SElinux man pages: yum install -y selinux-policy-devel;mandb

vSphere/vCenter 6.5 Announced

While I’m sadly not at VMWorld this year, I’m following the announcements quite closely and it’s fab to see some blog posts already on the VMware blog for the newly launched version 6.5 product.

It’s great to see the VMware Update Manager now included in the VCSA (this has been anticipated for some time), as well as direct REST APIs, the HTML5 client and a new HA option for vCenter.

I’m sure there will be further announcements and analysis during this week, but for most people in the VMware community, this should fix a significant number of ‘pain points’ within the VMware base product set.

ESXi 6 – weird host HA error

I came across a strange fault with VMware HA today, where a host was reporting an error in its ability to support HA, and  wouldn’t “Reconfigure for HA”

Attempts to perform the reconfigure failed and generated a failed task with the status “Cannot install the vCenter Server agent service. Cannot upload agent”

Screen Shot 2016-08-04 at 15.59.32

Taking the host in and out of Maintenance Mode had no effect, and I could find no pertinent errors in the host logs.

I couldn’t find anything particularly relevant in a google search either, but on digging through the VCenter logs I found the following:

 2016-08-04T15:29:28.567+01:00 info vpxd[16756] [Originator@6876 sub=HostUpgrader opID=909E5426-000012CB-b0-7d] [VpxdHostUpgrader] Fdm on host-6787 has build 3018524. Expected build is 3634793 - will upgrade
2016-08-04T15:29:28.725+01:00 info vpxd[16756] [Originator@6876 sub=HostAccess opID=909E5426-000012CB-b0-7d] Using vpxapi.version.version10 to communicate with vpxa at host guebesx-dell-001.skybet.net
2016-08-04T15:29:28.910+01:00 warning vpxd[16756] [Originator@6876 sub=Libs opID=909E5426-000012CB-b0-7d] SSL: Unknown SSL Error
2016-08-04T15:29:28.911+01:00 info vpxd[16756] [Originator@6876 sub=Libs opID=909E5426-000012CB-b0-7d] SSL Error: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
2016-08-04T15:29:28.911+01:00 warning vpxd[16756] [Originator@6876 sub=Libs opID=909E5426-000012CB-b0-7d] SSL: connect failed
2016-08-04T15:29:28.911+01:00 warning vpxd[16756] [Originator@6876 sub=Default opID=909E5426-000012CB-b0-7d] [NFC ERROR] NfcNewAuthdConnectionEx: Failed to connect to peer. Error: The remote host certificate has these problems:
--> * The host certificate chain is incomplete.
--> * unable to get local issuer certificate
2016-08-04T15:29:28.912+01:00 error vpxd[16756] [Originator@6876 sub=vpxNfcClient opID=909E5426-000012CB-b0-7d] [VpxNfcClient] Unable to connect to NFC server: The remote host certificate has these problems:
--> * The host certificate chain is incomplete.
--> * unable to get local issuer certificate
2016-08-04T15:29:28.913+01:00 error vpxd[16756] [Originator@6876 sub=HostAccess opID=909E5426-000012CB-b0-7d] [VpxdHostAccess] Failed to upload files: vim.fault.SSLVerifyFault
2016-08-04T15:29:28.918+01:00 error vpxd[16756] [Originator@6876 sub=DAS opID=909E5426-000012CB-b0-7d] [VpxdDasConfigLRO] InstallDas failed on host guebesx-dell-001.skybet.net: class Vim::Fault::AgentInstallFailed::Exception(vim.fault.AgentInstallFailed)
2016-08-04T15:29:28.919+01:00 info vpxd[16756] [Originator@6876 sub=MoHost opID=909E5426-000012CB-b0-7d] [HostMo::UpdateDasState] VC state for host host-6787 (uninitialized -> init error), FDM state (UNKNOWN_FDM_HSTATE -> UNKNOWN_FDM_HSTATE), src of state (null -> null)
2016-08-04T15:29:28.950+01:00 info vpxd[16756] [Originator@6876 sub=vpxLro opID=909E5426-000012CB-b0-7d] [VpxLRO] -- FINISH task-internal-15007334
2016-08-04T15:29:28.950+01:00 info vpxd[16756] [Originator@6876 sub=Default opID=909E5426-000012CB-b0-7d] [VpxLRO] -- ERROR task-internal-15007334 -- -- DasConfig.ConfigureHost: vim.fault.AgentInstallFailed:
--> Result:
--> (vim.fault.AgentInstallFailed) {
--> faultCause = (vmodl.MethodFault) null,
--> reason = "AgentUploadFailed",
--> statusCode = <unset>,
--> installerOutput = <unset>,
--> msg = ""
--> }
--> Args:

I’m not sure what had caused the certificate error, but a simple disconnect and reconnect of the host cleared the fault and allowed the HA agent to configure successfully.

UK VMUG 2015 Report – part 1

I missed VMworld this year due to a number of reasons, so was determined to attend this years’ UK VMUG to catch up with the VMware community.

As usual, there was the community vCurry night, prior to the event (note: one year can we please have a change from Lamb curry and Lamb koftas? ) and the associated vQuiz, ably hosted by VMUG leader Stuart Thompson in his unique style.

The following morning I arrived bright and early for the complimentary breakfast, before making my way up to the main hall for the introduction and keynote.

Slide decks are being made available here and I will update with links when the videos are posted.


UK VMUG leader Alaric Davies (@alaricdavies) opened the show with a high energy welcome, including congratulations (and Tshirts) for the new UK VCDX awardees Gregg Robertson (@GreggRobertson5) and Sam McGeown (@sammcgeown), as well as news of the 4 VMworld 2016 ticket and travel costs giveaway at the end of the conference.

Opening keynote – Joe Baguley (@joebaguley)

The day was worth attending for the keynote alone. VMware UK CTO Joe Baguley is an outstanding and inspiring speaker, and if we were from the other side of the Atlantic I’m pretty sure we’d be whooping and high-fiving at the end of his presentations.
I will bullet point his main points as it would be definitely worth your while viewing the full presentation if you missed it.

  • Digital accelerators: VMware viewed as #5 in the list of vendors most able to accelerate a companies move to digital.
  • Bridging Client-Server to Mobile-Cloud: This is the move from rigid structures to digital, or slow and predictable, to rapid and changing
  • Pat Gelsinger quote: “Riskiest thing in business is to take no risks”, the world is changing due to the move to the App Culture. Joe referenced his 15yo daughter who has moved banks 3 times to move to better banking apps.
  • App Economy: Business works on a cycle of producing an app, which produces data, which we analyse to improve app-cycle.pngthe app, and so on.
    Enterprise IT moves around this cycle slowly. If you move around it quicker, you win.
    Things like microservices, big data, and containers are designed to help companies go round the cycle quicker.
  • RAID -> SDDC: In the 1980’s 1 disk wasn’t big or fast enough, so RAID controllers were born to abstract this. Also provided “design for failure”, so were a “Cloud” of storage. SDDC provides this but for datacenters.
  • Unicorns (Google, AWS, FB) have the scale to run custom apps, on custom platforms, on commodity x86/storage/IP. We don’t have this luxury, but SDDC allows standard apps, on SDDC, on commodity x86/storage/IP
    – Infrastructure as code makes everything possible
  • Hardware – becoming a world of white boxes: The end isn’t nigh for hardware vendors, but the future is different – higher volume, smaller margins. Jevons paradox
    The future is racked commodity hardware.
  • Software stack: more layers of virtualisation (turtles all the way down)
  • Docker: virtualisation at the OS level, like Terminal Services, BSD jails. Provides shipping containers for code.
  • Farming: Most people in the room can name all their servers, and know where they are in the DC, we should treat servers like chickens not kittens (cattle, not pets). Developers wanted an API to create/destroy 1000s VMs per hour. Ops couldn’t deliver, so instead they asked for 1 big VM and used containerisation to deliver it.
    Now we’re rethinking infrastructure in terms of microservices, containers, Continuous Integration/Delivery.
  • Fragmented Ecosystems: have hidden costs, tool sprawl, governance issues.
    vSphere Integrated Containers – extension to vSphere to allow transition. Found that performance on VIC is better than bare metal due to hypervisor optimisations.
    Photon platform, new subscription based platform, optimised for containers, large scale API.
    Project Xenon, about dealing with the billions of containers/microservices of the future. See blog
  • Need to evolve skills, teams: the future is about delivering APIs not servers.
  • Simon Wardley reference – Pioneers/Settlers/Town Planners
    Pioneers get bored quickly, Settlers needed to take new tech and make it work, TPs needed to commoditise the working product.
    P/S/TP is a cycle, Pioneers use utility from Town Planners to build new things.

Joe finished with a cryptic mention of the future being Unikernels. It was a thoroughly engaging and inspiring presentation, and I hope you will take the opportunity to watch the video when it’s made available.

Now read part 2

Testing network connectivity from ESXi

Like most people, I’ve always used the “telnet” utility for testing whether a server/tcp port is accessible, when maybe a more complex application is barfing about it.

Of course there have been plenty of attempts to remove it from OSes, the usual reason being “telnet is an insecure protocol, you shouldn’t be using it”
Well… yes it’s an insecure protocol, but no-one in their right mind has been using telnet to log in to systems for many, many years!

Since SSH became prevalent, telnet has only been used for testing whether an IP address / TCP port combination is accessible. You get a number of different responses depending on the connectivity:

  • Immediate connection (and maybe some junk on the screen)
  • Immediate fail – connection is being Rejected – the server (or sometimes a firewall en-route) is forcibly blocking the connection
  • Timeout after a minute or so – connection is either being discarded by a firewall, or there is no route to the server

These 3 different responses are obviously helpful in your troubleshooting. You may be able to infer these different states by monitoring the connection table on the source, for states such as SYN_SENT or ESTABLISHED, but it’s a lot harder that way.

Anyway, enough of history, on ESXi there is no “telnet” utility. Boo! There is however, “nc”. I’m not sure if it’s always been there, but on 5.1U2 it certainly is (which is our minimum level of ESXi in the area I work).

Using the following command:
nc -z -w5 hostname tcpport

will give a suitable replacement for “telnet” eg:

~ # nc -z -w5 esxi-server 5989
Connection to esxi-server 5989 port [tcp/wbem-https] succeeded!

“ping” can also sometimes be used to help, although often the necessary ICMP messages are filtered by firewalls.