PowerCLI – Get-Patch only uses date not time

I’ve been putting together some PowerCLI to set ‘point-in-time’ baselines for VUM patch updates. This is mostly to aid in our interactions with our security colleagues, for example so that when they ask “Is everything patched up to date?” we can say, “The hosts are all compliant with the baseline of dd-mm-yyyy”.

However, when I was using Get-Patch -After ‘<date time>’ to generate delta baselines, I found that it was including the patches from the date/time supplied, rather than anything after.

For example:

PowerCLI C:\> (Get-PatchBaseline “ESXi-standard-baseline-*” | Get-Patch -targettype host -vendor VMware* | Measure-Object -Property “ReleaseDate” -Maximum).Maximum

05 October 2017 01:00:00

But then feeding that into Get-Patch didn’t have the desired effect:

PowerCLI C:\> Get-Patch -TargetType Host -Vendor “VMware*” -After “05 October 2017 01:00:00”

Name Product Release Date Severity Vendor Id
—- ——- ———— ——– ———
Updates esx-base,… {embeddedEsx… 05/10/2017 0… Critical ESXi650-201710401-BG
Updates esx-base,… {embeddedEsx… 05/10/2017 0… Critical ESXi600-201710301-BG

It was pulling out the last patches from the previous baseline, for inclusion in the new one.

Fortunately the resolution for this is pretty straightforward:

PowerCLI C:\> Get-Patch -TargetType host -Vendor “VMware*” | where {$_.ReleaseDate -gt “05 October 2017 01:00:00”}

Which returns no values in this instance, as there’s nothing currently in the patch database after that date.

Advertisements

Problem adding Active Directory auth to Netbackup Appliance

I’ve spent a fair amount of time recently on two things that I’ve not done much of before: Certificates, and integrating to AD authentication. Of the two, the AD auth has generally been straightforward, and installing signed certs has been….. well…  let’s just call it a learning curve.

Of course, I’ve documented the processes internally, and where possible I’ve automated the configuration for future use.

One of the things that came up recently was adding AD auth for some Netbackup appliances, where shared admin credentials had been in use and we were wanting to avoid individual local accounts. As this is a well documented process I was expecting it to be pretty straightforward, and after checking some details with Veritas Support (such as whether the credentials used for configuring the authentication were stored and used for each lookup – they’re not), I wrote up the change plan and began implementation.

On the first appliance, all was well, and it was as simple as the documentation implied. It was when I came to the next one that I hit problems.

I submitted the details and credentials for the local Active Directory, and after a short delay received the following errors:

- [Error] Unable to configure the appliance for Active Directory Authentication.
Check the credentials, authorization of user, and network connectivity issues
- [Error] Unable to join the domain. Please check the credentials used to join the domain, network connectivity, etc. Otherwise contact support
Unable to configure the appliance for Active Directory Authentication. Check
the credentials, authorization of user, and network connectivity issues
Unable to join the domain. Please check the credentials used to join the
domain, network connectivity, etc. Otherwise contact support
Command failed!

Obviously I then tried exactly the same again, unsurprisingly with the same result.

Next step was a tcpdump of traffic between the appliance and the domain controller, after a little trial and error around the filter to find the crux of the matter, I captured the following on port TCP/445:

10:08:56.983135 IP (tos 0x0, ttl 64, id 34248, offset 0, flags [DF], proto TCP (6), length 60)
10:08:56.983135 IP (tos 0x0, ttl 64, id 34248, offset 0, flags [DF], proto TCP (6), length 60)
appliance-fqdn.37346 > domaincontroller-fqdn.microsoft-ds: Flags [S], cksum 0x4556 (correct), seq 2083948561, win 14600, options [mss 1460,sackOK,TS val 91164121 ecr 0,nop,wscale 10], length 0
10:08:56.983365 IP (tos 0x0, ttl 127, id 27767, offset 0, flags [DF], proto TCP (6), length 60)
domaincontroller-fqdn.microsoft-ds > appliance-fqdn.37346: Flags [S.], cksum 0xe3c7 (correct), seq 1747220571, ack 2083948562, win 8192, options [mss 1460,nop,wscale 8,sackOK,TS val 499809341 ecr 91164121], length 0
10:08:56.983410 IP (tos 0x0, ttl 64, id 34249, offset 0, flags [DF], proto TCP (6), length 52)
appliance-fqdn.37346 > domaincontroller-fqdn.microsoft-ds: Flags [.], cksum 0x3286 (correct), seq 1, ack 1, win 15, options [nop,nop,TS val 91164121 ecr 499809341], length 0
10:08:57.001109 IP (tos 0x0, ttl 64, id 34250, offset 0, flags [DF], proto TCP (6), length 246)
appliance-fqdn.37346 > domaincontroller-fqdn.microsoft-ds: Flags [P.], cksum 0x1f58 (incorrect -> 0x02a5), seq 1:195, ack 1, win 15, options [nop,nop,TS val 91164139 ecr 499809341], length 194
SMB PACKET: SMBnegprot (REQUEST)
SMB Command = 0x72
Error class = 0x0
Error code = 0 (0x0)
Flags1 = 0x8
Flags2 = 0x1
Tree ID = 0 (0x0)
Proc ID = 12819 (0x3213)
UID = 0 (0x0)
MID = 1 (0x1)
Word Count = 0 (0x0)
smb_bcc=155
Dialect=PC NETWORK PROGRAM 1.0
Dialect=MICROSOFT NETWORKS 1.03
Dialect=MICROSOFT NETWORKS 3.0
Dialect=LANMAN1.0
Dialect=LM1.2X002
Dialect=DOS LANMAN2.1
Dialect=LANMAN2.1
Dialect=Samba
Dialect=NT LANMAN 1.0
Dialect=NT LM 0.12

10:08:57.001402 IP (tos 0x0, ttl 127, id 27768, offset 0, flags [DF], proto TCP (6), length 40)
domaincontroller-fqdn.microsoft-ds > appliance-fqdn.37346: Flags [R.], cksum 0x1836 (correct), seq 1, ack 195, win 0, length 0

The last two packets in the conversation are an SMB Negotiate request from the appliance, which the domain controller responds to with a TCP Reset packet. Rude!

At this point, the sensible thing was to compare with the working one, however as that was already successfully using AD auth, I wasn’t sure whether it would be a good comparison, however I found that using smbclient to try and establish a session on the problem appliance also produced the same result.

10:27:06.868465 IP (tos 0x0, ttl 64, id 38798, offset 0, flags [DF], proto TCP (6), length 60)
 appliance-fqdn.37879 > domaincontroller-fqdn.microsoft-ds: Flags [S], cksum 0x71e9 (correct), seq 3921922782, win 14600, options [mss 1460,sackOK,TS val 3537132880 ecr 0,nop,wscale 10], length 0
10:27:06.868694 IP (tos 0x0, ttl 127, id 29676, offset 0, flags [DF], proto TCP (6), length 60)
 domaincontroller-fqdn.microsoft-ds > appliance-fqdn.37879: Flags [S.], cksum 0xd55f (correct), seq 3976192417, ack 3921922783, win 8192, options [mss 1460,nop,wscale 8,sackOK,TS val 769833214 ecr 3537132880], length 0
10:27:06.868730 IP (tos 0x0, ttl 64, id 38799, offset 0, flags [DF], proto TCP (6), length 52)
 appliance-fqdn.37879 > domaincontroller-fqdn.microsoft-ds: Flags [.], cksum 0x241e (correct), seq 1, ack 1, win 15, options [nop,nop,TS val 3537132880 ecr 769833214], length 0
10:27:06.960943 IP (tos 0x0, ttl 64, id 38800, offset 0, flags [DF], proto TCP (6), length 246)
 appliance-fqdn.37879 > domaincontroller-fqdn.microsoft-ds: Flags [P.], cksum 0x2378 (incorrect -> 0x856b), seq 1:195, ack 1, win 15, options [nop,nop,TS val 3537132973 ecr 769833214], length 194
SMB PACKET: SMBnegprot (REQUEST)
SMB Command = 0x72
Error class = 0x0
Error code = 0 (0x0)
Flags1 = 0x8
Flags2 = 0x1
Tree ID = 0 (0x0)
Proc ID = 47233 (0xb881)
UID = 0 (0x0)
MID = 1 (0x1)
Word Count = 0 (0x0)
smb_bcc=155
Dialect=PC NETWORK PROGRAM 1.0
Dialect=MICROSOFT NETWORKS 1.03
Dialect=MICROSOFT NETWORKS 3.0
Dialect=LANMAN1.0
Dialect=LM1.2X002
Dialect=DOS LANMAN2.1
Dialect=LANMAN2.1
Dialect=Samba
Dialect=NT LANMAN 1.0
Dialect=NT LM 0.12

10:27:06.961426 IP (tos 0x0, ttl 127, id 29678, offset 0, flags [DF], proto TCP (6), length 261)
 domaincontroller-fqdn.microsoft-ds > appliance-fqdn.37879: Flags [P.], cksum 0x82f9 (correct), seq 1:210, ack 195, win 514, options [nop,nop,TS val 769833223 ecr 3537132973], length 209
SMB PACKET: SMBnegprot (REPLY)
SMB Command = 0x72
Error class = 0x0
Error code = 0 (0x0)
Flags1 = 0x88
Flags2 = 0x1
Tree ID = 0 (0x0)
Proc ID = 47233 (0xb881)
UID = 0 (0x0)
MID = 1 (0x1)
Word Count = 17 (0x11)
NT1 Protocol
DialectIndex=9 (0x9)

This time, the SMB negotiate elicited a reply rather than a reset, which selected dialect 9, “NT LANMAN 1.0”, which is the root of the problem. The local DCs for the appliance where AD auth was working had had NTLMv1/SMB1 temporarily enabled, whereas the ones for the appliance where it wasn’t working, didn’t have NTLMv1/SMB1 enabled.

I then started to look into enabling SMB2 on Samba. The appliances are built on RedHat 6.6, and use Samba 3.6.23. While this version of Samba was the first to support SMB2, it *only* supported it as a server, not as a client. The support for SMB2 as a client only came in at version 4.1.0. I also checked the very latest build of the appliance software, which turned out to use exactly the same version as the one we were on. At this point I felt I’d taken it as far as I could, and raised the issue with Veritas support.

Anyone who’s dealt with frontline support from an organisation like Veritas will understand the frustration I went through for the next two weeks, as the frontline engineer assigned tried to follow his call script, completely ignoring all the diagnostics I’d already done. Ultimately he did escalate to a backline engineer (I’d not stamped and shouted as it wasn’t an impacting issue) and it quickly got pushed to the Netbackup engineering team in the states.

In the mean time I’d spoken to the beta team about what version would be in the next release, which confirmed that it would be a version that fixed the problem.

The reply back from engineering was that I was the only person to have come across this issue (meaning that either people just don’t enable AD auth, or that those who do, have SMB1 enabled on their domains), so as it would require significant testing to uplift Samba to a much higher version, they wouldn’t be releasing a patch to fix it. However they confirmed what the beta team had said about it being fixed in the forthcoming version of Netbackup.

Anyway, I’ll be awaiting that release with interest, and I hope my documenting the issue here helps someone else.

Auto-install of .Net 3.5 on Windows 2012 R2

The “standard” way of installing .Net 3.5 into a Windows 2012 R2 server, is to mount the install DVD (or ISO image), and use Add Features to install it. Obviously this is a massive pain if you’ve got a lot to do, as you either need to copy 4.5Gb of image around, or use some out-of-band method of mounting the image, neither of which are ideal. The only sane option would be to extract the ISO to a CIFS share, and make that available to all servers, but this wasn’t an option here.

For automation, we would normally use the PowerShell command:

Install-WindowsFeature Net-Framework-Core -source \\image-path\sources\sxs

… so already, it looks like we don’t need all the image to do the install, just the “sources\sxs” directory.

A quick check shows that the “sources\sxs” directory is 289Mb, so much more manageable, but surely we can do better than this, as it includes a lot of other features.

Running a filter with procmon during the feature install allows you to capture all the file accesses to the sources\sxs directory, which can be exported as a CSV file:
"Time of Day","Process Name","PID","Operation","Path","Result","Detail"
"1:59:25.0591131 PM","TiWorker.exe","2896","ReadFile","D:","SUCCESS","Offset: 0, Length: 4,096, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal"
"1:59:25.1140710 PM","TiWorker.exe","2896","ReadFile","D:","SUCCESS","Offset: 4,096, Length: 4,096, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal"
"1:59:25.1183118 PM","TiWorker.exe","2896","ReadFile","D:","SUCCESS","Offset: 8,192, Length: 4,096, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal"
"1:59:25.1197002 PM","TiWorker.exe","2896","CreateFile","D:\sources\sxs","SUCCESS","Desired Access: Read Attributes, Disposition: Open, Options: Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a, Impersonating: NT AUTHORITY\SYSTEM, OpenResult: Opened"
"1:59:25.1215631 PM","TiWorker.exe","2896","QueryBasicInformationFile","D:\sources\sxs","SUCCESS","CreationTime: 3/21/2014 2:27:47 PM, LastAccessTime: 3/21/2014 2:27:47 PM, LastWriteTime: 3/21/2014 2:27:47 PM, ChangeTime: 3/21/2014 2:27:47 PM, FileAttributes: RD"
"1:59:25.1215789 PM","TiWorker.exe","2896","CloseFile","D:\sources\sxs","SUCCESS",""
"1:59:30.8039209 PM","TiWorker.exe","2896","ReadFile","D:","SUCCESS","Offset: 0, Length: 4,096, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal"
...etc....

This can then be condensed with (I’m using a *nix command line to do this as it’s more familiar to me):

cat Logfile.CSV | cut -d"," -f5 | cut -d"\\" -f4 | sort -u | grep -v -e "^$" -e "^\""

To give a list of subfolders in the “sources\sxs” directory that are required. These can be used to copy the relevant source files to a zip archive:
cat Logfile.CSV | cut -d"," -f5 | cut -d"\\" -f4 | sort -u | grep -v -e "^$" -e "^\"" | while read folder
do
zip -r net35.zip /Volumes/Win2012R2ISO/sources/sxs/$folder
done

…which generates an (approx) 88Mb zip file, much more suitable for installing via automation.

It’s then a fairly straightforward task to use your automation framework (Chef, Puppet etc) to copy the zip file down to the server, extract and run the powershell command to install.

RHCSA: Manage users and groups

  • Create, delete, and modify local user accounts
    useradd
    usermod
    userdel
  • Change passwords and adjust password aging for local user accounts
    passwd
    chage
    /etc/login.defs
  • Create, delete, and modify local groups and group memberships
    groupadd
    groupmod
    groupdel
  • Configure a system to use an existing authentication service for user and group information
    yum install sssd auth*
    system-config-authentication / authconfig-tui
    LDAP
    Base DN dc=example,dc=com
    LDAP server ldap://classroom.example.com
    Use TLS
    Download cert
    Method – LDAP password

RHCSA: Deploy, configure, and maintain systems

  • Configure networking and hostname resolution statically or dynamically
    nmcli, nmtui
    /etc/resolv.conf
    /etc/hosts
    hostnamectl
  • Schedule tasks using at and cron
    at 15:00 <<%
    echo test
    %
    crontab –e –u gertrude
    15,45 09-17 * * 6 echo test
  • Start and stop services and configure services to start automatically at boot
    systemctl start/stop sshd.service
    systemctl enable sshd.service
  • Configure systems to boot into a specific target automatically
    systemctl set-default graphical.target
    systemctl set-default multi-user.target
  • Install Red Hat Enterprise Linux automatically using Kickstart
    system-config-kickstart
  • Configure a physical machine to host virtual guests
    yum install qemu-kvm qemu-img libvirt virt-manager
  • Install Red Hat Enterprise Linux systems as virtual guests
    virt-install / virtmanager
  • Configure systems to launch virtual machines at boot
    virsh autostart ‘name’
  • Configure network services to start automatically at boot
    systemctl enable network
  • Configure a system to use time services
    yum install chrony
    /etc/ntp.conf
  • Install and update software packages from Red Hat Network, a remote repository, or from the local file system
    yum install package
    yum upgrade package
    rpm –Uvh / –ivh package
    yum-config-manager –add-repo http://content.example.com/rhel7.0/x86_64/dvd
  • Update the kernel package appropriately to ensure a bootable system
    yum upgrade kernel
    rpm -ivh kernel-<kernel version>.<arch>.rpm
  • Modify the system bootloader
    grubby –info=/boot/<kernel>
    grubby –remove-args=”arg1 arg2” –args=”argx argy” –update-kernel=/boot/<kernel>
    grubby –update-kernel=ALL

RHCSA: Create and configure file systems

  • Create, mount, unmount, and use vfat, ext4, and xfs file systems
    mkfs.xfs, mkfs.ext4, mkfs.vfat
    mount, umount
    /etc/fstab
  • Mount and unmount CIFS and NFS network file systems
    mount –t cifs server:/vol/share /mnt/share -o user=userid,pass=pword,dom=AD
    mount –t nfs server:/vol/share /mnt/share
  • Extend existing logical volumes
    vgextend
    lvextend
    xfs_growfs
    resize2fs
    ssm resize
  • Create and configure set-GID directories for collaboration
    setgid dirs make created files/dirs have the setgid group id
    chgrp mygroup ./directory
    chmod 2755 ./directory
  • Create and manage Access Control Lists (ACLs)
    getfacl
    setfacl
  • Diagnose and correct file permission problems
    chmod, chown, getfacl, setfacl

RHCSA: Configure local storage

  • List, create, delete partitions on MBR and GPT disks
    fdisk gdisk parted
    blkid lsblk
  • Create and remove physical volumes, assign physical volumes to volume groups, and create and delete logical volumes
    using a new disk/partition with LVM: pvcreate /dev/device
    creating a new volume group: vgcreate VG00 /dev/device
    adding a PV to an existing volume group: vgextend VG00 /dev/device
    creating a logical volume: lvcreate -L 100G -n lvhome VG00
  • Configure systems to mount file systems at boot by Universally Unique ID (UUID) or label
    blkid to get the UUID/label, then add to /etc/fstab
    can set label on ext filesystems with tune2fs or e2label
  • Add new partitions and logical volumes, and swap to a system non-destructively
    as per above commands!
    need to set the fstype correctly with fdisk/gdisk/parted
    mkswap, swapon