Problem adding Active Directory auth to Netbackup Appliance

I’ve spent a fair amount of time recently on two things that I’ve not done much of before: Certificates, and integrating to AD authentication. Of the two, the AD auth has generally been straightforward, and installing signed certs has been….. well…  let’s just call it a learning curve.

Of course, I’ve documented the processes internally, and where possible I’ve automated the configuration for future use.

One of the things that came up recently was adding AD auth for some Netbackup appliances, where shared admin credentials had been in use and we were wanting to avoid individual local accounts. As this is a well documented process I was expecting it to be pretty straightforward, and after checking some details with Veritas Support (such as whether the credentials used for configuring the authentication were stored and used for each lookup – they’re not), I wrote up the change plan and began implementation.

On the first appliance, all was well, and it was as simple as the documentation implied. It was when I came to the next one that I hit problems.

I submitted the details and credentials for the local Active Directory, and after a short delay received the following errors:

- [Error] Unable to configure the appliance for Active Directory Authentication.
Check the credentials, authorization of user, and network connectivity issues
- [Error] Unable to join the domain. Please check the credentials used to join the domain, network connectivity, etc. Otherwise contact support
Unable to configure the appliance for Active Directory Authentication. Check
the credentials, authorization of user, and network connectivity issues
Unable to join the domain. Please check the credentials used to join the
domain, network connectivity, etc. Otherwise contact support
Command failed!

Obviously I then tried exactly the same again, unsurprisingly with the same result.

Next step was a tcpdump of traffic between the appliance and the domain controller, after a little trial and error around the filter to find the crux of the matter, I captured the following on port TCP/445:

10:08:56.983135 IP (tos 0x0, ttl 64, id 34248, offset 0, flags [DF], proto TCP (6), length 60)
10:08:56.983135 IP (tos 0x0, ttl 64, id 34248, offset 0, flags [DF], proto TCP (6), length 60)
appliance-fqdn.37346 > domaincontroller-fqdn.microsoft-ds: Flags [S], cksum 0x4556 (correct), seq 2083948561, win 14600, options [mss 1460,sackOK,TS val 91164121 ecr 0,nop,wscale 10], length 0
10:08:56.983365 IP (tos 0x0, ttl 127, id 27767, offset 0, flags [DF], proto TCP (6), length 60)
domaincontroller-fqdn.microsoft-ds > appliance-fqdn.37346: Flags [S.], cksum 0xe3c7 (correct), seq 1747220571, ack 2083948562, win 8192, options [mss 1460,nop,wscale 8,sackOK,TS val 499809341 ecr 91164121], length 0
10:08:56.983410 IP (tos 0x0, ttl 64, id 34249, offset 0, flags [DF], proto TCP (6), length 52)
appliance-fqdn.37346 > domaincontroller-fqdn.microsoft-ds: Flags [.], cksum 0x3286 (correct), seq 1, ack 1, win 15, options [nop,nop,TS val 91164121 ecr 499809341], length 0
10:08:57.001109 IP (tos 0x0, ttl 64, id 34250, offset 0, flags [DF], proto TCP (6), length 246)
appliance-fqdn.37346 > domaincontroller-fqdn.microsoft-ds: Flags [P.], cksum 0x1f58 (incorrect -> 0x02a5), seq 1:195, ack 1, win 15, options [nop,nop,TS val 91164139 ecr 499809341], length 194
SMB PACKET: SMBnegprot (REQUEST)
SMB Command = 0x72
Error class = 0x0
Error code = 0 (0x0)
Flags1 = 0x8
Flags2 = 0x1
Tree ID = 0 (0x0)
Proc ID = 12819 (0x3213)
UID = 0 (0x0)
MID = 1 (0x1)
Word Count = 0 (0x0)
smb_bcc=155
Dialect=PC NETWORK PROGRAM 1.0
Dialect=MICROSOFT NETWORKS 1.03
Dialect=MICROSOFT NETWORKS 3.0
Dialect=LANMAN1.0
Dialect=LM1.2X002
Dialect=DOS LANMAN2.1
Dialect=LANMAN2.1
Dialect=Samba
Dialect=NT LANMAN 1.0
Dialect=NT LM 0.12

10:08:57.001402 IP (tos 0x0, ttl 127, id 27768, offset 0, flags [DF], proto TCP (6), length 40)
domaincontroller-fqdn.microsoft-ds > appliance-fqdn.37346: Flags [R.], cksum 0x1836 (correct), seq 1, ack 195, win 0, length 0

The last two packets in the conversation are an SMB Negotiate request from the appliance, which the domain controller responds to with a TCP Reset packet. Rude!

At this point, the sensible thing was to compare with the working one, however as that was already successfully using AD auth, I wasn’t sure whether it would be a good comparison, however I found that using smbclient to try and establish a session on the problem appliance also produced the same result.

10:27:06.868465 IP (tos 0x0, ttl 64, id 38798, offset 0, flags [DF], proto TCP (6), length 60)
 appliance-fqdn.37879 > domaincontroller-fqdn.microsoft-ds: Flags [S], cksum 0x71e9 (correct), seq 3921922782, win 14600, options [mss 1460,sackOK,TS val 3537132880 ecr 0,nop,wscale 10], length 0
10:27:06.868694 IP (tos 0x0, ttl 127, id 29676, offset 0, flags [DF], proto TCP (6), length 60)
 domaincontroller-fqdn.microsoft-ds > appliance-fqdn.37879: Flags [S.], cksum 0xd55f (correct), seq 3976192417, ack 3921922783, win 8192, options [mss 1460,nop,wscale 8,sackOK,TS val 769833214 ecr 3537132880], length 0
10:27:06.868730 IP (tos 0x0, ttl 64, id 38799, offset 0, flags [DF], proto TCP (6), length 52)
 appliance-fqdn.37879 > domaincontroller-fqdn.microsoft-ds: Flags [.], cksum 0x241e (correct), seq 1, ack 1, win 15, options [nop,nop,TS val 3537132880 ecr 769833214], length 0
10:27:06.960943 IP (tos 0x0, ttl 64, id 38800, offset 0, flags [DF], proto TCP (6), length 246)
 appliance-fqdn.37879 > domaincontroller-fqdn.microsoft-ds: Flags [P.], cksum 0x2378 (incorrect -> 0x856b), seq 1:195, ack 1, win 15, options [nop,nop,TS val 3537132973 ecr 769833214], length 194
SMB PACKET: SMBnegprot (REQUEST)
SMB Command = 0x72
Error class = 0x0
Error code = 0 (0x0)
Flags1 = 0x8
Flags2 = 0x1
Tree ID = 0 (0x0)
Proc ID = 47233 (0xb881)
UID = 0 (0x0)
MID = 1 (0x1)
Word Count = 0 (0x0)
smb_bcc=155
Dialect=PC NETWORK PROGRAM 1.0
Dialect=MICROSOFT NETWORKS 1.03
Dialect=MICROSOFT NETWORKS 3.0
Dialect=LANMAN1.0
Dialect=LM1.2X002
Dialect=DOS LANMAN2.1
Dialect=LANMAN2.1
Dialect=Samba
Dialect=NT LANMAN 1.0
Dialect=NT LM 0.12

10:27:06.961426 IP (tos 0x0, ttl 127, id 29678, offset 0, flags [DF], proto TCP (6), length 261)
 domaincontroller-fqdn.microsoft-ds > appliance-fqdn.37879: Flags [P.], cksum 0x82f9 (correct), seq 1:210, ack 195, win 514, options [nop,nop,TS val 769833223 ecr 3537132973], length 209
SMB PACKET: SMBnegprot (REPLY)
SMB Command = 0x72
Error class = 0x0
Error code = 0 (0x0)
Flags1 = 0x88
Flags2 = 0x1
Tree ID = 0 (0x0)
Proc ID = 47233 (0xb881)
UID = 0 (0x0)
MID = 1 (0x1)
Word Count = 17 (0x11)
NT1 Protocol
DialectIndex=9 (0x9)

This time, the SMB negotiate elicited a reply rather than a reset, which selected dialect 9, “NT LANMAN 1.0”, which is the root of the problem. The local DCs for the appliance where AD auth was working had had NTLMv1/SMB1 temporarily enabled, whereas the ones for the appliance where it wasn’t working, didn’t have NTLMv1/SMB1 enabled.

I then started to look into enabling SMB2 on Samba. The appliances are built on RedHat 6.6, and use Samba 3.6.23. While this version of Samba was the first to support SMB2, it *only* supported it as a server, not as a client. The support for SMB2 as a client only came in at version 4.1.0. I also checked the very latest build of the appliance software, which turned out to use exactly the same version as the one we were on. At this point I felt I’d taken it as far as I could, and raised the issue with Veritas support.

Anyone who’s dealt with frontline support from an organisation like Veritas will understand the frustration I went through for the next two weeks, as the frontline engineer assigned tried to follow his call script, completely ignoring all the diagnostics I’d already done. Ultimately he did escalate to a backline engineer (I’d not stamped and shouted as it wasn’t an impacting issue) and it quickly got pushed to the Netbackup engineering team in the states.

In the mean time I’d spoken to the beta team about what version would be in the next release, which confirmed that it would be a version that fixed the problem.

The reply back from engineering was that I was the only person to have come across this issue (meaning that either people just don’t enable AD auth, or that those who do, have SMB1 enabled on their domains), so as it would require significant testing to uplift Samba to a much higher version, they wouldn’t be releasing a patch to fix it. However they confirmed what the beta team had said about it being fixed in the forthcoming version of Netbackup.

Anyway, I’ll be awaiting that release with interest, and I hope my documenting the issue here helps someone else.

Advertisements

Adventures in backups

I’ve been quiet for the last few months, mainly because I’ve been working on a Backup project, with not so much focus on Virtualisation.

Prior to this, I’d mostly left it to the professionals, as it had generally fallen into the remit of the storage teams, but when I finished off my previous projects, and the music stopped, the only chair remaining was on a ‘behind schedule’ backup capacity project.

I’m not going to go into the why’s and wherefore’s of why the project was stuck in limbo, but I decided to share my thoughts on what I’d learned from it.

  • Requirements, Requirements, Requirements
    If a project doesn’t have them, how do you know if you’re successful. This can, and almost certainly will, lead to scope creep in several directions if not nailed down.

    • Capacity
      • How much data are we trying to protect
      • How much replication traffic will there be between sites
      • How many simultaneous streams do we need to support
      • How many servers are we backing up
      • How much will de-duplication save
    • What kind of data are we protecting
      • VMs
      • Filesystems
      • Databases
    • How will we transfer the data
      • Network
      • SAN
    • What are we protecting against
      • Accidental deletion
      • Filesystem corruption
      • Loss of a server
      • Loss of a storage subsystem
      • Loss of a site
      • Rogue agents within the business
    • What’s the minimum RPO and maximum RTO we are aiming for
      • This will be affected by backup size/duration/policy
    • Are there any specific security requirements
      • Encryption – minimum cipher strength, on data and/or control traffic
      • Authorisation – granularity of access control

    I’m sure those who have spent more time dealing with backups than I have, could easily add to this list!

  • Get people to think about what they are requesting backups for, and the impact of taking them, or not taking them.
    If you don’t put some constraints on what should be backed up, you might end up trying to backup the world. Nervous admins will invariably ask for everything to be backed up, “just to be on the safe side”, when the service might be recoverable through an automated build process.

    Some questions to think about are:

    • Why do we need to backup this data (or server)?
    • Why do we need to backup this data now, if it hasn’t been backed up before?
    • Can we not recover the data or server any other way?
    • How would we recover the service, if the data is restored from backup?
    • What would be the impact if the data is lost and we couldn’t recover it?
    • What is the failure scenario we are aiming to recover from?
    • How quickly does the data need to be recovered? (RTO)
    • How recent does the backup need to be, to be worthwhile? (RPO)
    • How long do the backups need to be kept? (Retention)
    • Who owns the data?
    • Is the data subject to any Compliance legislation? (eg PCI DSS)
    • When can the backups be taken?
    • Is there any impact to the service when the backups are running?
    • Is there any impact to the service if the backups are not working?
    • If a backup is missed, do we need to reschedule it?
    • Do the backups need to go off-site, or to a different geographic region?
    • What is the size of the backup?
    • What is the delta change?
    • Will there be a regularly scheduled restore test?
    • Who can request a data restore?
    • Who can request expiry of the stored backups?
    • Who can request removal of the backup policy?

    If you don’t put some constraints on what should be backed up, you might end up trying to backup the world. This set of questions can help you verify the need for a backup, as well as important constraints and factors that will be needed for running them in day-to-day operations.

  • Other thoughts
      • Before you put the new backup service into production usage:
        • Get all your install/config/upgrade automation tested
        • Get everything on matching versions
        • Run vulnerability scans and fix any issues
        • Involve your operational teams
        • Get your processes and procedures agreed
      • Plan out and prioritise your project tasks, make sure you deliver what is required, save anything else for ‘phase 2’
    • And finally, learn to let go! Tie up (or hand over) any loose ends, and let the operational staff run the backups.
      Ok, I’m finding this one difficult, it’s become my baby for the last 4 months, but I’m trying.

      PLEASE TAKE AWAY MY ACCESS SO I CAN’T KEEP CHECKING IT’S ALL STILL WORKING!!!!

Windows Failover Cluster VM Snapshot Issue

I configured my first WFC servers a few weeks back, having previously been at an all Veritas Cluster Server shop. Nothing particularly special about them, in fact 2 of the clusters are just 2 node clusters with an IP resource acting as a VIP.

We came to configuring backups this week, and the day after the backup had run on one of the cluster nodes, I noticed that the resource had failed over to the second node in the cluster.

Digging into the eventlog showed a large number of NTFS warnings (eventIds 50, 98, 140), as well as errors for FailoverClustering  (eventIds 1069, 1177, 1564) and Service Control Manager (eventIds 7024, 7031, 7036).

wfcerrors

A bit of digging into KB articles such as KB1037959 reveals that snapshotting is not supported with WFC.

However, the issue seems to be caused by quiescing the VM and capturing the memory state with the snapshot. Just snapshotting the disk state does not appear to cause any issues with NTFS or Clustering in our testing, but obviously this is just a crash-consistent backup.