Objective 2.1 – Define Benefits of Running VMware NSX on Physical Network Fabrics


  • Identify physical network topologies (Layer 2 Fabric, Multi-Tier, Leaf/Spine, etc.)
    • Layer 2 Fabric
      L2 is the data link layer in ISO. Communication via MAC addresses (IPs looked up to MAC).
      All connected nodes are in the same broadcast domain.
    • 3 tier network
      • Core – routing between distribution switches, VPN, internet (Nexus 7000/9000)
      • Distribution – L3 routing between connected access switches (Nexus 5000)
      • Access – top of rack switches, typically L2 only, or fabric extender
      • Reduces structured cabling
    • Collapsed core
      • Core and distribution are combined
      • Top of rack switches may be L2 or L3
      • More likely for 100% virtualised DC
    •  Leaf/Spine
      • Like collapsed core but all links are forwarding
      • Because all links are forwarding, gives real horizontal scalability benefits
      • Can’t use Spanning Tree (as it is a path blocking protocol)
    • Traffic flows
      • East-West – ie between application layer. E-W flows have to be routed (hair-pinning)
      • North-South – up to internet/vpn, down to physical DB
    • Traffic flows in NSX
      • East-West – uses distributed routers/logical switches to keep the traffic in virt layer. Only a single L2 network used in the Transport Zone for VM to VM traffic
      • North-South – uses “edge” logical routers which sit on perimeter, can have ECMP load balancing of up to 8 devices, and OSPF used to control traffic flow.
      • F5 have created a VTEP which means physical F5 can be E-W traffic
  • Identify physical network trends
    • Multi-tier networks are migrating to Spine-Leaf networks because of Cloud Computing
    • Port speeds are increasing 1G->10G->40G
  • Explain the purpose of a Spine node
    • Part of the aggregation/spine layer that provides connectivity between racks
  • Explain the purpose of a Leaf node
    • Typically located inside a rack and provides network access to the servers inside that rack
  • Identify virtual network topologies (Enterprise, Service Provider Multi-Tenant, Multi-Tenant Scalable)
    • Enterprise
    • Service Provider – Multi-Tenant
    • Multi-Tenant Scalable
  • Explain benefits of Multi-Instance TCP/IP stack
    • Helps you move to a software defined datacenter model.
    • Useful in environments that are nearing the 4000 VLAN limitation
    • Abstracts the VM layer from the physically allocated VLANs
  • Describe challenges in a Layer 2 Fabric topology
    • Scalability (arp/mac table size)
    • Broadcast, Unknown unicast, and Multicast (BUM) traffic flooding
  • Describe challenges in a Multi-Tier topology
    • Scalability
    • Fault tolerance
    • Energy efficiency
    • Cross-sectional bandwidth
    • Higher layers are highly oversubscribed
  • Describe challenges in a Leaf/Spine topology
    • Specific VLANs exist only in a single rack
    • Full Mesh connectivity between Leaf and Spine layers becomes complex as size increases.
    • Can’t use Spanning Tree multipathing – use ECMP, TRILL, ISIS etc
  • Differentiate physical/virtual QoS implementation
    • Physical
      • Generally marked for QoS by an ingress router interface
      • Difficult to manage
      • Different tenants may have different QoS values
      • Supports L2 QoS sometimes referred to as “Class of Service” (CoS) and L3 QoS referred to as “DSCP marking”
    • Virtual
      • Marked for QoS by the Hypervisor
      • Network infrastructure must trust values set by the Hypervisor (trusted boundary)
      • NSX can either trust and DSCP marking applied by a VM or explicitly modify and set it at the Logical Switch level.
  • Differentiate single/multiple vSphere Distributed Switch (vDS) Distributed Logical Router implementations
    • Single vDS
      • Can share compute resources between Compute and Edge clusters
      • One of the requirements of a single-VDS–based design is that a single VLAN is defined for VXLAN transport network.
    • Multiple vDS

      • Although a design with a single VDS spanning both compute and edge cluster is possible, there are several advantages in keeping separate VDS for compute and edge:
      • Flexibility of span of operational control: typically compute/virtual infrastructure admin and network admin are separate entities and thus each domain can manage the cluster specific tasks. These benefits are already a factor in designing a dedicated cluster and rack for specific services and further substantiated by the VDS design choices.
      • Flexibility in managing uplink connectivity on computes and edge clusters – see for example the above discussion on uplink design and the recommendation of using different teaming options for compute and edge clusters.
      • Typically the VDS boundary is aligned with the transport zone and thus VMs connected to logical switches can span the transport zone. However, the vMotion boundary is always limited by the extension of a VDS, so keeping a separate VDS for compute resources ensures that those workloads will never be moved (by mistake or by choice) to ESXi hosts dedicated to other services.
      • Flexibility in managing VTEP configuration – see above discussion on VTEP design choices.
      • Avoiding exposing VLAN-backed port-groups used by the services deployed in the edge racks (NSX L3 routing and NSX L2 bridging) to the compute racks.
  • Differentiate NSX Edge High Availability (HA)/Scale-out NSX Edge HA implementations
    • NSX Edge HA
      If the active edge fails, the standby takes over and assumes the outside IP address of the previously active edge. To notify the upstream infrastructure (the L2 switches that potentially interconnect the Edge and the first physical router) a GARP message is sent out. For this mechanism to work, a VLAN must be extended between the edge racks. Tunnel interfaces connecting the VXLAN endpoints do not have to extend any VLAN. Before the failover, the hypervisor VTEPs sent traffic to the VTEP of the hypervisor hosting the edge. After failover, that traffic is sent to the VTEP of the hypervisor that hosts the newly active edge.
    • Scale out NSX Edge HA
      NSX 6.1 introduces support for Active/Active HA using ECMP This HA model provides two main advantages:

      • An increased available bandwidth for north-south communication (up to 80 Gbps per tenant).
      • A reduced traffic outage (in terms of % of affected flows) for NSX Edge failure scenarios.


  • Differentiate Collapsed/Separate vSphere
    • Collapsed vSphere Cluster topology

      • In small/medium data center deployments a single vCenter is usually deployed for managing all the NSX components. The recommendation is still to dedicate separate clusters and set of racks for compute resources. It is also recommended to deploy separate edge and management clusters to accommodate future growth.
      • The edge and management racks are usually consolidated and the corresponding clusters of ESXi hosts share Top-of-Rack connectivity
    • Separate vSphere Cluster topology
      Most enterprise deployments make use of a dedicated vCenter for the management cluster. This vCenter is usually already deployed even before the NSX platform is introduced in the architecture. When that happens, one or more dedicated vCenter servers part of the management cluster are usually introduced to manage the resources of the NSX domain (edge and compute clusters)
      There are several advantages in adopting such approach:

      • Avoids circular dependencies – the management cluster should always be outside of the domain it manages.
      • Mobility of management cluster for remote DC operation.
      • Integration with existing vCenter.
      • Ability to deploy more than one NSX-domain.
      • Upgrade of main vCenter does not affect the NSX domains.
      • SRM and other explicit state management are possible.
      • Additionally, a large-scale design employs one or more dedicated racks (infrastructure racks) and ToR switches to host the management cluster
  • Differentiate Layer 3 and Converged cluster infrastructures
    • Layer 3 in Access Layer
      This architecture is designed to allow for future growth, working for deployments that begin small but can grow to large-scale while keeping the same overall architecture. The guiding principle is that VLANs do not span beyond a single rack. This has a significant impact on how a physical switching infrastructure can be built and how well it scales.
      Edge Racks using HA need a VLAN to be extended between the edge racks. Tunnel interfaces connecting the VXLAN endpoints do not need to extend any VLAN.
      If the Management cluster is deployed across 2 racks (to survive a rack failure scenario) they require extending VLANs across those racks for management workloads such as VCenter, NSX Controllers, NSX Manager, and IP Storage. The recommended way to provide this is using dedicated cross-rack L2 cables to interconnect the ToR switches and dual home every server to both ToR switches
    • Converged cluster
      With a converged cluster, the Management and Edge are provided in the same vSphere cluster, with a shared L2 layer. This limits the scalability.


  • VMware NSX Network Virtualization Design Guide
  • NSX User’s Guide

One thought on “Objective 2.1 – Define Benefits of Running VMware NSX on Physical Network Fabrics

  1. Pingback: VMware VCP-NV NSX Study Resources | darrylcauldwell.com

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s