VMware NSX architecture

The VMware NSX architecture consists of multiple layers which provide an opportunity for growth and scalability. One of the main ideas is the independent work of the NSX data layer if the above layers of the architecture are not functional. The VMware NSX architecture is basically divided into three main layers: Data Plane, Control Plane and Management Plane.

VMware NSX architecture

VMware NSX architecture

VMware NSX components

Let’s check the main components of NSX architecture. More details can be found in this great book.

VMware NSX Manager – management plane, registered with vCenter on 1:1, REST API entry point
VMware NSX Controller Cluster – control plane function for logical switching and routing, maintains all info about hosts, logical switches and routers
VMware NSX Virtual Switch – Distributed switch + VXLAN, DLR, firewall. Components are installed as VIBs on hosts
Edge Services Gateway – Edge service.s such as firewall, NAT, DHCP, VPN, load balancing and high availability
VXLAN – overlay protocol that provides creation of logical layer2 networks over existing IP networks
Distributed Logical Firewall – allows segmentation of virtual data based on virtual machines names, attributes, IP addresses and ports
Distributed Logical Router – provides optimal east-west traffic routing in hypervisor, VMs on different subnets does not need to go over traditional routing interface
East-West traffic – Network traffic inside data center with Edge gateway as the border point
North-South traffic – Network traffic that is exchanged between the outside world and data center

Data Plane

The data plane is the VMware NSX layer where network traffic takes place. NSX components are installed directly on the ESXI host via the NSX manager. If all the other layers do not work, the existing network traffic will continue to flow. Additional NSX components are installed in the form of standard VIBs in the hypervisor kernel: VXLAN, Distributed Logic Router and Firewall. Only component that is not installed as VIB but virtual machine is Edge gateway. NSX switch or logic switch is a standard distributed switch with additional functionality (VXLAN,DLR and Firewall).

VXLAN allows L2 network traffic to pass through L3 network devices creating an additional network tunnel, which is used for traffic flow. VXLAN adds the additional payload to the standard network frame (MTU 1500). Standard MTU size should be extended to 1600.

Distributed logic router (DLR) is part of the L3 network. Main reason for introducing this component is avoiding the so-called “hairpinning” effect, where every time a virtual machine sends traffic to another subnet, traffic has to leave ESXI host and go through a physical L3 router. Directing routing functionality to the ESXi host itself, i.e. the kernel hypervisor, there is significant optimization and resource savings together with the significant reduction of unnecessary network traffic. DLR is installed on every ESXI host where it is used as classic L3 router between all virtual machines. Of course, instead of physical Ethernet and Fast Ethernet interfaces, DLR contains logical interfaces called LIFs that are used as the default gateway for virtual machines.

NSX distributed firewall is part of the ESXI kernel, installed on all ESXI hosts. NSX firewall is placed very close to the virtual machine – the virtual network card. Each packet that leaves the virtual machine before encapsulation and before it arrives at the destination virtual machine is analysed by NSX distributed firewall. Security rules can be applied from L2 to L4 and third-party software packages are used to upgrade inspection up to L7. VMware NSX documentation can provide more details about NSX architecture and firewall options.

NSX Edge is a multi-functional virtual machine that is not part of the ESXI kernel and has the following features:

– Routing (OSPF, BGP, Static)
– Firewall
– Load balancer
– IPSEC VPN
– L2 VPN
– SSL VPN
– DHCP and DNS relay

Unlike the DLR, NSX Edge is installed on the perimeter of the data center network and serves as the north-south traffic end point. In other words, it splits the traffic that takes place between ESXI hosts (east-west) from the outside world (north-south). Installation process is done through the NSX Manager responsible for installing and managing the entire NSX architecture. NSX Manager is part of the management layer.

Control plane

The VMware NSX control plane is the central part of the architecture and consists of the following components: NSX Logical Router VM, NSX Controller Cluster and User World Agent. The most important component of the control layer is the NSX Controller Cluster which performs the following functions:

  1. Eliminates the need to configure multicast traffic on network devices
  2. Eliminates ARP broadcast network traffic
  3. Distributes routing information to ESXi hosts
  4. Performs load balancing between three mandatory NSX Controller Clusters
  5. Maintains tables for VXLAN and MAC addresses
  6. Maintains ARP tables

NSX Logical Router VM is a virtual appliance used in the case dynamic routing implementation. Supported dynamic routing protocols are OSPF and BGP. Static routing is implemented by distributed logical routers (DLRs) that runs on each ESXI hypervisor and contains all the necessary routing information. Once dynamic protocols are configured, NSX Logical Router VM receives the necessary routing information via NSX Manager. The routing information is then forwarded to the NSX Controller Cluster that distributes routing information to all ESXi hosts.

NSX Manager distributes user world agents to ESXi hosts during installation of NSX components. There are two types of agents: netcpa and vsfwd. Agent netcpa uses SSL encryption within the control layer, sends information about NSX Controller Cluster virtual machines such as IP and MAC addresses, receives information directly from the NSX Manager via the message bus agent. Agent vsfwd receives security policies via NSX Manager, saves firewall statistics and sends this information to NSX Manager. NSX Manager sends commands to create and delete a distributed logical routers and edge routers.

Security

The NSX manager creates self-signed certificates for the control and data layer. The entire data layer is encrypted with SSL protocol. Mutual authentication exists during layer communication process.

Management plane

The VMware NSX control layer consists of: vCenter, NSX Manager, and Message Bus Agent. vCenter and NSX Manager are always in the 1:1 relationship, i.e. if you have more vCenters you need same number of NSX managers.

NSX Manager is the central place for managing the entire NSX infrastructure. Primary features are:

  1. User interface and VMware NSX API
  2. Installs user world agents, VXLANs, distributed logic routing and kernel modules for distributed firewall
  3. Installs VMware NSX Controllers
  4. Configures ESXI hosts through message bus agent
  5. Generates self-signed certificates for secure communication

Message bus agent is a communication bridge between NSX Manager and ESXI hosts.

Consumption Model communicates with NSX Manager via the API and consists of self-service portals, Cloud management, vRealize Automation and many more.

VMware NSX components installation order

  1. Installation of NSX Manager
  2. NSX Manager registers with vCenter Server
  3. NSX Manager installs 3 NSX Cluster Virtual Machine Controllers
  4. NSX Manager installs the necessary VIBs on ESXi hosts
  5. NSX Manager installs the NSX Edge router and performs a configuration of network services