NSX Controller is a highly available virtual appliance responsible for installing virtual networks across the entire NSX architecture. The NSX controller is part of the control layer and is logically separated from the data layer where all network traffic takes place. It is the central point for all logical switches, keeps information about all virtual machines, ESXI hosts, logical switches and VXLANs. The minimum and maximum number of NSX controllers is three. The NSX controller communicates with the NSX Manager via the SSL API, and also relies on the SSL connection to communicate with ESXI hosts.

What are the functions of the VMware NSX controller?

  1. Distribution of VXLAN and logical routing information to ESXi hosts
  2. Scalable and highly available cluster system
  3. Balancing the load between the NSX cluster controllers
  4. Maintaining the VXLAN, MAC and ARP tables
  5. Removing the dependency on the multicast configuration on the physical network
  6. Removing ARP broadcast traffic in the VXLAN network

Communication with other NSX components

NSX Controllers communication

NSX Controllers communication

NSX Manager -> NSX Controllers (TCP/443) – Manager to Controller communication. The control layer communicates with the NSX Controller via the security port 443.

ESXI Host -> NSX Controllers (TCP/1234) – secure connection over SSL communication channel

NSX Controller <-> NSX Controller (TCP/2878, 2888, 3888) – State Sync between controllers

NSX Controller <-> NSX Controller (TCP/7777) – Inter-Controller RPC Port

NSX Controller <-> NSX Controller (TCP/30865) – Controller Cluster – State Sync

NSX Controllers -> NTP Server (TCP/123, UDP/123) time synchronization with NTP server(s)

Communication between the two levels, the ESXI hosts and the NSX controller is done with help of user world agent. Communication channel is encrypted and runs on ESXI hosts in the form of netcpa deamon. All virtual machines that are connected to the network inform the NSX controllers about all changes in the network configuration via the user world agent. The user world agent is not used to distribute firewall information. Learning NSX gives more details about the NSX Manager.

VMware NSX controller tables

MAC table

The NSX controller matches VTEP interface with virtual machine by pairing the MAC addresses with the corresponding VTEP interfaces. The table is populated by user world agent.

ARP table

It is used to eliminate ARP broadcast traffic by matching the IP address of the virtual machine with the associated MAC addresses. When virtual machine is started, and network packets begin to flow, local VTEP maps VMs IP’s with MAC address. This information is then sent to NSX controllers.

VTEP table

The table stores information about all VTEP IP and MAC addresses with the associated VNIs and subnets.

What if the NSX Controller becomes unavailable?

Since the minimal installation includes three NSX controllers, let’s take a look at different scenarios:

Failure of one NSX controller

  1. The majority of the NSX controllers in the retained and everything works smoothly
  2. In case of a primary NSX controller failure, the remaining two NSX controllers participate in the election process and new primary NSX controller is selected

Failure of two NSX controllers

  1. Cluster majority is lost
  2. The control layer is switched to read-only mode, but the data layer works smoothly
  3. It is recommended to delete the remaining controller and reinstall all three from scratch

Failure of all three NSX controllers

The same scenario as the failure of two NSX controllers

Resource allocation between the NSX controllers

We know that, unlike the NSX manager, the NSX controller must be installed in the form of three virtual machines. One of the NSX controllers is selected as a primary controller, and allocates resources to other controllers. The load is distributed in slides when one controller fails, and slides are distributed to the remaining NSX controllers.

Design guidelines

  1. NSX controllers should be installed inside the same vCenter where NSX Manager is installed
  2. Together with Anti-affinity rules cluster should be configured with at least three ESXi hosts
  3. VMware HA, which in the event of a failure of one ESXi host, starts the controller on another ESXI host
  4. Arrange ESXi hosts in different racks so that the failure of the top-of-the-rack switch in the rack does not affect the failure of the entire cluster
  5. It is recommended to configure 4vCPU and 4GB for each NSX controller