Nexus 2248TP FEX connected to a Nexus 7000: part 1 basic connection
Cisco published a configuration guide for connecting a Nexus 2248TP FEX to a Nexus 7000. I’ll explain the configuration process to configure the FEX for basic connectivity.
A FEX is a Nexus 2000 series switch. In a very high level explanation, this switch is a switchport module in a separate 1U chassis, which is configured and controlled from a Nexus 5000/7000. There is 1 drawback: the switchports on the FEX can only be used for host ports. It’s not possible to connect other switches to a FEX port because of the fact that BPDUguard is enabled by default. It’s not possible to disable BPDUguard. Switchports on the FEX can be used for layer 2 and layer 3 connections. For more information about the FEX itself I’ll refer to this link
All configuration is done on the Nexus 7000 with NX-OS 6.0(1). In this scenario, the 2248TP FEX is connected to switchport ethernet 1/1 (10GE port) of the Nexus 7000 with a twinax cable.
First, let’s configure a switchport for the FEX:
switch(config)# int ethernet 1/1 switch(config-if)# switchport mode fex-fabric Error: feature-set fex is not enabled
Okay, we have to install (!) and enable the fex feature before we can continue
switch(config)# install feature-set fex switch(config)# feature-set fex
And try again to configure the fex-fabric mode:
switch(config)# int ethernet 1/1 switch(config-if)# switchport mode fex-fabric switch(config-if)# fex associate 101 switch(config-if)# no shut
This fex associate number defines the physical FEX to the FEX configuration. The range is 101 – 199. In this case, the FEX connected to switchport 1/1 will be numbered as 101. It’s possible to assign a FEX serial number to the FEX configuration. With this configuration, only the FEX with that particular serialnumber can be connected to these ports, and receives this configuration.
switch(config-if)# sh int status -------------------------------------------------------------------------------- Port Name Status Vlan Duplex Speed Type -------------------------------------------------------------------------------- mgmt0 -- connected routed full 100 -- Eth1/1 -- intFailEr 1 full auto SFP-H10GB-A
As you see, the switchport is in error mode. Let’s first finish the FEX configuration before we troubleshoot the connection.
switch(config)# fex 101 switch(config-fex)# description FEX-test switch(config-fex)# type n2248t
Let’s check the logging:
switch(config)# sh logging 2012 Feb 15 08:42:50 switch %ETHPORT-5-IF_DOWN_ERROR_DISABLED: Interface Ethernet1/1 is down (Error disabled. Reason:Port is not a port-channel member)
The error message self-explaining: make the switchport member of a port channel:
switch(config)# int eth1/1 switch(config-if)# channel-group 1 command failed: port not compatible [Ethernet Layer] ** You can use force option to override the port's parameters ** (e.g. "channel-group X force") ** Use "show port-channel compatibility-parameters" to get more information on failure switch(config-if)# channel-group 1 force
The configuration looks like:
interface port-channel1 switchport switchport mode fex-fabric fex associate 101
Now it’s time to check the portchannel:
switch(config-if)# sh interface port-channel 1 fex-intf Fabric FEX Interface Interfaces --------------------------------------------------- Po1
Grab a coffee and wait a few minutes.
switch(config)# sh fex detail
FEX: 101 Description: FEX-test state: Online Sequence
FEX version: 6.0(1) [Switch version: 6.0(1)]
FEX Interim version: 6.0(0.4)
Switch Interim version: 6.0(1)
Extender Model: N2K-C2248TP-1GE, Extender Serial: FOC1549621C
Part No: 73-13232-01
Card Id: 99, Mac Addr: d8:24:bd:fc:25:02, Num Macs: 64
Module Sw Gen: 12594 [Switch Sw Gen: 21]
pinning-mode: static Max-links: 1
Fabric port for control traffic: Eth1/1
Fabric interface state:
Po1 - Interface Up. State: Active
Eth1/1 - Interface Up. State: Active
Fex Port State Fabric Port
Eth101/1/1 Down Po1
Eth101/1/2 Down Po1
Eth101/1/3 Down Po1
Eth101/1/4 Down Po1
Eth101/1/5 Down Po1
Eth101/1/6 Down Po1
Eth101/1/7 Down Po1
Eth101/1/8 Down Po1
Eth101/1/9 Down Po1
Eth101/1/10 Down Po1
Eth101/1/11 Down Po1
Eth101/1/12 Down Po1
Eth101/1/13 Down Po1
Eth101/1/14 Down Po1
Eth101/1/15 Down Po1
Eth101/1/16 Down Po1
Eth101/1/17 Down Po1
Eth101/1/18 Down Po1
Eth101/1/19 Down Po1
Eth101/1/20 Down Po1
Eth101/1/21 Down Po1
Eth101/1/22 Down Po1
Eth101/1/23 Down Po1
Eth101/1/24 Down Po1
Eth101/1/25 Down Po1
Eth101/1/26 Down Po1
Eth101/1/27 Down Po1
Eth101/1/28 Down Po1
Eth101/1/29 Down Po1
Eth101/1/30 Down Po1
Eth101/1/31 Down Po1
Eth101/1/32 Down Po1
Eth101/1/33 Down Po1
Eth101/1/34 Down Po1
Eth101/1/35 Down Po1
Eth101/1/36 Down Po1
Eth101/1/37 Down Po1
Eth101/1/38 Down Po1
Eth101/1/39 Down Po1
Eth101/1/40 Down Po1
Eth101/1/41 Down Po1
Eth101/1/42 Down Po1
Eth101/1/43 Down Po1
Eth101/1/44 Down Po1
Eth101/1/45 Down Po1
Eth101/1/46 Down Po1
Eth101/1/47 Down Po1
Eth101/1/48 Down Po1
Logs:
02/15/2012 08:55:16.78268: Module register received
02/15/2012 08:55:16.79093: Image Version Mismatch
02/15/2012 08:55:16.79695: Registration response sent
02/15/2012 08:55:16.79854: Requesting satellite to download image
02/15/2012 09:03:01.684836: Image preload successful.
02/15/2012 09:03:01.901494: Deleting route to FEX
02/15/2012 09:03:01.906381: Module disconnected
02/15/2012 09:03:01.910050: Module Offline
02/15/2012 09:03:01.918355: Deleting route to FEX
02/15/2012 09:03:01.923126: Module disconnected
02/15/2012 09:03:01.925539: Offlining Module
02/15/2012 09:04:14.606783: Module register received
02/15/2012 09:04:14.608763: Registration response sent
02/15/2012 09:04:15.411239: Module Online Sequence
In case you’ve a lot of FEX’s in your datacenter and you have to locate this particular FEX, it’s possible to start a beacon (blinking blue LED on the FEX) with this command:
switch(config)# fex 101 switch(config-fex)# beacon
We’re done! In a upcoming blog post I’ll cover multiple uplinks to the FEX and the pinning configuration.
unfortunately “beacon” start is not supported in the configuration. 🙁
Have you tried indeed? ❓
BR/Dimitris
Which nx-os version are you using? I’ll double check the beacon config on monday.
Great post, my question does the new NX-OS 6.0(1) support dual home connection, I mean for instance 4 FEX;s connected to each N7010 in a meshed-topology.
Thx
Maz
Since nx-os 5.2 it’s possible to multihome a FEX to multiple N7k’s in a portchannel over vPC.
I’ll try to make a new blogpost this week, about the configuration needed for this.