ClusterFrontendOnCisco3560E

From BroWiki

About

Implementing a frontend for a 10Gb link is not easy even if the amount of traffic on the link is no where near line rate. One method architecting a system for doing this is laid out in a paper by Fabian Schneider [1] which takes advantage of the port group capabilities of managed switches. If the managed switch can split traffic among the ports in a port group by src and dst ip address in order to keep both directions of sessions on the same port, then the switch can be used as a frontend. One such switch that supports this is the Cisco 3560E.

There is a problem however, on Cisco hardware port groups only support up to 8 teamed ports and if a cluster has more than 8 workers nodes the switch can't scale high enough. This can be worked around by having Click routers [2] listening on the port channel ports and doing their own load balancing by rewriting MAC addresses on the packets then passing those packets on to another switch with the associated static MAC addresses applied to ports where the worker nodes will listen.

This technique is currently in use as the frontend for the Bro cluster at The Ohio State University. It's operating on a 10Gb/s fiber link, but the technique has been tested and shown to work on 1Gb/s Cisco hardware as well.

Hardware

This is hardware for tapping LR fiber, but it's applicable to SR fiber too.

• 4 port NetOptics 10G fiber tap (RGN-LR-IL4) [3]
• 24 port Cisco 3560E (3560E-24TD)
• (2) 10G X2 modules (X2-10GB-LR)

3560E Switch Configuration Example

The static MAC addresses that are assigned at the bottom of this config snippet contain the MAC addresses of the two router interfaces that are seen on the tapped link. This causes all of the packets going both directions over the 10G fibers to flow towards the Port-Channel. If a link is being tapped where many MAC addresses may be seen, a different technique should be used.

This configuration is also assuming that vlan 123 is being trunked over the 10G fiber link.

 no errdisable detect cause all
 system mtu routing 9198
 port-channel load-balance src-dst-ip
 !
 interface Port-channel1
  switchport access vlan 123
  switchport mode access
  load-interval 30
  delay 1
 !
 interface GigabitEthernet0/1
  switchport access vlan 123
  switchport mode access
  channel-group 1 mode on
  spanning-tree portfast
 !
 interface GigabitEthernet0/2
  switchport access vlan 123
  switchport mode access
  channel-group 1 mode on
  spanning-tree portfast
 !
 interface TenGigabitEthernet0/1
  switchport access vlan 123
  switchport trunk encapsulation dot1q
  switchport mode trunk
  load-interval 30
  speed nonegotiate
 !
 interface TenGigabitEthernet0/2
  switchport access vlan 123
  switchport trunk encapsulation dot1q
  switchport mode trunk
  load-interval 30
  speed nonegotiate
 !
 mac-address-table static 001b.c05f.304a vlan 123 interface Port-channel1 
 mac-address-table static 001c.b019.eab7 vlan 123 interface Port-channel1

One way of visualizing the packet flow is with a little graphic (packets flow left to right):

+-----------------------+                                  +-----------------------+           
|TenGigabitEthernet0/1  |\                                /|GigabitEthernet0/1     |           
+-----------------------+ \      +----------------+      / +-----------------------+           
                           X-----| Port-channel1  |-----X                                      
+-----------------------+ /      +----------------+      \ +-----------------------+           
|TenGigabitEthernet0/2  |/                                \|GigabitEthernet0/2     |           
+-----------------------+                                  +-----------------------+