Spine and Leaf Practical Applications, EGP and IGP combined!

So far, all examples to date have been extremely simple, and stand rather well on their own (OSPF Clos would work just FINE in a campus network if you have cheap L3) but may not effectively address some more advanced use cases.

In short, we're about to enter niche territory.

Generally speaking, BGP is regarded as the most advanced networking topic out there, until it isn't. Most of the complexity lies in iBGP (same Autonomous System everywhere), because BGP's primary loop prevention mechanism is AS-Path (count on the number of AS this route transits).

To successfully implement iBGP (a staple of every carrier, so totally do-able), a network engineer must choose one of the following paths (non-inclusive list):

In this case, the more scalable option is to opt for a route reflector, but it isn't that easy.

iBGP, when compared to eBGP:

  • Doesn't care about hop count to a peer speaker. As long as the route reflector is less than 255 hops away, there is no issue
  • Usually doesn't care about resolving paths to peer speakers - that's a problem for something else.

To provide a good example of an expandable, scalable fabric that can offer eBGP as a service to subtending network devices, we will implement IS-IS as the intra-fabric routing protocol, and then leverage iBGP with the spine switches as route reflectors.

First things first, diagram is here: (YAML)

EGP+IGP Fabric

First, we'd configure the spines. Note that future releases (my home lab is rockin' IOS 12.2.55, too old for this) the BGP Dynamic Neighbor that comes with more modern network operating systems is really useful.

Note: route reflector client status is configured on the SERVER side:

 1router bgp 65000  
 2 bgp log-neighbor-changes  
 3 neighbor 10.6.0.0 remote-as 65000  
 4 neighbor 10.6.0.0 update-source L0  
 5 neighbor 10.6.0.0 route-reflector-client  
 6 neighbor 10.6.0.1 remote-as 65000  
 7 neighbor 10.6.0.1 update-source L0  
 8 neighbor 10.6.0.1 route-reflector-client  
 9 neighbor FD00:6::0 remote-as 65000  
10 neighbor FD00:6::0 update-source L0  
11 neighbor FD00:6::0 route-reflector-client  
12 neighbor FD00:6::1 remote-as 65000  
13 neighbor FD00:6::1 update-source L0  
14 neighbor FD00:6::1 route-reflector-client  
15 maximum-paths 2  
16 !  
17 address-family ipv4  
18  neighbor 10.6.0.0 activate  
19  neighbor 10.6.0.1 activate  
20  maximum-paths 2  
21  no auto-summary  
22  network 10.6.0.240 mask 255.255.255.254  
23  network 10.6.240.0 mask 255.255.255.254  
24  network 10.6.240.2 mask 255.255.255.254  
25 exit-address-family  
26 !  
27 address-family ipv6  
28  neighbor FD00:6::0 activate  
29  neighbor FD00:6::1 activate  
30  network FD00:6::240/127  
31  network FD00:6:240::/126  
32  network FD00:6:240::4/126  
33  no synchronization  
34  maximum-paths 2  
35 exit-address-family  
36!  

And then the leaf configuration:

 1  
 2router bgp 65000  
 3 bgp log-neighbor-changes  
 4 neighbor 10.6.0.240 remote-as 65000  
 5 neighbor 10.6.0.240 update-source L0  
 6 neighbor 10.6.0.241 remote-as 65000  
 7 neighbor 10.6.0.241 update-source L0  
 8 neighbor FD00:6::240 remote-as 65000  
 9 neighbor FD00:6::240 update-source L0  
10 neighbor FD00:6::241 remote-as 65000  
11 neighbor FD00:6::241 update-source L0  
12 maximum-paths 2  
13 !  
14 address-family ipv4  
15  neighbor 10.6.0.240 activate  
16  neighbor 10.6.0.241 activate  
17  maximum-paths 2  
18  no auto-summary  
19  network 10.6.0.240 mask 255.255.255.254  
20  network 10.6.240.0 mask 255.255.255.254  
21  network 10.6.240.2 mask 255.255.255.254  
22 exit-address-family  
23 !  
24 address-family ipv6  
25  neighbor FD00:6::240 activate  
26  neighbor FD00:6::241 activate  
27  network FD00:6::240/127  
28  network FD00:6:240::/126  
29  network FD00:6:240::4/126  
30  no synchronization  
31  maximum-paths 2  
32 exit-address-family  
33!  

Note that no BGP peers are up yet - and BGP knows what the problem is, too!

 1bgp-rr0-s0#show ip bgp  
 206:12:15: %SYS-5-CONFIG_I: Configured from console by consoleneigh  
 3BGP neighbor is 10.6.0.0,  remote AS 65000, internal link  
 4  BGP version 4, remote router ID 0.0.0.0  
 5  BGP state = Active  
 6  Last read 00:03:02, last write 00:03:02, hold time is 180, keepalive interval is 60 seconds  
 7  Message statistics:  
 8    InQ depth is 0  
 9    OutQ depth is 0  
10                         Sent       Rcvd  
11    Opens:                  0          0  
12    Notifications:          0          0  
13    Updates:                0          0  
14    Keepalives:             0          0  
15    Route Refresh:          0          0  
16    Total:                  0          0  
17  Default minimum time between advertisement runs is 0 seconds  
18  
19 For address family: IPv4 Unicast  
20  BGP table version 1, neighbor version 0/0  
21  Output queue size : 0  
22  Index 1, Offset 0, Mask 0x2  
23  Route-Reflector Client  
24  1 update-group member  
25                                 Sent       Rcvd  
26  Prefix activity:               ----       ----  
27    Prefixes Current:               0          0  
28    Prefixes Total:                 0          0  
29    Implicit Withdraw:              0          0  
30    Explicit Withdraw:              0          0  
31    Used as bestpath:             n/a          0  
32    Used as multipath:            n/a          0  
33  
34                                   Outbound    Inbound  
35  Local Policy Denied Prefixes:    --------    -------  
36    Total:                                0          0  
37  Number of NLRIs in the update sent: max 0, min 0  
38  
39  Address tracking is enabled, the RIB does not have a route to 10.6.0.0  
40  Address tracking requires at least a /0 route to the peer  
41  Connections established 0; dropped 0  
42  Last reset never  
43  Transport(tcp) path-mtu-discovery is enabled  
44  No active TCP connection  

Note how it says the RIB does not have a route to 10.6.0.0 - that's because iBGP doesn't resolve next-hops for us. Let's fix it by rolling out an Interior Gateway Protocol (IGP) to support iBGP here. I'm using IS-IS for a few reasons - namely:

  • Like BGP, one routing protocol for both IPv4 and IPv6
  • Selective flooding with ISPF
  • I'm too hip for OSPF now
 1router isis CLOS-1  
 2 net 42.0000.0000.0000.0240.00  
 3 is-type level-2-only  
 4 ispf level-2  
 5 log-adjacency-changes  
 6!  
 7interface   
 8ip router isis CLOS-1  
 9interface Loopback0  
10ip router isis CLOS-1  

This is applied to every router, while changing the net-ID for each device.It's fun watching the adjacencies pop up, so I'll add that here too.

1  
2*Mar  1 06:31:34.670: %CLNS-5-ADJCHANGE: ISIS: Adjacency to 0000.0000.0240 (FastEthernet0/22) Up, new adjacency  
3*Mar  1 06:31:34.670: %CLNS-5-ADJCHANGE: ISIS: Adjacency to 0000.0000.0241 (FastEthernet0/23) Up, new adjacency   
4*Mar  1 06:31:41.565: %BGP-5-ADJCHANGE: neighbor 10.6.0.240 Up  
5*Mar  1 06:31:47.169: %BGP-5-ADJCHANGE: neighbor 10.6.0.241 Up  

Note how BGP pops up immediately after IS-IS resolves the next-hop for the loopback in this case. Sadly, it doesn't look like my ancient lab switches support IS-IS for IPv6 - so I'll add OSPFv3 for the next topic- actually using Clos in a datacenter network

 1bgp-rr0-s0#show ip bgp sum  
 2BGP router identifier 10.6.0.240, local AS number 65000  
 3BGP table version is 3, main routing table version 3  
 42 network entries using 234 bytes of memory  
 54 path entries using 208 bytes of memory  
 63/1 BGP path/bestpath attribute entries using 420 bytes of memory  
 70 BGP route-map cache entries using 0 bytes of memory  
 80 BGP filter-list cache entries using 0 bytes of memory  
 9BGP using 862 total bytes of memory  
10BGP activity 4/0 prefixes, 6/0 paths, scan interval 60 secs  
11  
12Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd  
1310.6.0.0        4 65000       8       7        3    0    0 00:03:03        1  
1410.6.0.1        4 65000       6       5        3    0    0 00:01:51        1  
15FD00:6::        4 65000       0       0        0    0    0 never    Active  
16FD00:6::1       4 65000       0       0        0    0    0 never    Active  

Configurations generated by this lab, if you want to replicate are here.

Posts in this Series