In this article I would explain how to setup a failover to second ISP when you have multiple campuses with different ISPs.

To begin with we have two sites "HQ" and "Branch". Each site has single ISP, NAT and DHCP. My "ISPs" are running BGP between each other.

With the current setup if uplink on one site goes down, that site would loose access to internet and would not be accessible.

To mitigate that we would add PTP link between our "Edge" routers and spin up dynamic routing, so when uplink goes down we would reach internet via other site's "Edge" router.

Why Dynamic Routing?

Simple, with dynamic routing if we add another subnet routing information would update by itself, simplifying management of our "Edge" devices.

Choosing a right dynamic protocol can be a challenge. Depending on our needs we might want to use EIGRP, iBGP, OSPF or RIP(please don't). Each protocol has benefits and drawbacks. In majority of a cases you would use OSPF, so that's what we are going to do in this example.

And here is a code that I used to configure PTP with OSPF:

enable
conf t
int gig0/2 ###interface of our PTP link
no shut 

ip address 192.168.254.1 255.255.255.252  ###ip for our PTP .2 for other side
ip nat inside ###would need this later when configuring NAT

router ospf 1111 ###starting OSPF with process id of 1111
network 192.168.254.0 255.255.255.252 area 0 
redistribute connected ###announce connected networks to OSPF

end
wr me

After we configured OSPF on both ends it might take a few moments before we see this:

00:52:08: %OSPF-5-ADJCHG: Process 1111, Nbr 192.168.254.1 on GigabitEthernet0/2 from LOADING to FULL, Loading Done

This message tells us that we successfully exchanged routers with our OSPF neighbor. We can verify that we are see routes from OSPF by running:

Edge-HQ#sh ip route ospf 
O E2 192.168.0.0 [110/20] via 192.168.254.2, 00:07:11, GigabitEthernet0/2

Now we are exchanging information about our networks on different sites, but we still can't failover to other ISP. We need to tweak our NAT and allow OSFP to advertise default routes to neighbors.

My NAT is configured with following commands:

###Edge-HQ

interface GigabitEthernet0/0
 ip address 1.1.1.2 255.255.255.252
 ip nat outside
 
interface GigabitEthernet0/1
 ip address 10.0.0.1 255.255.0.0
 ip nat inside
 
interface GigabitEthernet0/2
 ip address 192.168.254.1 255.255.255.252
 ip nat inside
 
ip nat inside source list 7 interface GigabitEthernet0/0 overload
access-list 7 permit 10.0.0.0 0.0.255.255

### "7" Here is a number of my access-list, you might want to change it


###Edge-Branch

interface GigabitEthernet0/0
 ip address 2.2.2.2 255.255.255.252
 ip nat outside

interface GigabitEthernet0/1
 ip address 192.168.0.1 255.255.255.0
 ip nat inside

interface GigabitEthernet0/2
 ip address 192.168.254.2 255.255.255.252
 ip nat inside

ip nat inside source list 7 interface GigabitEthernet0/0 overload
access-list 7 permit 192.168.0.0 0.0.0.255

And to allow NAT of other site's network, I'm going to add site network to access-list (ACL):

###Edge-HQ
conf t
access-list 7 permit 192.168.0.0 0.0.0.255
end
wr me

#Edge-Branch
conf t
access-list 7 permit 10.0.0.0 0.0.255.255
end
wr me

And finally adding advertisement for default routes via OSPF:

###Apply this code to every "Edge" router
conf t
router ospf 1111
default-information originate
end
wr me

Testing failover

Now when all the config is done, when I shut one of my uplinks I see following:

Edge-Branch#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
Edge-Branch(config)#int gig
Edge-Branch(config)#int gigabitEthernet 0/0
Edge-Branch(config-if)#shut
Edge-Branch(config-if)#
%LINK-5-CHANGED: Interface GigabitEthernet0/0, changed state to administratively down
%LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/0, changed state to down
Edge-Branch#end
%SYS-5-CONFIG_I: Configured from console by console


Edge-Branch#sh ip route 
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2, E - EGP
       i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area
       * - candidate default, U - per-user static route, o - ODR
       P - periodic downloaded static route

Gateway of last resort is 192.168.254.1 to network 0.0.0.0

     192.168.0.0/24 is variably subnetted, 2 subnets, 2 masks
C       192.168.0.0/24 is directly connected, GigabitEthernet0/1
L       192.168.0.1/32 is directly connected, GigabitEthernet0/1
     192.168.254.0/24 is variably subnetted, 2 subnets, 2 masks
C       192.168.254.0/30 is directly connected, GigabitEthernet0/2
L       192.168.254.2/32 is directly connected, GigabitEthernet0/2
O*E2 0.0.0.0/0 [110/1] via 192.168.254.1, 00:08:38, GigabitEthernet0/2

Default router for "Edge-Branch" changed to "Edge-HQ". Branch router recognized that despite of not having uplink to ISP, it can reach internet via other edge router.

Another important note is that when packet is being received my ISP, it would come from public IP of site that sent said packet. In our case packet would have public IP of "Edge-HQ" router.

Another benefit of PTP link is that we can access resources in different site without S2S/L2L VPN tunnels. Here is me pinging "Client-HQ" from "Client-Branch".

C:\>ping 192.168.0.2

Pinging 192.168.0.2 with 32 bytes of data:

Reply from 192.168.0.2: bytes=32 time=7ms TTL=128
Reply from 192.168.0.2: bytes=32 time=3ms TTL=128
Reply from 192.168.0.2: bytes=32 time=3ms TTL=128
Reply from 192.168.0.2: bytes=32 time=3ms TTL=128

Mitigating single point of failure for PTP link

With current setup if our PTP link goes down we would loose access to recourses on other site.

There are a number of ways we can mitigate that:

1) Add another physical link and bond them in Etherchannel.

That would allow us to mitigate single link failure. Has a drawback of extra cost if we lease this link from provider, but no extra overhead.

2)Setup a GRE tunnel between edge routers across public internet.

While this would work and allow access to recourses, it does not provide encryption. And you would want encryption for your internal traffic. Slight overhead, might need to adjust MTU to 1476.

3)Setup a S2S/L2L VPN tunnel between edge routers across public internet.

This would work like a GRE tunnel with added benefit of encryption. Many decent enterprise or prosumer routers/firewalls have this option. Even more overhead, not only we need to receive traffic, we also need to decrypt it.

VPN tunnels and routing

When adding VPN tunnels to mitigate failure of PTP link, it is important to add choose your routing accordingly. Now when you have two paths to choose from you need to determine primary. In my example choosing PTP for primary path is a natural choice, since it doesn't add overhead of encryption and requires less hops.

Here is when Administrative Distance (AD) comes in. Every route in router has AD, exact values are vendor dependent. When multiple routes exists for same network, route with lower AD is preferred.

Since a link with OSPF is my preferred path I need to make sure when adding static route for my tunnel I'm going to use AD higher than 110.

conf t
ip route 192.168.0.0 255.255.255.0 *tunnel IP* 250
end
wr me

With that config when our PTP link goes down and routes are no longer learned via OSPF, we can reach "Edge-Branch" internal network via our tunnel. I would dive deeper into S2S/L2L VPN tunnels in my future guides.

As a network engineer, one of the most powerful tools in your arsenal for managing and controlling network traffic is Border Gateway Protocol (BGP). BGP allows you to influence the flow of traffic within and across autonomous systems by manipulating various attributes associated with route advertisements. In this article, we'll delve into the intricacies of BGP attributes and explore how you can use them effectively to shape and optimize network traffic according to your organization's needs.

Before we dive into the practical aspects of influencing network traffic, let's first establish a solid understanding of BGP attributes. BGP routes are accompanied by a set of attributes that describe various characteristics of the route. These attributes are crucial for BGP routers to make routing decisions. Some of the key BGP attributes include:

1. AS_Path: This attribute specifies the sequence of autonomous systems that a route has traversed. It helps prevent routing loops and influences the selection of the best path.
2. Next Hop: The Next Hop attribute indicates the IP address of the next router along the path to the destination network.
3. Weight: Weight is a Cisco-specific attribute used to influence the preferred route for outbound traffic from a router.
4. Local Preference: Local Preference is an attribute used to influence the outbound routing decision within an autonomous system (AS).
5. Multi-Exit Discriminator (MED): MED is used to influence the inbound routing decision when multiple entry points exist into an AS.
6. Community: Communities are tags that can be attached to routes to influence routing policies within and across ASes.

Now that we have a basic understanding of BGP attributes, let's explore how we can leverage them to influence network traffic:

Inbound Traffic Management: Using attributes like AS_Path Prepending or MED, you can influence how traffic enters your network from external peers or providers. This allows you to control which entry points traffic takes into your network, thereby optimizing inbound traffic flows.

router bgp 65001
  neighbor 203.0.113.1 remote-as 65002
  neighbor 203.0.113.1 route-map PrependAS out
  
route-map PrependAS permit 10
  match as-path 1
  set as-path prepend 65001 65001 65001 

This is an actual prepending, here we tell other AS how many ASes traffic would travesre until it reaches us. Traffic would choose path with less ASes.

clear ip bgp 203.0.113.1 soft

Here we gently clearing current sessions so traffic gently would choose new path 
when current sessions end

Outbound Traffic Control: By adjusting attributes like Local Preference or Weight, you can influence the selection of outbound routes from your network. This is particularly useful for directing traffic towards preferred egress points or optimizing the use of available network resources.

router bgp 65001
  neighbor 192.0.2.1 remote-as 65002
  neighbor 192.0.2.1 route-map OutboundTrafficControl out
  
ip prefix-list Network-Prefixes seq 10 permit 10.0.0.0/24
route-map OutboundTrafficControl permit 10
  match ip address prefix-list Network-Prefixes
  set local-preference 200

Default local-preference 100. Traffic would choose path with higher local preference

Policy-based Routing: BGP communities provide a mechanism for implementing policy-based routing. By tagging routes with specific communities, you can apply routing policies based on predefined criteria such as geographic location, customer type, or service level agreements (SLAs).

Best Practices and Considerations:
While manipulating BGP attributes offers significant flexibility in controlling network traffic, it's essential to approach it with caution and adhere to best practices:

• Document Your Changes: Keep detailed documentation of any BGP attribute modifications to ensure clarity and facilitate troubleshooting.
• Monitor and Analyze: Regularly monitor and analyze network traffic patterns to assess the effectiveness of your BGP attribute manipulations and make adjustments as necessary.
• Testing and Validation: Before implementing changes in a production environment, conduct thorough testing and validation to minimize the risk of disruptions. From my experience it is also a good idea to get a peer review.

Conclusion:
In summary, BGP attributes are powerful tools that network engineers can use to influence the flow of traffic within and across their networks. By understanding the various BGP attributes and how they can be manipulated, you can effectively optimize network performance, improve reliability, and ensure efficient resource utilization. However, it's crucial to approach BGP attribute manipulation with care, following best practices and exercising caution to avoid unintended consequences. With careful planning and implementation, you can leverage BGP attributes to masterfully control network traffic according to your organization's specific requirements and objectives.