Failover to Redundant ISP by Using PTP link

In this article I would explain how to setup a failover to second ISP when you have multiple campuses with different ISPs.

To begin with we have two sites "HQ" and "Branch". Each site has single ISP, NAT and DHCP. My "ISPs" are running BGP between each other.

With the current setup if uplink on one site goes down, that site would loose access to internet and would not be accessible.

To mitigate that we would add PTP link between our "Edge" routers and spin up dynamic routing, so when uplink goes down we would reach internet via other site's "Edge" router.

Why Dynamic Routing?

Simple, with dynamic routing if we add another subnet routing information would update by itself, simplifying management of our "Edge" devices.

Choosing a right dynamic protocol can be a challenge. Depending on our needs we might want to use EIGRP, iBGP, OSPF or RIP(please don't). Each protocol has benefits and drawbacks. In majority of a cases you would use OSPF, so that's what we are going to do in this example.

And here is a code that I used to configure PTP with OSPF:

enable
conf t
int gig0/2 ###interface of our PTP link
no shut 

ip address 192.168.254.1 255.255.255.252  ###ip for our PTP .2 for other side
ip nat inside ###would need this later when configuring NAT

router ospf 1111 ###starting OSPF with process id of 1111
network 192.168.254.0 255.255.255.252 area 0 
redistribute connected ###announce connected networks to OSPF

end
wr me

After we configured OSPF on both ends it might take a few moments before we see this:

00:52:08: %OSPF-5-ADJCHG: Process 1111, Nbr 192.168.254.1 on GigabitEthernet0/2 from LOADING to FULL, Loading Done

This message tells us that we successfully exchanged routers with our OSPF neighbor. We can verify that we are see routes from OSPF by running:

Edge-HQ#sh ip route ospf 
O E2 192.168.0.0 [110/20] via 192.168.254.2, 00:07:11, GigabitEthernet0/2

Now we are exchanging information about our networks on different sites, but we still can't failover to other ISP. We need to tweak our NAT and allow OSFP to advertise default routes to neighbors.

My NAT is configured with following commands:

###Edge-HQ

interface GigabitEthernet0/0
 ip address 1.1.1.2 255.255.255.252
 ip nat outside
 
interface GigabitEthernet0/1
 ip address 10.0.0.1 255.255.0.0
 ip nat inside
 
interface GigabitEthernet0/2
 ip address 192.168.254.1 255.255.255.252
 ip nat inside
 
ip nat inside source list 7 interface GigabitEthernet0/0 overload
access-list 7 permit 10.0.0.0 0.0.255.255

### "7" Here is a number of my access-list, you might want to change it


###Edge-Branch

interface GigabitEthernet0/0
 ip address 2.2.2.2 255.255.255.252
 ip nat outside

interface GigabitEthernet0/1
 ip address 192.168.0.1 255.255.255.0
 ip nat inside

interface GigabitEthernet0/2
 ip address 192.168.254.2 255.255.255.252
 ip nat inside

ip nat inside source list 7 interface GigabitEthernet0/0 overload
access-list 7 permit 192.168.0.0 0.0.0.255

And to allow NAT of other site's network, I'm going to add site network to access-list (ACL):

###Edge-HQ
conf t
access-list 7 permit 192.168.0.0 0.0.0.255
end
wr me

#Edge-Branch
conf t
access-list 7 permit 10.0.0.0 0.0.255.255
end
wr me

And finally adding advertisement for default routes via OSPF:

###Apply this code to every "Edge" router
conf t
router ospf 1111
default-information originate
end
wr me

Testing failover

Now when all the config is done, when I shut one of my uplinks I see following:

Edge-Branch#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
Edge-Branch(config)#int gig
Edge-Branch(config)#int gigabitEthernet 0/0
Edge-Branch(config-if)#shut
Edge-Branch(config-if)#
%LINK-5-CHANGED: Interface GigabitEthernet0/0, changed state to administratively down
%LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/0, changed state to down
Edge-Branch#end
%SYS-5-CONFIG_I: Configured from console by console


Edge-Branch#sh ip route 
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2, E - EGP
       i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area
       * - candidate default, U - per-user static route, o - ODR
       P - periodic downloaded static route

Gateway of last resort is 192.168.254.1 to network 0.0.0.0

     192.168.0.0/24 is variably subnetted, 2 subnets, 2 masks
C       192.168.0.0/24 is directly connected, GigabitEthernet0/1
L       192.168.0.1/32 is directly connected, GigabitEthernet0/1
     192.168.254.0/24 is variably subnetted, 2 subnets, 2 masks
C       192.168.254.0/30 is directly connected, GigabitEthernet0/2
L       192.168.254.2/32 is directly connected, GigabitEthernet0/2
O*E2 0.0.0.0/0 [110/1] via 192.168.254.1, 00:08:38, GigabitEthernet0/2

Default router for "Edge-Branch" changed to "Edge-HQ". Branch router recognized that despite of not having uplink to ISP, it can reach internet via other edge router.

Another important note is that when packet is being received my ISP, it would come from public IP of site that sent said packet. In our case packet would have public IP of "Edge-HQ" router.

Another benefit of PTP link is that we can access resources in different site without S2S/L2L VPN tunnels. Here is me pinging "Client-HQ" from "Client-Branch".

C:\>ping 192.168.0.2

Pinging 192.168.0.2 with 32 bytes of data:

Reply from 192.168.0.2: bytes=32 time=7ms TTL=128
Reply from 192.168.0.2: bytes=32 time=3ms TTL=128
Reply from 192.168.0.2: bytes=32 time=3ms TTL=128
Reply from 192.168.0.2: bytes=32 time=3ms TTL=128

Mitigating single point of failure for PTP link

With current setup if our PTP link goes down we would loose access to recourses on other site.

There are a number of ways we can mitigate that:

1) Add another physical link and bond them in Etherchannel.

That would allow us to mitigate single link failure. Has a drawback of extra cost if we lease this link from provider, but no extra overhead.

2)Setup a GRE tunnel between edge routers across public internet.

While this would work and allow access to recourses, it does not provide encryption. And you would want encryption for your internal traffic. Slight overhead, might need to adjust MTU to 1476.

3)Setup a S2S/L2L VPN tunnel between edge routers across public internet.

This would work like a GRE tunnel with added benefit of encryption. Many decent enterprise or prosumer routers/firewalls have this option. Even more overhead, not only we need to receive traffic, we also need to decrypt it.

VPN tunnels and routing

When adding VPN tunnels to mitigate failure of PTP link, it is important to add choose your routing accordingly. Now when you have two paths to choose from you need to determine primary. In my example choosing PTP for primary path is a natural choice, since it doesn't add overhead of encryption and requires less hops.

Here is when Administrative Distance (AD) comes in. Every route in router has AD, exact values are vendor dependent. When multiple routes exists for same network, route with lower AD is preferred.

Since a link with OSPF is my preferred path I need to make sure when adding static route for my tunnel I'm going to use AD higher than 110.

conf t
ip route 192.168.0.0 255.255.255.0 *tunnel IP* 250
end
wr me

With that config when our PTP link goes down and routes are no longer learned via OSPF, we can reach "Edge-Branch" internal network via our tunnel. I would dive deeper into S2S/L2L VPN tunnels in my future guides.