Sometimes a customer situation finds aligns with perfect symmetry with my studies.
A customer of mine is using one ISP, which happens to be a T1 connection. Really cool if you live in Lickbucket, AK or insist that 1998 was the coolest year ever. Fortunately for me and my sanity, my customer does not fit into either category. The T1 connection is used for both Internet access and site-to-site VPN connections to a number of satellite offices. The VPNs are utilized for direct dialing between IP phones. Not only is this solution undersized for their needs, but recently the T1 connection failed, causing the whole main office to go down.
Right now, the customer is using a pair of ASAs at the main site. While the ASA doesn’t offer a BGP solution like Juniper (grrr…), the ASA does offer redundant links out to the Internet. However, unlike BGP, the IP address schemes for each ISP will be disparate. This is no big deal if you have only outbound connections, but what about VPNs? If we go with an additional ISP, we then need to split the traffic so we can utilize both links, but we still need a failover solution in case either link dies.
One feature of the ASA is the ability to put multiple peers on the firewall. With this option, the VPN connections at each of the satellite offices can point to both external IP addresses for the redundant ISP links, providing for failover for the VPN as well.
In order to test out the theories, I setup the following lab:
With this lab, a local ASA simulates two ISP connections to R2 and R3. R4 is the ISP connection to the remote location, and the remote ASA connects to R4. R1 and R5 loopbacks are going to simulate end hosts. R4 has three loopbacks that is going to simulate Internet connections. R2, R3, and R4 are running EIGRP, and R3 is using an offset list to prevent asymmetric routing. BGP could have been used to create a more “realistic” Internet connection, but…well, screw that. EIGRP is easy, and the routing protocol isn’t the point of the lab.
After setting up EIGRP, getting the routes setup correctly, and opening up the firewalls to allow ICMP through, the meat of the lab is now at hand. The failover between ISPs is done by SLA monitoring. The SLA monitor sends a ping to a downstream router of choice, and if the router is responding, then the routes associated with the SLA monitor stay in place. However, if the downstream router does not ping, then a second static route that is given a high metric (in this example, 250), takes over in the routing table. Here is what the configuration in the lab would look like for the default static routes:
route outside 0.0.0.0 0.0.0.0 192.168.1.2 1 track 1
route backup 0.0.0.0 0.0.0.0 192.168.2.2 254
!
sla monitor 123
type echo protocol ipIcmpEcho 192.168.1.2 interface outside
num-packets 3
frequency 10
sla monitor schedule 123 life forever start-time now
!
track 1 rtr 123 reachability
Two default routes exist. The first default route points to the first “ISP” and has a metric of 1, and there is a track of 1 placed after it. The track 1 matches the Response Time Reporter (rtr) that is the last statement of the SLA monitoring configuration. The second default route points to the second “ISP” and has a metric of 254.
SLA monitor is configured with the interface and the downstream router (or host) that will receive the ping. In this example, we are pinging R2 from the outside interface on the ASA to verify connectivity. In the real world, this is going to be one of the better options, as many ISPs don’t really like having intermediate routers or hosts pinged through their systems. The SLA monitor schedule line starts the monitoring process. The final line is the Response Time Reporter associated with tracking the ping and linking the ping to the primary route using the track command.
After configuration, the routing table looks like the following:
ASA# sh route
Codes: C - connected, S - static, I - IGRP, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2, E - EGP
i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area
* - candidate default, U - per-user static route, o - ODR
P - periodic downloaded static route
Gateway of last resort is 192.168.1.2 to network 0.0.0.0
S 1.1.1.1 255.255.255.255 [1/0] via 10.10.10.2, inside
C 10.10.10.0 255.255.255.0 is directly connected, inside
C 192.168.1.0 255.255.255.0 is directly connected, outside
C 192.168.2.0 255.255.255.0 is directly connected, backup
S* 0.0.0.0 0.0.0.0 [1/0] via 192.168.1.2, outside
The default route is currently going through the outside interface. However, if the link to 192.168.1.2 is severed by doing a no shut on the fa0/0 interface on R2, the following result is:
ASA# sh route
Codes: C - connected, S - static, I - IGRP, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2, E - EGP
i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area
* - candidate default, U - per-user static route, o - ODR
P - periodic downloaded static route
Gateway of last resort is 192.168.2.2 to network 0.0.0.0
S 1.1.1.1 255.255.255.255 [1/0] via 10.10.10.2, inside
C 10.10.10.0 255.255.255.0 is directly connected, inside
C 192.168.1.0 255.255.255.0 is directly connected, outside
C 192.168.2.0 255.255.255.0 is directly connected, backup
S* 0.0.0.0 0.0.0.0 [254/0] via 192.168.2.2, backup
The default route has now changed to the backup interface, and the next hop is going to ISP2. Notice the metric: it is now at 254 rather than 1, as it is using the floating static route that was created. After doing a no shut on fa0/0 on R2, the routes go back to normal.
One down, one to go…
The failover on the ASA is working, but what about the VPNs? There are now two different ISPs providing two different IP addresses on the outside. So the ASA at the remote site needs to connect to the backup interface on the local ASA for the primary VPN, but when the primary connection fails, it will need to connect to the outside interface of the local ASA. One way to accomplish this is to create two peers on the ASA on a single crypto map. However, not only do two peers need to be added, but two different tunnel groups (tunnel groups require the IP address of the endpoint for the name on S2S VPN connections).
The local ASA looks like the following:
access-list VPN extended permit ip host 1.1.1.1 host 6.6.6.6
!
crypto ipsec transform-set STRONG esp-3des esp-sha-hmac
crypto map VPN 10 match address VPN
crypto map VPN 10 set peer 10.10.40.2
crypto map VPN 10 set transform-set STRONG
crypto map VPN interface outside
crypto map VPN interface backup
crypto isakmp enable outside
crypto isakmp enable backup
crypto isakmp policy 10
authentication pre-share
encryption 3des
hash sha
group 2
lifetime 86400
!
tunnel-group 10.10.40.2 type ipsec-l2l
tunnel-group 10.10.40.2 ipsec-attributes
pre-shared-key cisco
This is a pretty standard crypto map solution. The access list at the top is creating a tunnel from l0 on R1 to l0 on R6. Everything else will go out the “Internet”. The next hurdle is the routing. Redundancy would require that if ISP2 went down, then the VPN would go through ISP1. We can accomplish that by doing the same thing that we did for the outside link, only we create another SLA monitor for the backup link and track that connection:
route backup 1.1.1.1 255.255.255.255 192.168.2.2 1 track 2
route backup 10.10.40.2 255.255.255.255 192.168.2.2 1 track 2
route outside 1.1.1.1 255.255.255.255 192.168.1.2 254
route outside 10.10.40.2 255.255.255.255 192.168.1.2 254
!
sla monitor 456
type echo protocol ipIcmpEcho 192.168.2.2 interface backup
num-packets 3
frequency 10
sla monitor schedule 456 life forever start-time now
!
track 2 rtr 456 reachability
Two routes have been added that will go to the backup interface for the VPN. The first route is the endpoint that will go through the VPN tunnel. The second IP is the actual VPN termination point on the firewall. By creating the routes in this manner, the VPN will be forced through the backup ISP unless it goes down. If the backup ISP goes down, the VPN will go through the outside ISP until it comes back up. This allows both connections to still be used rather than having one connection sitting idle yet costing the company money.
Routing is complete, so now the remote ASA needs to be configured to use the backup ISP as the primary and the outside ISP as the secondary for the VPN.
access-list VPN extended permit ip host 6.6.6.6 host 1.1.1.1
!
crypto ipsec transform-set STRONG esp-3des esp-sha-hmac
crypto map VPN 10 match address VPN
crypto map VPN 10 set peer 192.168.2.1 192.168.1.1
crypto map VPN 10 set transform-set STRONG
crypto map VPN interface outside
crypto isakmp enable outside
crypto isakmp policy 10
authentication pre-share
encryption 3des
hash sha
group 2
lifetime 86400
!
tunnel-group 192.168.2.1 type ipsec-l2l
tunnel-group 192.168.2.1 ipsec-attributes
pre-shared-key cisco
tunnel-group 192.168.1.1 type ipsec=l2l
tunnel-group 192.168.1.1 ipsec-attributes
pre-shared-key cisco
The configuration is pretty much the same, but there are some distinct differences. First, the access-list for the VPN is reversed to match the outgoing traffic as with any VPN connection between two IPSec devices. Second, the set peer statement has two IP addresses: The primary address followed by the backup address is the primary is unreachable. Finally, there are two tunnel groups created. This needs to be done because the tunnel group must match the IP address. Because we have two potential tunnels, we need two tunnel groups.
So with everything in place, testing can now begin. The following scenarios should be tested:
- Current state: When we ping 6.6.6.6 from 1.1.1.1 on R1, the ping should be successful, and phase one should complete between 192.168.2.1 and 10.10.40.2. This can be confirmed with a sh crypto isakmp on both ASAs. A ping to 2.2.2.2, 3.3.3.3, or 4.4.4.4 on R4 from R1 (either 1.1.1.1 or 10.10.10.2) should also complete. A traceroute to those IP address should show the route going through R2.
- R2 goes down: Ping from 1.1.1.1 to 6.6.6.6 is successful. Ping to R4 on any interface is successful, but the traceroute should show that the path now goes through R3. The crypto maps on both ASAs remain the same as in scenario 1.
- R3 goes down: Ping from 1.1.1.1 to 6.6.6.6 is successful. P Ping to R4 on ay interface is successful, and the traceroute should go through R2 as in scenario 1. However, the crypto maps on the ASAs should now show a connection between 10.10.40.2 and 192.168.1.1. This can be confirmed with a sh crypto isakmp on both ASAs.
If these scenarios are successful, then the lab setup is confirmed. First, scenario 1. The routing table on the ASA:
ASA# sh route
Codes: C - connected, S - static, I - IGRP, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2, E - EGP
i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area
* - candidate default, U - per-user static route, o - ODR
P - periodic downloaded static route
Gateway of last resort is 192.168.1.2 to network 0.0.0.0
S 1.1.1.1 255.255.255.255 [1/0] via 10.10.10.2, inside
S 6.6.6.6 255.255.255.255 [1/0] via 192.168.2.2, backup
C 10.10.10.0 255.255.255.0 is directly connected, inside
S 10.10.40.2 255.255.255.255 [1/0] via 192.168.2.2, backup
C 192.168.1.0 255.255.255.0 is directly connected, outside
C 192.168.2.0 255.255.255.0 is directly connected, backup
S* 0.0.0.0 0.0.0.0 [1/0] via 192.168.1.2, outside
So far, so good. A traceroute to an “Internet” address (one of the loopbacks):
R1#traceroute 5.5.5.5
Type escape sequence to abort.
Tracing the route to 5.5.5.5
1 192.168.1.2 28 msec 44 msec 16 msec
2 10.10.20.1 8 msec * 28 msec
Good! The traceroute is going through R2. Is the VPN going through R3? It should be terminating on the 192.168.2.1 address:
ASA2# sh crypto isakmp sa
Active SA: 1
Rekey SA: 0 (A tunnel will report 1 Active and 1 Rekey SA during rekey)
Total IKE SA: 1
1 IKE Peer: 192.168.2.1
Type : L2L Role : responder
Rekey : no State : MM_ACTIVE
After pinging 6.6.6.6 from R1 with a 1.1.1.1 source, we see on ASA2 that the active peer is 192.168.2.1, so we are going through R3 and terminating on the backup interface of ASA1. So for scenario 2, we shut the fa0/0 interface on R2 to verify that all traffic will go through R3. After shutting the interface down, we look at the routing table on ASA1:
ASA# sh route
Codes: C - connected, S - static, I - IGRP, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2, E - EGP
i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area
* - candidate default, U - per-user static route, o - ODR
P - periodic downloaded static route
Gateway of last resort is 192.168.2.2 to network 0.0.0.0
S 1.1.1.1 255.255.255.255 [1/0] via 10.10.10.2, inside
S 6.6.6.6 255.255.255.255 [1/0] via 192.168.2.2, backup
C 10.10.10.0 255.255.255.0 is directly connected, inside
S 10.10.40.2 255.255.255.255 [1/0] via 192.168.2.2, backup
C 192.168.1.0 255.255.255.0 is directly connected, outside
C 192.168.2.0 255.255.255.0 is directly connected, backup
S* 0.0.0.0 0.0.0.0 [254/0] via 192.168.2.2, backup
The default gateway has now changed to the backup interface, and the backup interface is being used for the VPN. A traceroute to the “Internet”:
R1#traceroute 5.5.5.5
Type escape sequence to abort.
Tracing the route to 5.5.5.5
1 192.168.2.2 20 msec 44 msec 8 msec
2 10.10.30.1 8 msec * 28 msec
The traceroute is now going through R3 instead of R2. The VPN:
ASA2# sh crypto isakmp sa
Active SA: 1
Rekey SA: 0 (A tunnel will report 1 Active and 1 Rekey SA during rekey)
Total IKE SA: 1
1 IKE Peer: 192.168.2.1
Type : L2L Role : responder
Rekey : no State : MM_ACTIVE
The backup interface on the ASA is still terminating the VPN. Scenario 3 is up next:
ASA# sh route
Codes: C - connected, S - static, I - IGRP, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2, E - EGP
i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area
* - candidate default, U - per-user static route, o - ODR
P - periodic downloaded static route
Gateway of last resort is 192.168.1.2 to network 0.0.0.0
S 1.1.1.1 255.255.255.255 [1/0] via 10.10.10.2, inside
S 6.6.6.6 255.255.255.255 [250/0] via 192.168.1.2, outside
C 10.10.10.0 255.255.255.0 is directly connected, inside
S 10.10.40.2 255.255.255.255 [250/0] via 192.168.1.2, outside
C 192.168.1.0 255.255.255.0 is directly connected, outside
C 192.168.2.0 255.255.255.0 is directly connected, backup
S* 0.0.0.0 0.0.0.0 [1/0] via 192.168.1.2, outside
All of the routes are now going through the outside interface. Traceroutes should now go through R2:
R1#traceroute 5.5.5.5
Type escape sequence to abort.
Tracing the route to 5.5.5.5
1 192.168.1.2 24 msec 16 msec 12 msec
2 10.10.20.1 40 msec * 28 msec
All traffic is going through R2 for the Internet. The VPN:
ASA2# sh crypto isakmp sa
Active SA: 1
Rekey SA: 0 (A tunnel will report 1 Active and 1 Rekey SA during rekey)
Total IKE SA: 1
1 IKE Peer: 192.168.1.1
Type : L2L Role : responder
Rekey : no State : MM_ACTIVE
Yep, the ASA is terminating on the outside interface rather than the backup. All of our testing is complete.
