Sample Configuration for BGP with Two Different Service Providers (Multihoming)

http://www.cisco.com/en/US/customer/tech/tk365/technologies_configuration_example09186a008009456d.shtml#defaultroutes

Document ID: 23675

PDF Downloads Sample Configuration for BGP with Two Different Service Providers (Multihoming)

Share on printShare on emailShare on favoritesShare on googleShare on twitterShare on facebook

Related Documents • Sample Configuration for BGP with Two Different Service Providers (Multihoming) [IP Routing] • Sample IPv6 Configuration for BGP with Two Different Service Providers (Multihoming) [IP Routed Protocols] • Load Sharing with BGP in Single and Multihomed Environments: Sample Configurations [IP Routing] • How to Use HSRP to Provide Redundancy in a Multihomed BGP Network • How BGP Routers Use the Multi-Exit Discriminator for Best Path Selection [IP Routing] More... Related Products/Technology • Border Gateway Protocol (BGP) Related Discussion • Multihoming & NAT • BGP Multihoming • BGP multihomed - 2 routers, 2 ISP's... • bgp sample config • ASA, Dual ISP problem

Contents Introduction Prerequisites       Requirements       Components Used       Conventions Background Information Configure       Network Diagram       Configuration to Receive Full Internet Routing Table       Configuration to Receive Directly-Connected Routes       Configuration to Receive Default Routes Only Verify Troubleshoot Cisco Support Community - Featured Conversations Related Information

Introduction

Border Gateway Protocol (BGP) is one of the key protocols to use to achieve Internet connection redundancy. When you connect your network to two different Internet service providers (ISPs), it is called multihoming. Multihoming provides redundancy and network optimization. It selects the ISP which offers the best path to a resource. When you are running BGP with more than one service provider, you run the risk that your autonomous system (AS) will become a transit AS. This causes Internet traffic to pass through your AS and potentially consume all of the bandwidth and resources on the CPU of your router. This document addresses this issue, with appropriate configuration examples.

Prerequisites

Requirements

Ensure that you have knowledge of these topics before you attempt this configuration:

• The Cisco BGP Implementation
• Basic BGP Configuration Task List
• BGP Case Studies

Components Used

The information in this document is based on Cisco 2500 Series Routers that run Cisco IOS^® Software Release 12.2(27).

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Conventions

Refer to Cisco Technical Tips Conventions for more information on document conventions.

Background Information

If you want to receive the full Internet routing table, use the Configuration to Receive Full Internet Routing Table on your local router (Router A in the examples in this document).

If you want to receive routes that are directly connected to your service providers, but use default routes to the rest of the Internet, try theConfiguration to Receive Directly-Connected Routes.

If you want to receive only default routes from the directly connected service providers, use the Configuration to Receive Default Routes Only.

For more information about the regular expressions used in the configurations in this document, refer to Using Regular Expressions in BGP.

Configure

In this section, you are presented with the information to configure the features described in this document.

Note: Use the Command Lookup Tool (registered customers only) to find more information on the commands used in this document.

Network Diagram

This document uses this network setup:

In that network diagram, 1.0.0.0/8 and 2.0.0.0/8 are advertised by AS 300 to the outside.

Configuration to Receive Full Internet Routing Table

The following configuration allows Router A to peer with BGP speakers in other autonomous systems. The route-map localonlyallows only the locally generated routes to be advertised to both of the service providers. In other words, they filter the Internet routes from one service provider that go back to the other service provider. This prevents the risk that your autonomous system will become a transit AS for Internet traffic.

Note: Cisco recommends 1 GB of memory for the entire Internet routing table from a single peer.

Router A
Current configuration: router bgp 300 network 1.0.0.0 network 2.0.0.0 neighbor 10.10.10.10 remote-as 100 neighbor 10.10.10.10 route-map localonly out !--- Outgoing policy route-map that filters routes to service provider A (SP-A). neighbor 20.20.20.20 remote-as 200 neighbor 20.20.20.20 route-map localonly out !--- Outgoing policy route-map that filters routes to service provider B (SP-B). end

This AS-Path access list only permits locally originated BGP routes:

ip as-path access-list 10 permit ^$

This is an example of a route map that uses that AS-Path access list to filter the routes advertised to the external neighbors in the service provider networks:

route-map localonly permit 10
 match as-path 10

Configuration to Receive Directly-Connected Routes

Router A

Current configuration: router bgp 300 network 1.0.0.0 network 2.0.0.0 neighbor 10.10.10.10 remote-as 100 neighbor 10.10.10.10 route-map localonly out !--- Outgoing policy route-map that filters routes to SP-A. neighbor 10.10.10.10 route-map as100only in !--- Incoming policy route-map that filters routes from SP-A. neighbor 20.20.20.20 remote-as 200 neighbor 20.20.20.20 route-map localonly out !--- Outgoing policy route-map that filters routes to SP-B. neighbor 20.20.20.20 route-map as200only in !--- Incoming policy route-map that filters routes from SP-B. end

Because you only want to accept routes that are directly connected to the service providers, you must filter the routes that they send to you, as well as the routes that you advertise. This access list and route map permit only locally originated routes; use it to filter outbound routing updates:

ip as-path access-list 10 permit ^$

!--- Configuring the as-path access-list 10 as 
!--- IP as-path access-list 10 permit ^$, means that the router 
!--- will only send the prefixes which are generated locally in 
!--- AS 300 and not from any other AS.

   route-map localonly permit 10
   match as-path 10

This access list and route map filter out anything that is not sourced within the first service provider network; use it to filter the routes that are learned from service provider A (SP-A).

ip as-path access-list 20 permit ^100$

route-map as100only permit 10
 match as-path 20

This access list and route map filter out anything that is not sourced within the second service provider network; use it to filter the routes that are learned from service provider B (SP-B).

ip as-path access-list 30 permit ^200$

route-map as200only permit 10
  match as-path 30

You also need two default routes that are distributed back into the rest of your network, one pointed to each of the service provider entry points:

ip route 0.0.0.0 0.0.0.0 10.10.10.10
ip route 0.0.0.0 0.0.0.0 20.20.20.20

Configuration to Receive Default Routes Only

Router A

Current configuration: router bgp 300 network 1.0.0.0 network 2.0.0.0 neighbor 10.10.10.10 remote-as 100 neighbor 10.10.10.10 route-map localonly out !--- Outgoing policy route-map that filters routes to SP-A. neighbor 10.10.10.10 prefix-list ABC in neighbor 20.20.20.20 remote-as 200 neighbor 20.20.20.20 route-map localonly out !--- Outgoing policy route-map that filters routes to SP-B. neighbor 20.20.20.20 prefix-list ABC in ip prefix-list ABC seq 5 permit 0.0.0.0/0 !--- Prefix list to allow only default route updates. end

Because you want Router A to receive only default routes and no other networks from SP-A and SP-B, you must permit only the default route and deny all other BGP updates. Use this prefix list to allow only the default route update 0.0.0.0/0 and to deny all other BGP updates on Router A:

ip prefix-list ABC seq 5 permit 0.0.0.0/0

Apply that prefix list on the inbound updates on individual BGP neighbors in this way:

neighbor 10.10.10.10 prefix-list ABC in
neighbor 20.20.20.20 prefix-list ABC in

For more information on how to configure a prefix list, refer to the Configuring BGP Filtering Using Prefix Lists section of Configuring BGP

For explanations of what each command does, refer to Configuring BGP and BGP Commands.

Verify

Use this section to confirm that your configuration works properly.

The Output Interpreter Tool (registered customers only) (OIT) supports certain show commands. Use the OIT to view an analysis ofshow command output.

You can use the show ip route and show ip bgp commands to check the IP routing table and BGP routing table entries.

Troubleshoot

There is currently no specific troubleshooting information available for this configuration.

Cisco Support Community - Featured Conversations

Cisco Support Community is a forum for you to ask and answer questions, share suggestions, and collaborate with your peers. Below are just some of the most recent and relevant conversations happening right now.

Best Practices for Using Multiple Network Interfaces (NICs) with NI Products

Overview

It is becoming commonplace to have more than one Ethernet adapter in a PC or embedded controller, especially in systems where wired and wireless adapters are both present. However, the presence of more than one Network Interface Card (NIC) can cause connectivity problems if each network interface is not carefully configured to account for the overall network layout. This tutorial outlines basic Ethernet networking concepts, explains why incorporating multiple NICs can cause problems, and provides best practices to follow when working with a multi-NIC system. In addition, several common scenarios are explored, including an NI Real-Time Hypervisor system that features a complex network configuration.

Table of Contents

1. Background: Ethernet Networking Concepts
2. Guidelines for Configuring Multi-NIC Systems
3. Example Scenarios
4. Additional Resources

Background: Ethernet Networking Concepts

Before explaining best practices for configuring controllers with multiple NICs, it is necessary to understand some basic concepts related to Ethernet networking. This section explains how individual NICs are addressed, how packets are routed on Ethernet networks, and what a typical Ethernet packet transfer looks like. 

If you are already familiar with these concepts or are not concerned about the "why" behind the recommendations in this paper, then you can skip to the Guidelines for Configuring Multi-NIC Systems section.

Data Link Layer: MAC Addressing

Just like a home or business address must be specified in order to send a piece of postal mail, every port on every Ethernet adapter is assigned a unique address called a Medium Access Control (MAC) address by the manufacturer. MAC addresses consist of 48 bits, and are commonly expressed in groups of 2 hexadecimal digits (e.g. 01:23:45:67:89:ab). The MAC address or addresses for a specific Ethernet adapter are commonly printed on a label on the device itself, and are also accessible via operating system networking settings. 

When Ethernet data is sent between two computers, it is sent in small pieces called "packets", and those packets contain the MAC address of the receiving interface. Each interface is continually listening for packets sent to its unique address.

[+] Enlarge Image

Figure 1. Each network interface controller (NIC) in a computer has a unique medium access control (MAC) address. If multiple NICs are installed in one computer, they each have their own MAC addresses.

Internet Layer Addressing

If every computer in the world were connected to one large wire and knew every other computer's MAC address, then no further addressing schemes would be required. However, this is impractical for several reasons. First, with such a large shared connection packet collisions would occur frequently, causing communication to be unreliable. Second, storing the address of every other computer on the planet on each machine would require a large amount of memory, and is not a good strategy due to the constant addition of new devices.

IP Addresses (IPv4)

Clearly, a different scheme of addressing is needed to enable computers to communicate with each other via Ethernet. In practice, users can assign one or more Internet Protocol (IP) addresses to each MAC address, and set these IP addresses up in such a way that most network traffic occurs in small "pools" rather than across a wider group of machines. In practice, IP addresses are typically specified by a 32-bit value, and are commonly written as four groups of 3-digit decimal values (e.g. 192.168.100.100). Consider a network with 10 computers configured such that the computers have IP addresses ranging from 192.168.100.0 to 192.168.100.9. If all machines follow a rule where all addresses with format 192.168.100.xxx are assumed to be inside the internal network, then it is easy to tell whether communicating with a given IP address requires accessing some other network or not.

Sample rule: all interfaces with IP addresses in the range 192.168.100.0 to 192.168.100.255 are inside of the local network, and interfaces with other addresses are outside.

IP Address	Inside or Outside Local Network? (following the sample rule)
192.168.100.6	Inside
192.168.100.104	Inside
192.169.100.100	Outside
192.169.101.255	Outside

Table 1. Since IP addresses can be assigned to individual NICs by the user, it is possible to assign similar IP addresses to a group of interfaces and then easily determine whether a given IP address is inside or outside of the local network.

This simple idea has significant benefits. Whereas MAC addresses do not typically follow a pattern in a given set of machines (they are randomly distributed by vendor), since IP addresses are user-configurable they can be set up to follow a logical, hierarchical pattern. Using this pattern, small groups of computers can communicate with each other with only local traffic, therefore reducing data collisions in the larger network. In addition, since there are more than 2^32 connected Ethernet devices on the planet, IP addresses can be reused across multiple smaller networks while those machines are still accessible via a process called Network Address Translation (NAT). A detailed discussion of NAT is beyond the scope of this tutorial.

When configuring IP addresses, you are likely to run across addresses of the form 10.x.x.x, 192.168.x.x, or 172.x.x.x. Each of these ranges have been specially designated for private network use, so you can be sure that no other public server (e.g. google.com) will use the same IP address if you use one of these.

It should be noted that the information in this section pertains to the most commonly used version of the Internet Protocol called IPv4. The next version of the Internet Protocol, IPv6, uses 128-bit addresses but is not widely used today (although adoption is expected to grow in the next several years).

Subnet Masks

So, when looking at an IP address to send data to, how does a given computer decide if it is inside of the local network, or whether it will rely accessing an outside network? In other words, if we want computers to follow a rule as was the case with the example in the previous section, how is that rule specified?

The key component here is the subnet mask, a group of 32 bits expressed in 4 groups of numbers just like the IP address (e.g. 255.255.255.192). A binary 1 value in one of the 32 bit positions means that if the IP address bit is different in this position between the sender and receiver, then Ethernet packets must be routed to an outside network. On the contrary, a binary zero in a given position means that if the IP address bit is different then it does not matter; if all other bits are the same between the sender and receiver's IP address then the receiver is assumed to be on the same local network.

Figure 2. Subnet masks provide a way to specify the "rule" for which IP addresses are included in a local network and which fall outside of the local network.

Gateways and Default Gateways

How are Ethernet packets passed between local and outside networks? This task is typically performed by a device called a router, that is also often referred to as a network gateway (in this tutorial the two terms will be used interchangeably). For each Ethernet adapter connected to a computer, the user often specifies an IP address, subnet mask, and default gateway. The default gateway is the address of the router that should be used when packets must be sent outside of the local network.

ARP Requests

At this point, note that at the lowest level, every single Ethernet packet that is sent must be sent to a MAC address. IP addresses provide some advantages including the use of subnets, but keep in mind that simply knowing the IP address of a remote computer's interface is not enough information to ensure that a packet reaches that computer. For example, to communicate between two computers on remote networks it may be necessary to pass through several routers. At a low-level, one packet must be sent between the sending computer and the first router, then another packet from the first router to the next, and so forth until the final packet finally reaches the destination computer. For the packet to be sent between each link, knowing the MAC address of the receiving router or computer's interface is critical.

Imagine starting with an IP address of a destination computer on the local network: how does the originating computer obtain the correct MAC address to send packets to? The Address Resolution Protocol (ARP) is used for this. Continuing with the example, the sending computer will send a special type of broadcast message called an ARP request on the local network (requesting the MAC address for a certain IP address). All Ethernet devices on the network are continually listening for these special requests, and will respond back to the sender with their MAC address if they match the IP address specified. Once the MAC address is received by the sending computer, that computer will usually store the MAC address / IP address pair in a cache table for later usage. This prevents the continual need to send ARP requests for the same destination interface.

Figure 3. For a computer to send data to a remote Ethernet interface, it must know the MAC address of that interface. In the case that the MAC address is not known, a computer can send an Address Resolution Protocol (ARP) request to a specified IP address, and the destination machine will reply back with its MAC address.

If a router on the network knows that the IP requested is external to the local network, then it can also respond to the ARP request with its own MAC address. This is called "proxy ARP".

Summary: Anatomy of an Ethernet Packet Transfer

Let us walk through the (simplified) process of sending a single Ethernet packet from a source PC to a destination PC. We will assume that the Source PC contains multiple Ethernet interfaces.

1. An Ethernet packet is ready to be sent by the source machine (this could be associated with a specific application or OS process). The packet contains information about the IP address of the receiver (this may have been obtained from a Domain Name Service (DNS) server).
2. If there is already a cached ARP entry containing the MAC address corresponding to the receiver's IP address, then proceed to step 4.
3. The Sender looks at the IP address of the packet, and the subnet mask of its Ethernet interfaces.
   1. If the IP address falls within the subnet of a specific interface, then an ARP request is sent from that interface.
   2. If the IP address falls outside of the subnet of any interfaces, then the sender looks at the default gateway entry for each interface.
      1. If there is only one default gateway entry for all interfaces, then use its MAC address as contained in the ARP table or send a new ARP request to find the MAC address of the gateway.
      2. If there are multiple default gateway entries, then use the one with the lowest metric (determined by several factors depending on the OS used). Find its MAC address in the ARP table, or send a new ARP request to find the MAC address of the gateway.
      3. If there are no default gateway entries, send an ARP request on one or more interfaces to see if a gateway responds with its own MAC (proxy ARP).
4. Send the Ethernet packet using the MAC address from the ARP cache or ARP request. If the ARP request does not produce a result, then the packet is not deliverable. If the receiver is on a remote subnet, then the packet is sent to a router and it is the router's job to send the packet over the next link so that it ultimately reaches the receiver.

Guidelines for Configuring Multi-NIC Systems

When working with a computer that features multiple network interfaces, you must pay closer attention to networking settings to avoid difficult-to-debug connectivity problems. Follow these guidelines to ensure the best operation of your multi-NIC system. This includes any PC or embedded controller running a desktop OS and LabVIEW, as well as any LabVIEW Real-Time target with multiple Ethernet interfaces.

Rule 1: Be Careful About Automatic IP Assignment (via DHCP or link-local addressing)

Most OSs are configured by default to obtain TCP/IP settings (IP address, subnet mask, and default gateway) automatically using a Dynamic Host Configuration Protocol (DHCP) server. If no DHCP server is found, then it is common practice for OSs to assign an IP address in the 169.254.x.x range,  which is referred to as a link-local IP address.

Make sure that you are familiar with the DHCP server(s) on your network, and what IP address ranges and subnets they make use of. Likewise, make sure that you know if any adapters in your computer use DHCP and no DHCP server is on the network, resulting in the use of a 169.254.x.x address. If you are not familiar with the automatic IP assignment details for your network, then there is a much higher chance that you will break Rule 2 or Rule 3 below.

If possible given your network configuration, it is best to manually specify IP settings for each NIC in your multi-NIC PC. However, if you are on a network with other DHCP clients, using DHCP may be unavoidable.

Rule 2: Avoid Assigning Multiple NICs in the Same Computer to the Same Subnet

Using multiple NICs on the same subnet is the #1 cause of connectivity issues on multi-NIC systems. While some OSs may be able to gracefully handle the presence of multiple NICs on one subnet, others may mistakenly attempt to send packets out of the wrong interface. You can remedy this issue by statically assigning one NIC (and the other networked computers attached to that NIC) to the 192.168.x.x range with subnet mask 255.255.0.0, and another NIC on the 10.0.x.x range with subnet mask 255.255.0.0. Note that this is just one possible configuration, any combination of two or more different subnets will work. It is recommended that you use one of the designated private IP address ranges to avoid conflicts with public servers on the internet.

In practice, there is rarely a good reason to use multiple NICs on the same subnet. One advanced configuration that warrants this setup is using multiple adapters to increase bandwidth is called adapter teaming, however this configuration is beyond the scope of this paper.

Figure 4. When configuring a controller with multiple NICs, each NIC should communicate with a different subnet. Configuring two or more NICs on the same subnet may cause communication problems.

One of the most common scenarios resulting in multiple NICs being assigned to the same subnet is when both a wired and wireless interface are used to connect to the same network. Once again, while some OSs may handle this case gracefully, it is a best practice to either configure the wired and wireless networks to exist on different subnets or to disable one network interface when using the other.

Rule 3: Avoid Specifying a Default Gateway for More Than One NIC

When a destination IP address does not fall within the subnet of any NIC, then a default gateway is used to relay packets to the outside network. If multiple default gateways exist, then packets may be routed to the wrong outside network, causing them to be undeliverable. 

In most cases, you should be able to address all computers attached to a network interface on the same private subnet, and leave the default gateway blank for that interface. Typically only one NIC with a gateway to the internet or corporate network should have a default gateway address specified. 

NIC 1 Connected to Corporate Network and Internet	NIC 2 Connected to Local Network
IP Address: 192.168.50.8	IP Address: 10.0.3.7
Subnet Mask: 255.255.0.0	Subnet Mask:255.255.255.0
Default Gateway:192.168.1.1	Default Gateway: 0.0.0.0 (or blank)

Table 2. When choosing TCP/IP settings for a multi-NIC system, it is a best practice to specify a default gateway for only one adapter. This is also true when one of the default gateway addresses is obtained via DHCP.

In other cases where gateways are attached to subnets that connect with different network interfaces, you can either manually specify network routes (this is an advanced practice) or leave the default gateway blank for all interfaces.

While it is more efficient to specify a default gateway on one interface (this avoids the need for an ARP on many packets transmitted to outside networks), leaving all default gateway entries blank can help avoid problems in networks with multiple gateways. However, this strategy will only work if network gateways support proxy ARP.

Example Scenarios

Case 1: Connecting to One Public and One Private Network

In this scenario, a host PC is connected to both a corporate network (via NIC #1) which allows internet access as well as a private network (via NIC #2) with several LabVIEW Real-Time targets. The corporate network is setup to assign a DHCP address in the range of 10.0.x.x with subnet mask 255.255.0.0 to NIC #1. The default gateway (router) address is 10.0.0.1.

Figure 5. In this scenario, a multi-NIC controller is connected to both a corporate network (with internet access) and a local private network.

Following the guidelines above, we can configure NIC #2, attached to the private network, with a static IP outside of the 10.0.x.x subnet range. While it might be acceptable to use an address in the 10.1.x.x range, as it is in a different subnet, it is possible that other 10.x.x.x IP addresses are used for additional corporate servers, etc. Therefore, it is safer to use the 192.168.x.x range for the private network.

Since we are working with a small number of LabVIEW Real-Time targets in the private network, we can use a subnet mask of 255.255.255.0 and assign each a static IP address in the range 192.168.0.x; this will allow for up to 255 connected interfaces in the subnet. For simplicity, the IP address of the host computer (NIC #2) will be set to 192.168.0.1.

It is important to leave the default gateway blank on the NIC #2 settings for the host machine. In this way, only the NIC #1 default gateway will be used, which is what we want in order to access the internet from the host. In addition, the default gateway can be left blank on the LabVIEW Real-Time devices, as they should only be communicating within the local subnet (there are no gateways connected).

NIC 1 Connected to Corporate Network and Internet	NIC 2 Connected to Local Private Network
IP Address: (DHCP) 10.0.x.x	IP Address: 192.168.0.1
Subnet Mask: (DHCP) 255.255.0.0	Subnet Mask: 255.255.255.0
Default Gateway: (DHCP) 10.0.0.1	Default Gateway: 0.0.0.0 (unspecified)

Table 3. This table shows one possible IP address configuration for the system shown in Figure 5 above. Notice that the two NICs connect to different subnets, and that only one Default Gateway address is specified.

Case 2: Configuring an NI Real-Time Hypervisor System with Virtual Ethernet Connection (local development)

NI Real-Time Hypervisor systems run a host OS (Windows or Linux) alongside LabVIEW Real-Time simultaneously. Each physical NIC in these systems can be assigned to either the host OS or LabVIEW Real-Time, and a virtual NIC (emulated in software) is also presented to each OS to simplify inter-OS communication.

Figure 6. In this scenario, a Real-Time Hypervisor controller is running both Windows XP and LabVIEW Real-Time in parallel. Both OSs are connected to each other through a set of virtual NICs, and each OS has a physical NIC to connect to the corporate network & internet as well.

In this scenario we will work with a Real-Time Hypervisor system that has two physical NICs, and has the internal Virtual Ethernet connection enabled. One physical interface will be assigned to Windows XP, and the other to LabVIEW Real-Time. Therefore, each OS will have access to two NICs (the Virtual Ethernet connection and one physical connection). The Windows XP side of the hypervisor system will be used for LabVIEW Real-Time application development and deployment.

Assume that the physical connections are used to enable communication to the internet from either OS via a corporate network. The virtual interface will be used only for communication between LabVIEW Real-Time and Windows XP. The IP address of both physical adapters connected to the corporate network will be dictated via DHCP, and will be in the range 10.0.0.x with subnet mask 255.255.255.0. The gateway address is set to 10.0.0.1.

Following the guidelines above, we should set a static IP for each of the Virtual Ethernet adapters using a different subnet than is used with the physical adapter on each OS. Although other addresses in the 10.x.x.x range could be used, to be conservative we will set the IP addresses of the Windows XP and LabVIEW Real-Time adapters to 192.168.0.1 and 192.168.0.2 respectively. The subnet mask of each is set to 255.255.255.0.

Because we want both Windows XP and LabVIEW Real-Time to access the internet via the physical NICs, they should be the only adapters that have a default gateway set. Therefore, we will leave the default gateway entry for both Virtual Ethernet NICs empty. Note that each OS ultimately has only one default gateway specified.

LV RT Physical NIC	LV RT Virtual NIC	Win XP Virtual NIC	Win XP Physical NIC
IP Address:(DHCP) 10.0.0.x	IP Address:   192.168.0.2	IP Address: 192.168.0.1	IP Address: (DHCP) 10.0.0.x
Subnet Mask:(DHCP) 255.255.255.0	Subnet Mask: 255.255.255.0	Subnet Mask: 255.255.255.0	Subnet Mask: (DHCP) 255.255.255.0
Default Gateway:(DHCP) 10.0.0.1	Default Gateway: 0.0.0.0 (unspecified)	Default Gateway: 0.0.0.0 (unspecified)	Default Gateway: (DHCP) 10.0.0.1

Table 4. This table shows one possible IP address configuration for the system shown in Figure 6 above. Since there are two OSs running on a single hypervisor controller in this case, the important thing to note is that each OS connects to separate subnets with each NIC, and a maximum of one default gateway.

Case 3: Configuring an NI Real-Time Hypervisor System with Virtual Ethernet (remote development)

In this scenario, two physical NICs are also used in a Real-Time Hypervisor system with the Virtual Ethernet connection enabled as in Case 2 above. However, the physical NIC assigned to LabVIEW Real-Time will be used for deployment from a networked host PC rather than for internet access.

Figure 7. In this scenario, the LabVIEW Real-Time side of an NI Real-Time Hypervisor system is connected to a remote Windows PC for LabVIEW Real-Time application development.

The Windows physical adapter will still be assigned a DHCP address in the range 10.0.0.x with subnet mask 255.255.255.0. The Virtual Ethernet adapters on Windows XP and LabVIEW Real-Time will once again use static IP addresses of 192.168.0.1 and 192.168.0.2 respectively, with subnet masks of 255.255.255.0 and no default gateway.

To ensure that the physical NIC used with LabVIEW Real-Time is on a different subnet than the Virtual Ethernet NIC, we can use an IP address of the form 192.168.1.x with subnet mask 255.255.255.0. Therefore, the remote PC adapter and LabVIEW Real-Time adapter connected together can use static IP addresses of 192.168.1.1 and 192.168.1.2 respectively with subnet masks of 255.255.255.0. Once again, no default gateway is needed since neither adapter needs to access an outside network (no gateways are present on this subnet).

Win Host NIC	LV RT Physical NIC	LV RT Virtual NIC	Win XP Virtual NIC	Win XP Physical NIC
IP Address: 192.168.1.1	IP Address: 192.168.1.2	IP Address:   192.168.0.2	IP Address: 192.168.0.1	IP Address: (DHCP) 10.0.0.x
Subnet Mask: 255.255.255.0	Subnet Mask:255.255.255.0	Subnet Mask: 255.255.255.0	Subnet Mask: 255.255.255.0	Subnet Mask: (DHCP) 255.255.255.0
Default Gateway: 0.0.0.0 (unspecified)	Default Gateway:0.0.0.0 (unspecified)	Default Gateway: 0.0.0.0 (unspecified)	Default Gateway: 0.0.0.0 (unspecified)	Default Gateway: (DHCP) 10.0.0.1

Table 5. With a remote LabVIEW Real-Time development machine added to this scenario, IP address settings must be chosen such that the remote PC can connect with the LabVIEW Real-Time side of the hypervisor system. 

Many other configurations are also possible, including using DHCP with the remote development PC and LabVIEW Real-Time target (to allow connection with the internet), or using additional NICs in the LabVIEW Real-Time target or remote host computer to connect to the internet.

Case 4: Connecting to Two Routed Networks (advanced)

One more advanced scenario involves configuring a computer (running only one OS this time) with two NICs that communicate with two local networks. Each network features a gateway that can be used to relay packets to additional outside networks.

Figure 8. In this advanced scenario, a multi-NIC computer connects to two routed networks. This TCP/IP configuration is more difficult to set up, especially if the outside networks are not supersets of the local network subnets.

Imagine that local network A uses IP addresses of the form 10.0.0.x with subnet mask 255.255.255.0, and local network B uses IP addresses of the form 10.0.1.x with subnet mask 255.255.255.0. Local network A is connected via a gateway to outside network C, which uses IP addresses of the form 192.168.0.x with subnet mask 255.255.255.0. Likewise, local network B is connected to outside network D via a gateway, which uses IP addresses of the form 192.168.1.x with subnet mask 255.255.255.0.

The goal is for our multi-NIC PC to be able to access any of the networks. We can connect one NIC to local network A and assign the IP address 10.0.0.5 with subnet mask 255.255.255.0, and we can connect the second NIC to local network B and assign the IP address 10.0.1.5 with subnet mask 255.255.255.0. If any default gateway were assigned, then packets destined for the outside networks may be sent through the wrong gateway, and so this is undesirable.

To solve this problem, we can leave the default gateway entries blank for the two NICs and configure packet routing more manually in the OS. While all OSs are configured differently, most enable the manual addition of routes by the user, and these routes can be configured to persist across reboots of the system. Specifically, we need to add a route with IP address 192.168.0.x and subnet mask 255.255.255.0 to use the IP address of the gateway between networks A and C (e.g. 10.0.0.1). The same needs to be done with IP address 192.168.1.x and subnet mask 255.255.255.0 for the gateway between networks B and D (e.g. address 10.0.1.1).

NIC 1	NIC 2
IP Address: 10.0.0.5	IP Address: 10.0.1.5
Subnet Mask: 255.255.255.0	Subnet Mask:255.255.255.0
Default Gateway: 0.0.0.0 (unspecified)	Default Gateway: 0.0.0.0 (unspecified)
Special Route: Use NIC 1 and gateway 10.0.0.1 to route to 192.168.0.x	Special Route: Use NIC 2 and gateway 10.0.1.1 to route to 192.168.1.x

Table 6. While typically only one NIC in a controller will connect with a gateway (to a corporate network or the internet), more advanced configurations such as the one pictured in Figure 8 above may require special OS routing entries to function properly. 

Additional routes may need to be added as the number of distinct subnets increases. In practice, most networks are set up to avoid the need for this complex configuration by ensuring that each computer is connected to only one gateway. 

Implementing Bidirectional Forwarding Detection

Thursday, January 13, 2011

Contributed By:
Stefan Fouant 

Many networks today are striving for "five nines" high availability and beyond.

What this means is that network operators must configure the network to detect and respond to network failures as quickly as possible, preferably on the order of milliseconds.

This is in contrast to the failure detection inherent in most routing protocols, which is typically on the order of several seconds or more.

For example, the default hold-time for BGP in JUNOS is 90 seconds, which means that in certain scenarios BGP will have to wait for upwards of 90 seconds before a failure is detected, during which time a large percentage of traffic may be blackholed. It is only after the failure is detected that BGP can reconverge on a new best path.

Another example is OSPF which has a default dead interval of 40 seconds, or IS-IS which has a default hold-time of 9 seconds (for DIS routers), and 27 seconds (for non-DIS routers). For many environments which support mission-critical data, or those supporting Voice/Video or any real-time applications, any type of failure which isn't detected in the sub-millisecond range is too long.

While it is possible to lower timers in OSPF or IS-IS to such an extent that a failure between two neighbors can be detected rather quickly (~1 second), it comes at a cost of increased protocol state and considerable burden on the Routing Engine's CPU. 

As an example, let us consider the situation in which a router has several hundred neighbors. Maintaining subsecond Hello messages for all of these neighbors will dramatically increase the amount of work that the Routing Engine must perform.

Therefore, it is a widely accepted view that a reduction in IGP timers is not the overall best solution to solve the problem of fast failure detection.

Another reason that adjusting protocol timers is not the best solution is that there are many protocols which don't support a reduction of timers to such an extent that fast failure detection can be realized. For example, the minimum BGP holdtime is 20 seconds, which means that the best an operator can hope for is a bare minimum of 20 seconds for failure detection.

Notwithstanding, this does nothing about situations in which there is no protocol at all, for example, Ethernet environments in which two nodes are connected via a switch as can be seen in the figure below. 

In this type of environment, R1 has no idea that R2 is not reachable, since R1's local Ethernet segment connected to the switch remains up.  Therefore, R1 can't rely on an 'Interface Down' event to trigger reconvergence on a new path and instead must wait for higher layer protocol timers to age out before determining that the neighbor is not reachable. 

(Note to the astute reader: Yes, Ethernet OAM is certainly one way to deal with this situation, but that is a discussion which is beyond the scope of this article).

Essentially, at the root of the problem is either a lack of suitable protocols for fast failure detection of lower layers, or worse, no protocol at all.  The solution to this was the development of Bidirectional Forwarding Detection, or BFD, developed jointly by Cisco and Juniper. 

It has been widely deployed and is continuing to gain widespread acceptance, with more and more protocols being adapted to use BFD for fast failure detection. 

So what is the Big Freaking Deal with Bidirectional Forwarding Detection anyway and why are so many operators implementing it in their networks? 

BFD is a simplistic hello protocol with the express purpose of rapidly detecting failures at lower layers.  The developers wanted to create a low overhead mechanism for exchanging hellos between two neighbors without all the nonessential bits which are typical in an IGP hello or BGP Keepalives. 

Furthermore, the method developed had to be able to quickly detect faults in the Bidirectional path between two neighbors in the forwarding plane. 

Originally, BFD was developed to provide a simple mechanism to be used on Ethernet links, as in the example above, prior to the development of Ethernet OAM capabilities.  Hence, BFD was developed with this express purpose in mind with the intent of providing fault identification in an end-to-end path between two neighbors.

Once BFD was developed, the protocol designers quickly found that it could be used for numerous applications beyond simply Ethernet.  In fact, one of the main benefits of BFD is that it provides a common method to provide for failure detection for a large number of protocols, allowing a singular, centralized method which can be reused. 

In other words, let routing protocols do what they do best - exchange routing information and recalculate routing tables as necessary, but not perform identification of faults at lower layers.  An offshoot of this is that it allows network operators to actually configure higher protocol timer values for their IGPs, further reducing the burden placed on the Routing Engine.

BFD timers can be tuned such that failure detection can be realized in just a few milliseconds, allowing for failure and reconvergence to take place in similar timeframes to that of SONET Automatic Protection Switching. 

A word of caution - while BFD can dramatically decrease the time it takes to detect a failure, operators should be careful when setting the intervals too low. 

Very aggressive BFD timers could cause a link to be declared down even when there is only a slight variance in the link quality, which could cause flapping and other disastrous behavior to ensue.  The best current practice with regards to BFD timers is to set a transmit and receive interval of 300ms and a multiplier of 3, which equates to 900ms for failure detection. 

This is generally considered fine for most environments, and only the most stringent of environments should need to set their timers more aggressive than this.

One question that is commonly asked is how is it that BFD can send hello packets in the millisecond range without becoming a burden on the router.  The answer to this question lies in the fact that BFD was intended to be lightweight and run in the forwarding plane, as opposed to the control plane (as is the case with routing protocols). 

It is true that while early implementations of BFD ran on the control plane, most of the newer implementations run in the forwarding plane, taking advantage of the dedicated processors built into the forwarding plane and alleviating the burden which would otherwise be place on the RE. 

In JUNOS releases prior to JUNOS 9.4, BFD Hello packets were generated via RPD running on the RE.  In order to enable BFD to operate in the PFE in JUNOS versions prior to JUNOS 9.4, the Periodic Packet Management Daemon (PPMD) had to be enabled, using the command 'set routing-options ppm delegate processing'. 

In JUNOS 9.4 and higher this is the default behavior and BFD Hello packets are automatically handled by PPMD operating within the PFE.

Cross-posted from SortestPathFirst