Friday, November 25, 2011

Redistributing a Static Default Route Assigned from an ISP via the IOS DHCP Client into EIGRP

Recently, I added a second ISP at one customer’s branch site, along with a second Cisco router for redundancy.  During the implementation process, I encountered a few issues, one of which I felt should be documented for everyone to know.  As the title of this blog states, it is somewhat related to redistributing the default route assigned by the ISP into the local EIGRP routing table.

First, let’s provide the topology onto which this issue occurred.  This customer already had a Cisco 2611XM router running IOS 12.4(15)T14 connected to a local ISP via an Ethernet handoff / WiMax modem.  The customer obtained similar service from a second ISP for redundancy purposes and purchased a Cisco 1921 for it.  My task was to make this Cisco 1921 and the newer ISP the primary, and reconfigure the existing router as a secondary.  The local LAN switch at the site is a Cisco 2950G (layer 2 only).  I made it “dual-uplink” to each of the new routers, with a basic HSRP configuration with the Cisco 1921 always being active (preempt) and the old 2611XM being standby.  The two routers are connected to each other via Ethernet on which I setup a /30 subnet to exchange routes.

The two ISPs did not offer the option to run a dynamic routing protocol for automatic failover.  However, after months of lengthy calls to the engineering department of the newer “primary” ISP, they finally offered a public static IP address I could assign to this new Cisco 1921’s Internet-facing interface.  The older ISP only provides public dynamic addressing obtained from their DHCP servers.

So, with this knowledge in mind, I decided to achieve automatic failover by configuring the primary router’s default static route with some IP SLA state object tracking, in hope that a failure of reachability to some Internet hosts would remove it from the route table.  I also planned on redistributing this default route into EIGRP (using a route map), for advertisement to the secondary router, and in the future further into the customer network when they upgrade their Cisco 2950G with layer 3 distribution switches.  --Planning ahead never hurts.  And, for the secondary router, the interface facing the Internet was already configured with “ip address dhcp”, and a global configuration command of “ip route 0.0.0.0 0.0.0.0 e1/0 dhcp” was already present.  I redistributed this static route into EIGRP using a route map calling a prefix list permitting only the 0.0.0.0/0 route.

Now the problem!  -- At first everything was fine.  Actually, I did failover tests by disconnecting various cables, and simulating router failures, and all worked perfect.  Until an hour and a half after the network was normalized, that is.  The secondary router now had a default route learned from the primary router only, instead of the one assigned by its ISP’s DHCP server.  I was able to ping the ISP’s DHCP server, so I knew it wasn’t an ISP issue.  But why did its default route get removed?  I ran “debug dhcp detail” and “debug ip routing static detail” and found that the IOS DHCP client was assigned a lease with a renewal of 5400 seconds (1h30m).  So, my first assumption was an IOS bug in the renewal process removed the route.  I disabled the interface, and re-enabled it.  The default route assigned by the ISP via option 3 was back.  Thinking it was a fluke, I waited, and sure enough, after 1h30m, the route was removed, and the only default route left was the one learned from the primary router. 

While researching, I read articles stating that a default route assigned by the IOS DHCP client has by default an administrative distance of 254, which is way higher than my primary router’s default route’s administrative distance of 170 (when it reached the secondary router), because it gets redistributed at the primary router from static to EIGRP. 

Anyway, a “show ip route” on the secondary router while the issue isn’t present, shows an administrative distance of 1, which clashes with the online and Cisco CCO articles I read.  I am not sure what to make of that.  But it seems buggy.  Before opening a TAC SR, I cleaned up the configuration of the secondary router a bit, in hope it would resolve the problem.

I wasn’t sure why the secondary router would need BOTH the interface command “ip address dhcp” AND the global command “ip route 0.0.0.0 0.0.0.0 e1/0 dhcp”.  Per the CCO documentation, if the interface command automatically adds the static default route to the routing table, why would I ALSO need the “ip route” command?  So I removed it.  I bounced the interface facing the ISP, and watched the debugs.  Static default route gets added to the routing table!  Ok, so, that “ip route” command seems useless.  I also wanted to make sure this route really gets assigned a lower administrative distance (1) than the one received from the primary router (170).  So, I found an interface command to accomplish this:  “ip dhcp client default-router distance 1”.  I bounced the interface again, and waited several hours.  Problem resolved! 

So, what appeared to be a redistribution problem at first glance ended up being a slight configuration mistake in the way I had configured the DHCP client on that secondary ISP interface, along with an IOS cosmetic bug showing an AD of 1 instead of 254.  Now both routers have their own static default routes redistributed into EIGRP for each other’s use.

If I missed something obvious, or you have more experience than myself on the DHCP client and static routing, please let me know!

Cheers