Sunday, July 19, 2015

Using Ubiquiti Access Points to Provide Wireless Internet to Remote Neighbors who don’t have Broadband Service

In rural areas such as here in Vermont, there are sometimes no other choices than dial-up or satellite Internet service.  But if you are lucky enough to have more bandwidth than you need, you could share it with neighbors who can’t get decent broadband service.  I therefore recently started to work on such a project using two Ubiquiti Nanostation Loco M2 to learn how that would work.  Here is my attempt at documenting the design.

First I made sure line of sight was present.  This dictates what model of Ubiquiti AirMax Nanostation M you pick.  The 2.4 GHz band was fine for my needs.  The other choices were 900 MHz or 5 GHz.  Details about this choice are out of scope for this article but you can learn more online or at

Also, from what I understand, you could pick a Nanostation NSM2 for your AP and Nanostation LocoM2 for the stations at various neighbors as long as both models transmit and receive on the same band.

The Cisco router I chose has a WAN port to connect to the ISP’s bridge, a LAN port for the internal network, and a third port for the Ubiquiti access point (AP). 

IP addressing:

The WAN port has a public subnet assigned showing as “” on the below drawing. 

The internal LAN has a private subnet of

And, the third port is setup as a 802.1Q trunk with subinterfaces as follows. 

We allow two VLANs on this trunk…

VLAN 603 will be the access VLAN which is the public subnet “” shown below.  This VLAN will not be tagged so we added the “native” keyword on the encapsulation command.

VLAN 604 will be the management VLAN which is the private subnet of

Here is the Cisco router’s interface configuration connected to the AP.

interface GigabitEthernet0/2
no ip address
duplex auto
speed auto
media-type rj45
negotiation auto
interface GigabitEthernet0/2.603
encapsulation dot1Q 603 native
ip address
interface GigabitEthernet0/2.604
description WISP MGMT
encapsulation dot1Q 604
ip address

The address will be the default gateway for the WISP client’s router’s WAN interface, and the will allow us to manage the AP and stations.  Here is the drawing:

WISP Diagram

Any consumer grade router can be used for the remote site and because those have NAT/PAT enabled by default, we can have overlapping internal networks.

It took me quite a bit of time to research and come up with a working configuration so I hope the following screens will help someone out there.  Thanks to the good folks who posted their working configs on the forums!  Here is what I did to make it work. 

On the AP:




On the STA:




Note- the IP Aliases allow a tech to connect a laptop directly to the equipment and configure it locally via, the default management IP for AirOS, if needed.  Also, since the VLAN 604 management IP addresses are private (not routed on the Internet), you can only manage them from the local site, hence they are not exposed to the Internet’s threats.

Sunday, September 29, 2013

Using a Cisco Router Running IOS with Bell Canada Business Internet DSL

Most of the small businesses I support have Cisco 800 or 1900 series ISR routers (running Cisco IOS).  And, in the US, I’ve never encountered this problem before with any of the various service providers I’ve dealt with (i.e. Verizon, Fairpoint, etc.).  However, in Canada, on routers with the same PPPoE client config loaded, same version of IOS and same router model, after the router or the DSL modem looses power, PPPoE and subsequently IPCP fails to re-establish the link (obtain the IP address from Bell’s server, etc.).  After one year of troubleshooting and being told by Bell tech support they don’t support Cisco routers or help with configuration of such routers, I was finally able to resolve this problem. 

The problem lies with the modem model supplied by the ISP.  Bell Canada ships by default the 2-Wire 2701 modem/router combo.  If their customers need to use their own Cisco routers (due to VPN requirements, etc.), this 2701 must be converted to a bridge.  Such instructions are not provided and customers are on their own to figure out how to do this.  I was able to do it following an article I found on a Google search.  However, something in this device prevents sending the PADI messages to the Bell network and/or receiving the PADO messages from the Bell PPPoE servers.  So, the result is the router isn’t able to obtain an IP address and finish establishing the WAN link.

It wasn’t until I escalated the issue to a senior executive that I was able to speak with a tier 3 technical support employee, who also wasn’t certain why my known good PPPoE config wasn’t working.  I provided a packet capture which showed my PADI messages sent, but not receiving any PADO messages from Bell.  So, the support engineer suggested shipping us a different DSL modem model.  We received a Speedstream 4200 that was already set in bridge mode (no router function was enabled), which is what we wanted initially!

As soon as the 4200 was connected, we didn’t have to reboot either the router or the modem, and connectivity was established immediately.  A reboot of the router, and a reboot of the modem produced the same immediate connectivity results.  Problem solved! 

NOTE- In all of Bell Canada’s documentation, chatting with a technician pre-sale, and on their website marketing materials, it shows their Business Internet product available with an optional static IP.  This is what we wanted (as opposed to a permanent IP address delivered by PPPoE/IPCP).  However, in reality, a static IP address simply isn’t something Bell offers.  I advised Bell that what they offer shouldn’t be called a “static IP”.  However, it remains to be seen if they will update their website and documentation.

For the record, here is my working PPPoE client config:

interface FastEthernet0/0/0
description BELL BUSINESS INTERNET ADSL / DOWN 8 Mbps / UP 670 Kbps
no ip address
duplex auto
speed auto
pppoe enable group global
pppoe-client dial-pool-number 1

interface Dialer0
bandwidth qos-reference 640
ip address negotiated
ip access-group ACL_INTERNET_TO_FA0_0_0 in
ip mtu 1492
ip inspect CBAC_SOURCE_LAN out
ip virtual-reassembly in
encapsulation ppp
ip tcp adjust-mss 1452
dialer pool 1
dialer-group 1
ppp authentication chap pap callin
ppp chap hostname
ppp chap password 7 [removed]
ppp pap sent-username password 7 [removed]
ppp ipcp dns request
ppp ipcp route default
ppp ipcp address accept
no cdp enable
crypto map CRYPTO_MAP
service-policy output PM_SHAPING

dialer-list 1 protocol ip permit

Friday, November 25, 2011

Redistributing a Static Default Route Assigned from an ISP via the IOS DHCP Client into EIGRP

Recently, I added a second ISP at one customer’s branch site, along with a second Cisco router for redundancy.  During the implementation process, I encountered a few issues, one of which I felt should be documented for everyone to know.  As the title of this blog states, it is somewhat related to redistributing the default route assigned by the ISP into the local EIGRP routing table.

First, let’s provide the topology onto which this issue occurred.  This customer already had a Cisco 2611XM router running IOS 12.4(15)T14 connected to a local ISP via an Ethernet handoff / WiMax modem.  The customer obtained similar service from a second ISP for redundancy purposes and purchased a Cisco 1921 for it.  My task was to make this Cisco 1921 and the newer ISP the primary, and reconfigure the existing router as a secondary.  The local LAN switch at the site is a Cisco 2950G (layer 2 only).  I made it “dual-uplink” to each of the new routers, with a basic HSRP configuration with the Cisco 1921 always being active (preempt) and the old 2611XM being standby.  The two routers are connected to each other via Ethernet on which I setup a /30 subnet to exchange routes.

The two ISPs did not offer the option to run a dynamic routing protocol for automatic failover.  However, after months of lengthy calls to the engineering department of the newer “primary” ISP, they finally offered a public static IP address I could assign to this new Cisco 1921’s Internet-facing interface.  The older ISP only provides public dynamic addressing obtained from their DHCP servers.

So, with this knowledge in mind, I decided to achieve automatic failover by configuring the primary router’s default static route with some IP SLA state object tracking, in hope that a failure of reachability to some Internet hosts would remove it from the route table.  I also planned on redistributing this default route into EIGRP (using a route map), for advertisement to the secondary router, and in the future further into the customer network when they upgrade their Cisco 2950G with layer 3 distribution switches.  --Planning ahead never hurts.  And, for the secondary router, the interface facing the Internet was already configured with “ip address dhcp”, and a global configuration command of “ip route e1/0 dhcp” was already present.  I redistributed this static route into EIGRP using a route map calling a prefix list permitting only the route.

Now the problem!  -- At first everything was fine.  Actually, I did failover tests by disconnecting various cables, and simulating router failures, and all worked perfect.  Until an hour and a half after the network was normalized, that is.  The secondary router now had a default route learned from the primary router only, instead of the one assigned by its ISP’s DHCP server.  I was able to ping the ISP’s DHCP server, so I knew it wasn’t an ISP issue.  But why did its default route get removed?  I ran “debug dhcp detail” and “debug ip routing static detail” and found that the IOS DHCP client was assigned a lease with a renewal of 5400 seconds (1h30m).  So, my first assumption was an IOS bug in the renewal process removed the route.  I disabled the interface, and re-enabled it.  The default route assigned by the ISP via option 3 was back.  Thinking it was a fluke, I waited, and sure enough, after 1h30m, the route was removed, and the only default route left was the one learned from the primary router. 

While researching, I read articles stating that a default route assigned by the IOS DHCP client has by default an administrative distance of 254, which is way higher than my primary router’s default route’s administrative distance of 170 (when it reached the secondary router), because it gets redistributed at the primary router from static to EIGRP. 

Anyway, a “show ip route” on the secondary router while the issue isn’t present, shows an administrative distance of 1, which clashes with the online and Cisco CCO articles I read.  I am not sure what to make of that.  But it seems buggy.  Before opening a TAC SR, I cleaned up the configuration of the secondary router a bit, in hope it would resolve the problem.

I wasn’t sure why the secondary router would need BOTH the interface command “ip address dhcp” AND the global command “ip route e1/0 dhcp”.  Per the CCO documentation, if the interface command automatically adds the static default route to the routing table, why would I ALSO need the “ip route” command?  So I removed it.  I bounced the interface facing the ISP, and watched the debugs.  Static default route gets added to the routing table!  Ok, so, that “ip route” command seems useless.  I also wanted to make sure this route really gets assigned a lower administrative distance (1) than the one received from the primary router (170).  So, I found an interface command to accomplish this:  “ip dhcp client default-router distance 1”.  I bounced the interface again, and waited several hours.  Problem resolved! 

So, what appeared to be a redistribution problem at first glance ended up being a slight configuration mistake in the way I had configured the DHCP client on that secondary ISP interface, along with an IOS cosmetic bug showing an AD of 1 instead of 254.  Now both routers have their own static default routes redistributed into EIGRP for each other’s use.

If I missed something obvious, or you have more experience than myself on the DHCP client and static routing, please let me know!


Thursday, April 28, 2011

Integrating DHCP, DNS, NTP, NAT, IOS Firewall, Port Forwarding, IPsec VPN and WiFi Services Into a Single Cisco Router For Small Businesses

In the past year, a couple of my small business customers gave me the task to upgrade their Internet access router to a more robust one.  Their Linksys/D-Link wasn’t up to the task anymore and kept having to be rebooted or had other problems.  So, I decided it was time to ditch the consumer-grade routers for business-oriented hardware.  Based on Cisco marketing materials, I thought the newer Integrated Services Routers (ISR) might be relatively easy to configure to accomplish this upgrade.  I would later discover it wasn’t so obvious! 

At the beginning, my customers gave me the following requirements:

- Employees must be able to securely access the internal LAN from anywhere on the Internet

- Contractors must also have secure access to the internal LAN but be  restricted to a single machine since there is no need for them to use any other internal resources

- DNS name resolution by internal LAN hosts for the company’s own zone must resolve to internal IP addresses

- DNS name resolution for all other zones as well as the company’s own external addresses should use the ISP DNS servers

These small businesses did not have a DNS server or any LDAP directory of any kind.  One had mostly Mac computers and a development server hosting test web sites for its clients.  And the other was so small (a few computers, and a NAS appliance) that I decided a Cisco 871W router would probably be adequate.

The Cisco 871 turned out to be a good choice for these environments.  However, I had problems integrating the different services needed into one box.  There are so many choices of different technologies in IOS for accomplishing the same task, that it sometimes becomes difficult to choose components that work well together in the same box.  For example, I needed to ensure CBAC (the IOS Firewall), PAT (NAT overload), inbound static NAT (port forwarding), remote access IPSec VPN, NTP, DNS, the DHCP service and a WiFi guest network in addition to the regular private secure network could all co-exist together. 

I will share a configuration that took me months to write and test in hope it will save some of you time.

First, I always start with the IP addressing scheme.  It is easier to write configurations if the IP addresses are already chosen.  I will explain how I came up with each subnet shortly:

Company/Site Summary:

Internal LAN (VLAN101):

Guest LAN (VLAN102):



Employee remote access VPN pool:

Contractor remote access VPN pool:


I typically choose /25 subnets for the LAN segments because that gives more than enough (block of 126) usable addresses for all situations in a small business.  A /24 would be fine as well.  Keep in mind that a Cisco 871 is sized for a site with less than 20 users and less than 20 outside VPN remote access users.  So, a /25 is already overkill here, however, it’s good practice to size the IP addressing so it never needs to be changed in case of expansion because re-IP projects are never fun, although, the router itself is easy enough to upgrade with the existing addressing.  Don’t assign more than a /24 (block of 254) as that’s generally not good practice and broadcasts on the subnet may become an issue when all addresses become utilized.

The /22 site summary is a bit arbitrary in this case because it’s a single site company but it helps in my mind to establish boundaries from the beginning, even if they have to eventually change.  This way, if the company is bought out or acquires another, it’s then pretty clear what needs to happen from a network standpoint as long as it’s clearly documented.

The loopback0 address is the main IP for the router, and I use it to manage it, and point clients to it for DNS queries, etc.

Finally, the loopback1 address is required for static NAT bypass, as I’ll describe later in this article.

To get started, choose an IOS version.  Make sure you purchase the router with an appropriate IOS feature set as well as a service contract, which allows access to the download section.  Keep in mind, the IOS that comes with the router may not be the one you need.  Although make sure that if you purchase the router with a specific feature set, you stick to it, otherwise you might violate the licensing agreement.

For my two deployments, I installed the ADVANCED SECURITY feature set and initially chose 12.4(22)T1 because it had the new object groups I found useful when I was configuring PIX firewalls in the past.  It is also a recent enough version which  includes all the above services I was mentioning earlier.  I discovered however that the newer DNS View service had a bug (ID # CSCsx53968) which logged the following error each time the router received a DNS query:

%FIB-4-FIBCBLK: Missing cef table for tableid 65535 during CEF samecable event

So, I waited for the fix in 12.4(22)T2.  This is the IOS version running today on my routers, which is not perfect, but doesn’t contain any show stoppers.  I’ll discuss this in more details shortly.

Now that we have the correct IOS running, we can begin the configuration.  We’ll start with just the basics to enable Internet connectivity through the router using NAT, DHCP and CBAC.  I assume we are starting with a blank configuration, meaning an “erase startup-config” and “reload” was done.

First, the access list that will work in conjunction with CBAC.  It will allow only specifically permitted traffic to enter the router through Fa4:

ip access-list extended ACL_INTERNET_TO_FA4
 permit tcp any any eq 22 log ALLOW_SSH
 permit esp any any
 permit udp any any eq isakmp
 permit udp any any eq non500-isakmp
 permit tcp any any eq www

Then, a basic CBAC inspect list.  Normally, you would probably add layer 7 applications to this list, however the layer 4 transports are good enough, meaning they cover all connectivity that will be needed for the end-users for this demonstration.  Also, the “router-traffic” keyword is needed because we want traffic originated in the router to be inspected by CBAC, and consequently its return traffic allowed back in.  For example, if I am logged into the router, I want to be able to ping a host on the Internet and have my echo-replies allowed back in via Fa4.  If I didn’t have “icmp router-traffic”, return packets would be dropped by the implicit “deny ip any any” in ACL_INTERNET_TO_FA4.  I do not plan on having any TCP traffic initiated by the router out to the Internet, so the “router-traffic” keyword for “tcp” is not necessary.

ip inspect name CBAC_INS_OUT icmp router-traffic
ip inspect name CBAC_INS_OUT tcp
ip inspect name CBAC_INS_OUT udp router-traffic

Then we prepare the outbound NAT access list for outside interface Fa4.  In other words, this access list determines which traffic doesn’t get translated (VPN traffic) and which does (all other traffic).

ip access-list extended ACL_NAT
 deny   ip
 permit ip any

A route map will also be needed for applying the NAT list we created to the outside interface later on:

route-map RM_NAT permit 10
 match ip address ACL_NAT

We now have almost everything we need to enable connectivity.  The last few pieces are DHCP and to apply the CBAC and NAT config to our outside interface.  In the following DHCP server config, we exclude the first 63 addresses including the default gateway from the scope.  We also assign a domain name to our DHCP client interfaces, a default gateway, and some DNS servers.  Notice that for VLAN 101, we assign the router’s loopback0 address as the DNS server.  For VLAN 102, we don’t want guest users to use the router for their DNS queries, so we assign public Internet DNS servers ( and instead.

ip dhcp excluded-address
ip dhcp excluded-address
ip dhcp pool POOL_DHCP_VLAN101
ip dhcp pool POOL_DHCP_VLAN102
   lease 0 1

Lets now configure the interfaces, our final piece in establishing basic ‘wired’ connectivity.  Wireless will be added later.  The “bridge irb”, “bridge 101 route ip” and “bridge 102 route ip” are used to bridge VLAN 101 across the physical FastEthernet interfaces as well as the radio interfaces.  Same thing for VLAN 102.  This way, you can have a single VLAN used across different media.  That’s how most consumer-grade routers are configured by default.  The BVI interfaces are used to assign an IP address for the router for the bridge group.  These BVI interfaces become the “entry point” into the router from the two different bridged media.  So, a simple way to think about traffic flow is traffic on VLAN 101 and the dot11radio0.101 (interface to be configured later) will always enter the router via BVI101, which will now be the place where we can configure access groups and other interface commands if needed.

bridge irb

int lo0
 ip address

int fa4
 description ISP
 ip address 69.x.y.z
 ip access-group ACL_INTERNET_TO_FA4 in
 ip inspect CBAC_INS_OUT out
 ip nat outside

int vlan101
 description EMPLOYEES_192.168.64.0/25
 no ip address
 ip nat inside
 bridge-group 101
 bridge-group 101 spanning-disabled
 no shut

int vlan102
 description GUESTS_192.168.64.128/25
 no ip address
 ip nat inside
 bridge-group 102
 bridge-group 102 spanning-disabled
 no shut

interface BVI101
 ip address
 ip nat inside

interface BVI102
 ip address
 ip nat inside

int range fa0-3
 switchport access vlan 101

bridge 101 route ip
bridge 102 route ip

ip nat inside source route-map RM_NAT interface Fa4 overload

With all that, you should now be able to connect a computer into one of the switch ports and ping a host on the Internet.  You should also be able to ping that same host from the router command line interface (CLI).

Now we need to add internal DNS services for VLAN 101.  We will also be able to leverage this DNS server for our VPN clients once configured.  First add internal host records:

ip host
ip host
ip host
ip host

Then create the default DNS server attributes and forwarders.  This tells the router to query these servers if it can’t resolve a name.  In other words, for all other names except the ones in the above “ip host” commands:

ip dns view default
 domain name
 domain name-server
 domain name-server

Create a DNS view list that will call the default view above:

ip dns view-list DNS_VIEW_LIST
 view default 20

Enable the DNS service:

ip dns server

Apply the DNS view list to interface BVI101, which is the interface that will receive all traffic from VLAN 101 (the internal wired LAN) and dot11radio0.101 (the internal wireless LAN).  It receives all traffic from internal hosts because it is the default gateway, as assigned to the hosts by our DHCP configuration above.

int BVI101
 ip dns view-group DNS_VIEW_LIST

At this time, internal clients should be able to resolve the host records we added above, as well as Internet hosts.  Note- there are many more Cisco DNS View commands available including some to turn a router into an authoritative DNS server, among other things.  I have not found a need to use them in any of my small business customer networks so far.  If anyone has experience setting up more complex “real world” DNS configurations, feel free to comment or email me.

Port forwarding – one of my two customers needed their business partners to be able to HTTP to one of the internal development server on premises.  Normally, I would have suggested to locate the server in a datacenter colocation facility, so it would be attached to highly-available switches, and benefit from dual-homed ISP, as well as proper DMZ security.  However, in this case, for several reasons, it made more sense to keep it on premises.  So, I had to port-forward this TCP traffic to it.  To accomplish this, the ACL applied inbound on Fa4 needs to allow TCP port 80.  We already prepared ACL “ACL_INTERNET_TO_FA4” above with “permit tcp any any eq www” for this.  We also need to add our static NAT command which will forward all TCP Port 80 traffic received on interface Fa4 to

ip nat inside source static tcp 80 int Fa4 80

This setup will work just fine except for remote access VPN clients.  If we were to leave the configuration as is, the Cisco VPN Clients would be subject to the static NAT command for return traffic.  In other words, the router will try to put a global public IP address into the packet’s destination field before sending the packet out interface Fa4 instead of the internal VPN pool address, and the connections will fail.  Therefore, even if you have no remote access VPN on the router, I still suggest to complete the configuration as follows to prevent headaches later.  Create a Loopback1 interface with a /30 mask, and direct all traffic passing from the internal LAN to the VPN pools & (summarized as to make an extra hop to bypass the static NAT by using a route-map and IP local policy:

int lo1
 desc STATIC_NAT_BYPASS_192.168.67.252/30
 ip address

ip access-list extended ACL_NAT_STATIC
 permit ip

route-map RM_NAT_STATIC permit 10
 match ip address ACL_NAT_STATIC
 set ip next-hop

int bvi101
 ip policy route-map RM_NAT_STATIC

Now we will add the remote access VPN configuration.  There are several methods to do this, however, I’ve tried to use the most recent IOS commands including IPsec profiles, which were introduced with DMVPN but can also be useful to simplify our configuration.

This first step is to make sure the VPN traffic does not get NAT translated.  We need to be able to connect between and the internal addresses via this NAT router.  We’ve already accomplished this by preparing our access list “ACL_NAT” above with the command “deny ip”.  The site/company summary is used here because we want nothing in that range to ever translate when going to 

Secondly, we’ll create the VPN IP address pools:



Third, we need to create access lists to tell the router what traffic to encrypt.  We’ll need one for the employees VPN group and one for the contractor VPN group.  The employees will have access to the entire LAN, while the contractors will only have access to the machine at

ip access-list extended ACL_CRYPTO_RAS_VPN_EMPLOYEES
 permit ip

ip access-list extended ACL_CRYPTO_RAS_VPN_CONTRACTORS
 permit ip host

Normally, for a more robust setup, at a larger company, we’d have a RADIUS or TACACS+ server to provision user accounts for remote access VPN.  However, here, we use the local AAA service on the router.

aaa new-model
aaa authentication login AAA_AUTHEN_LOGIN_LOCAL local
aaa authorization network AAA_AUTHOR_NETWORK_LOCAL local

username testuser secret testpassword

Create the IKE policy using pre-shared keys, DH group 2, and 3DES encryption:

crypto isakmp policy 30
 encr 3des
 authentication pre-share
 group 2

Create the two IKE VPN groups and assign a domain name, as well as our DNS server IP address so the clients can resolve our internal names.  Each group also has a different IP address pool so we can configure access lists to restrict traffic further if we need.

crypto isa client configu group EXAMPLE_GROUP_EMPLOYEES

crypto isa client configu group EXAMPLE_GROUP_CONTRACTORS

Create profiles with appropriate local authentication and assign the VPN groups to their individual profiles so they can be used in crypto maps later on.

crypto isakmp profile PROFILE_IKE_RAS_VPN_EMPLOYEES
   description Cisco VPN clients profile for employees
   match identity group EXAMPLE_GROUP_EMPLOYEES
   client authentication list AAA_AUTHEN_LOGIN_LOCAL
   isakmp authorization list AAA_AUTHOR_NETWORK_LOCAL
   client configuration address respond

   description Cisco VPN clients profile for contractors
   match identity group EXAMPLE_GROUP_CONTRACTORS
   client authentication list AAA_AUTHEN_LOGIN_LOCAL
   isakmp authorization list AAA_AUTHOR_NETWORK_LOCAL
   client configuration address respond

Create a transform set with 3DES/ESP/SHA encryption for our VPN traffic:

crypto ipsec transform-set ESP-3DES-SHA esp-3des esp-sha-hmac

Create a dynamic crypto map which can be assigned to clients that negotiate dynamic IP addresses, which happens to be all clients, as opposed to a site-to-site tunnel between two static, public IP addresses.  Associate the profiles to this transform set:

crypto dynamic-map CRYPTO_DYN_MAP 20
 set transform-set ESP-3DES-SHA 
 set isakmp-profile PROFILE_IKE_RAS_VPN_CLIENTS

crypto dynamic-map CRYPTO_DYN_MAP 30
 set transform-set ESP-3DES-SHA 

Assign the dynamic crypto map to a crypto map that can then be applied to our external interface Fa4:

crypto map CRYPTO_MAP 110 ipsec-isakmp dyn CRYPTO_DYN_MAP

int fa4
 crypto map CRYPTO_MAP

At this point, the router is ready to accept Cisco VPN Client connections configured with the VPN groups and keys.  Clients will also be prompted with extended authentication verified against local username commands.  Once connected, VPN clients should also be able to use the router to resolve DNS names, as opposed to normally using the DNS servers received by their ISP. 

One problem however:  Testing this function indeed shows that clients obtained the correct DNS server of, but that address is neither responding to ping nor DNS queries (UDP port 53).  Troubleshooting reveals this is an IOS bug relating to CBAC’s “router-traffic” command, which we applied to ICMP and UDP in our inspect list above.  Removing the “router-traffic” keyword fixes the issue but also prevents the router from being able to initiate UDP and ICMP connections to hosts on the Internet.  In our case, the router needs NTP to sync its clock with an NTP server, so we cannot remove the “udp router-traffic” from our config.  I reported the problem to Cisco several weeks ago, and they are working on a fix.  The bug ID number is CSCtd40862.  While troubleshooting, I discovered the workaround is to add an additional entry in ACL_INTERNET_TO_FA4 to allow traffic from the VPN pools to reach the Loopback0 interface address.  This is clearly a workaround until a fixed IOS image is released:

ip access-list extended ACL_INTERNET_TO_FA4
permit ip host log TEMP_TIL_IOS_BUG_FIX

permit ip host log TEMP_TIL_IOS_BUG_FIX

UPDATE- 4/28/2011 – Cisco has fixed this bug in their IOS 15.1(3)T and 12.4(24)T5 images.  I tested both and the above “TEMP_TIL_IOS_BUG_FIX” entry is no longer needed. 

The VPN traffic can now successfully reach our router’s Loopback0 for its DNS queries, as well as the BVI101 address if ever needed for troubleshooting via ICMP echo.

Finally, let’s configure our wireless infrastructure.  As stated above, we will enable both the private and guest subnets for wireless access. 

Both wireless LANs will be configured with WPA-PSK with AES encryption for the sake of preventing data confidentiality and other breaches.  The guest network will have a short, well known (provided to guests) WPA pre-shared key while the internal WLAN will have a longer (20 characters or more), difficult to crack one.  The guest network could just as well be open (no encryption) and left for users to protect themselves by using a technology of their choice such as VPN, SSL, etc.

dot11 vlan-name VLAN101_192.168.64.0/25 vlan 101
dot11 vlan-name VLAN102_192.168.64.128/25 vlan 102

dot11 ssid EXAMPLE
 vlan 101
 authentication open 
 authentication key-management wpa
 wpa-psk ascii EXAMPLE_WPA_PSK101

dot11 ssid EXAMPLE_GUEST
 vlan 102
 authentication open 
 authentication key-management wpa
 wpa-psk ascii EXAMPLE_WPA_PSK102

interface Dot11Radio0
 no ip address
 no dot11 extension aironet
 encryption vlan 101 mode ciphers aes-ccm 
 encryption vlan 102 mode ciphers aes-ccm 
 speed basic-1.0 basic-2.0 basic-5.5 6.0 9.0 basic-11.0 12.0 18.0 24.0 36.0 48.0 54.0
 station-role root
 no cdp enable

interface Dot11Radio0.101
 encapsulation dot1Q 101
 no cdp enable
 bridge-group 101
 bridge-group 101 subscriber-loop-control
 bridge-group 101 spanning-disabled
 bridge-group 101 block-unknown-source
 no bridge-group 101 source-learning
 no bridge-group 101 unicast-flooding

interface Dot11Radio0.102
 encapsulation dot1Q 102
 no cdp enable
 bridge-group 102
 bridge-group 102 subscriber-loop-control
 bridge-group 102 spanning-disabled
 bridge-group 102 block-unknown-source
 no bridge-group 102 source-learning
 no bridge-group 102 unicast-flooding

With the above, the guest BSSID (EXAMPLE_GUEST) will be broadcasted, while the other one (EXAMPLE) will not.  So, it will be clear to guests which BSSID they need to associate to.

We now need to prevent traffic from the guest VLAN to connect to anything in the company summary except its own subnet and the Internet:

ip access-list extended ACL_VLAN102_TO_BVI102
 permit ip any
 deny   ip any
 permit ip any any

int bvi102
ip access-group ACL_VLAN102_TO_BVI102 in

This was a long blog post to write!  Let me know if it was useful and if you have comments, corrections, etc.  Thanks for reading.

Saturday, September 18, 2010

Cisco IOS Destination-NAT

Throughout my career in networking, I’ve often implemented Source-NAT translation on Cisco routers (the practice of replacing the source IP address in packets) to either hide the private address or because the subnet overlapped with another location.  However, I’ve rarely implemented Destination-NAT (the practice of replacing the destination IP address in packets).  Destination-NAT, it turns out is not as self-explanatory as Source-NAT when it comes to CLI commands.  And, the materials I found on CCO and Google searches did not clearly explain how to do this simple task in Cisco IOS.  So, I think it warrants a quick article in case others need!

Why would you want to Destination-NAT?  It is needed when a service provider assigns private addressing (RFC 1918) to a service you need to target and you already have a route in your network for that subnet, which goes elsewhere.  Or, you need to direct certain lines of business (LOB) to use one circuit over another, to reach the same service.  In this case, you can have one LOB target the real IP to reach the service over a specific circuit, and another LOB would target the Destination-NAT IP.  This way, the two LOBs don’t have to share the circuit.

If you are translating one IP to one IP, there is no need for a pool, or an ACL, for either source OR destination-NAT.  You would use the “ip nat inside source static” or “ip nat outside source static”.  I’ll explain which one to use for which scenario.  I just depends of the direction the session is initiated.  I usually assign the interface for my network (the one closer to the network core) as the “inside” interface using the “ip nat inside”.  And the interface facing the other network as “outside” using the “ip nat outside” command.  This also happens to be the way zone based firewalls work, although NAT has nothing to do with security or firewalls, and you could very well do the opposite.

Before I give a Destination-NAT example, let’s do a Source-NAT because that’s more common.  Here is the topology:

Core network where sessions get initiated ( –> NAT router –> Other network (

In this scenario, we hide the core network’s real host IP address ( for that flow with the NAT range IP address of  So, the “other network” sees our traffic coming from the NAT address.

The IOS CLI command to do this is:

ip nat inside source static

Of course, you have to ensure there is a route for the translated address in the routing tables along the path for any router to reach that destination. 

Now, let’s say we also need to use this NAT router to translate the destination of our packet.  In other words, there is a LOB who needs to reach via a different path, or we simply cannot advertise into our core, for various reasons.  We need the host on our core to be able to target by using destination address

At the NAT router, you can’t use “ip nat inside destination static” because there is no such command in IOS.  Instead, you would use the following:

ip nat outside source static add-route

As you can see, we are dealing with the outside NAT interface, which means we NAT in the return direction, which is why we have to reverse the order of the translation and use “source” instead of “destination”!

The “add-route” keyword may be needed if there is no route (or a route with an incorrect next-hop) in the routing table for the old “not-translated” destination.

Again, you have to ensure the route is advertised inside your core, so the hosts can reach it.

Note- not all applications are tolerant to NAT, so make sure you test!

Monday, August 9, 2010

Upgrading an HP EliteBook 2530p Laptop from 80GB SSD to 160GB SSD

This week, I upgraded one of my client’s out-of-space hard drive in his HP EliteBook 2530p laptop from the factory-shipped first generation 80GB Intel X18-M 1.8in SATA solid state drive (p/n SSDSA1MH080G1) to a newer second generation 160GB Intel X18-M 1.8in SATA solid state drive (p/n SSDSA1MH160G2).  The first generation 5mm drives max out at 80GB and there are no other Intel drives with more capacity.  Since the 8mm X18 drives don’t physically fit into the 2530p, we had to wait for Intel to release the second generation drives, which are 5mm and come with 160GB capacity.

The upgrade was fairly simple and worked as I had planned.  However, I am documenting what I did in case someone else wants the info.  I did not find any forum post on how to do such an upgrade on this particular hardware.

First, I used the Windows 7 Backup and Restore feature to “Create a system image” onto an external USB hard drive.  This will be used to restore the image onto the new drive.  This works also with Windows Vista Business and Ultimate using the “Complete PC Backup”.

I downloaded the HP EliteBook 2530p “Maintenance and Service Guide” from and followed the instructions on removing and re-installing the SSD.  This was painless and there were just a few screws to deal with.

Once the new 160GB SSD was installed, I booted with the Windows 7 installation CD and restored the previously saved image from the USB drive.  The restore process actually works great but it does not use the “unallocated” space at the end of the disk.  So, we still have an 80GB SSD with 80GB wasted.  To expand the partition, I used the latest stable gparted release (which today is

I booted with the CD onto which I burned the gparted .iso image and accepted all the defaults and booted with gparted live.

When in gparted, I selected the “HP_TOOLS” partition (1GB, located at the end of our 80GB partition) and dragged it all the way to the end of the unallocated space of the new 160GB drive using the mouse pointer.  That left some unallocated space in between the C: drive and the HP_TOOLS drive.  So, I was able to expand the C: drive onto this unallocated space, again by dragging the mouse pointer.  We now have a roughly 159GB drive!  I applied the changes, which returned a success message.  I exited gparted and rebooted the notebook.  C: drive now has lots of additional space!

UPDATED 5/16/2011--  Unrelated note- notebooks with regular hard drives must have their BIOS Device Configuration “SATA Mode” set to AHCI, whereas ones with SSD must have it set to “IDE”.  Otherwise, Windows will fail to start, and a “Startup Repair” will not be successful.

UPDATED 5/19/2011-- I also have been aware of a few BSOD due to a bug in firmware 02HD.  Updating to 02M3 on the 160GB SSD is supposed to fix this.  However, upgrading the firmware has been a bit difficult.  Downloaded the new firmware from “Software & Drivers” on however, running that in Windows will not work because the SSD I purchased was not from HP so it says it’s incompatible.  So, downloaded the update from, which consists of an ISO image to be burned on a CD for use to boot/update outside the OS.  The problem is the CD is not detected by any BIOS/Boot Device in any PC or laptop I’ve tried.  So, I followed instructions in, except I skipped step “C)” as my ISO CD did not have a config.sys file.  The upgrade was quick and worked just fine with the 2 files that were on the CD.

Saturday, September 26, 2009

Roaming Profiles Sync-to-Server Problems

For several years, I’ve used roaming profiles in my Microsoft Active Directory domain at home so my family members get to keep their settings after I reimage their computers, which happens once in a while.  Sure I could have used the USMT or FAST wizard or copied the data manually but this “enterprise” way of doing it allowed me to learn the technology enough to actually be able to recommend (or not) to my clients, based on personal experiences with it.  Note here that using ONLY roaming profiles to store user data and settings on a server is NOT recommended.  A comprehensive desktop strategy which integrates several technologies such as redirected folders, and appropriate exclusions via Group Policies, among other things is required for good results.  In my home setup, I’ve used Dan Holme’s solutions collection 3, as described in his Microsoft Press book “Windows Administration Resource Kit, Productivity Solutions for IT Professionals”, ISBN 978-0-7356-2431-3.  This has served me quite well.  Before implementing these solutions, roaming profiles and other technologies never worked well together in my network, so I highly recommend his book.

One particular headache with roaming profiles however is when the profile stops being uploaded to the server at logoff.  I’ve discovered this happens for different reasons, and each time, it was difficult to identify the root cause.  Searching the Internet for the particular error message (or lack thereof), or event ID, did not always lead to any conclusive fix.  Most of the time, solutions such as domain un-join/re-join, or deleting the local profile, I thought were too drastic, and felt like they never would reveal the true cause of the problem.  In general, I tend to troubleshoot in a more methodical way, and don’t fall for the random “shoot-in-every-direction-until-you-kill-the-problem” method.

So, I thought I’d share my findings here in hopes they’d be useful for others.

Once, I replaced a Windows Vista Business x86 no-brand machine for a user with a new Windows Vista Business x64 HP Pavilion machine.  The swap went well, and the user logged in and got his desktop and settings just like in his old machine.  At logoff however, I noticed it was very fast, and the usual 30 seconds to a minute the system takes to write the roaming profile to the server just wasn’t there.  No error message or event ID was present in this case, and had I not been paying attention and known better, I would have concluded “wow, this new machine is fast” and been done with it!  So, I looked on the server and noticed the last time the ntuser.dat file was written was when he had last logged off his old computer.  So, I Googled for a long time and finally found someone with an HP computer who had the same problem and who had to disable the “NVIDIA Display Driver Service” from msconfig/services tab.  When I implemented this workaround, the roaming profile started to get written to the server again at logoff.  And, the service has been disabled ever since with no adverse effect and no impact that I can tell.  This was reported to HP and they advised they would fix it.  I do not know the status of this case however.

Another time, on my own machine, I got an error message at logoff that my roaming profile was not completely synchronized with the server.  I therefore checked the server and found ntuser.dat had indeed been updated when I logged off.  I logged back in and then got a balloon pop up in systray which indicated that I was logged on with a temporary profile.  I went to the even log and found many event ID 1509 errors followed by 1504 corresponding to the previous logoff.  The event 1509 errors were “Windows cannot copy file C:\Users\username\AppData\Roaming\Macromedia\Flash Player\#SharedObjects\TSACWEND\…” “The filename or extension is too long”.  So, I looked into this directory and indeed found many nested folders under this.  Basically, this is cached data from my previous training session I had just taken at Cisco elementk.  The training was apparently based on Flash and had cached too many levels of folders, which prevented the roaming profile mechanism from copying them to the server.  Because I was done with the course and did not need these “cookie-like” files anymore, I deleted the “” folder and did a logoff.  No further errors!

As you can see, roaming profiles can be a bit finicky, however, if you take the time to research the error, it is possible to fix without resorting to the more drastic measures suggested in some online forums.