Thursday, April 29, 2010

Revolution Wi-Fi Posts On-Hold Until mid-May

Revolution Wi-Fi will be on-hold until mid-May while I'm on vacation. The posts will pick up where I left off at that time. Until then...


Wednesday, April 28, 2010

Cisco Location Tracking Overview

I've been doing a lot of work on real-time location tracking as part of the Cisco Unified Wireless Network (CUWN) as of lately. Recently, I visited a large warehouse, over 1 million sq. feet large in fact, to perform a location evaluation and tune performance as best possible. I thought I would share my findings and some tips on improving location accuracy.

First, a note on how Cisco's location server and Mobility Services Engine (MSE) perform locationing. Several methods are available to RTLS engineers, whether they use Wi-Fi, GPS, cellular or other technologies as the underlying transport. Those include:

  • Cell of Origin - simple; indicates only the cell to which the mobile device is currently registered.
  • Distance (lateration) - multiple algorithms available; most rely on very precise time measurements and clock synchronization between the mobile device and multiple sensors (GPS for example). Received signal strength (RSS) is one sub-type which does not rely on time measurements but rather on signal strength measurements and a knowledge of the path loss characteristics of the environment (we'll get back to this in a bit...)
  • Angle (angulation) - just as it sounds, uses the angle of incidence at which signals arrive at the receiving sensors to locate the mobile station. Relies heavily on geometric relationships between multiple sensors. Generally sensors use multiple antenna arrays to sample the receiving signal.
  • Pattern Recognition - uses the sampling and recording of radio signal behavior patterns in specific environments. Assumes unique signal characteristics are present at each possible location, creating a unique RF signature. Patterning solutions rely on one of the above mentioned algorithms for the basis of the observed signatures, with RSS being most common.
Cisco's implementation uses what they like to call "RF Fingerprinting". Think of this system as a hybrid between a pure RSS lateration algorithm and a full Pattern Recognition algorithm. Essentially, the network administrator gets the ease of setup of the RSS lateration while still being able to create unique RF signatures of a Pattern Recognition method.

However, Cisco makes it easy on network administrators by supplying 3 pre-defined Pattern Recognition models. Have you noticed the pre-defined RF calibration models in Cisco WCS when you import a floor map - Cubes and Walled Offices, Drywall Office Only, and Outdoor Open Space? In addition, network administrators have the ability to define custom RF calibration models suited to unique environments (like a warehouse for example). In this manner, customers get out-of-the-box functionality for simple deployments while retaining the ability to customize the algorithm for more complex deployments or challenging environments.

In future posts, I'll discuss setting up location tracking and provide some tips for improving location accuracy.


Monday, April 19, 2010

iPad DHCP Issue

Apparently the iPad and iPhone OS version 3.2 has some nasty DHCP behavior.

the iPad uses DHCP to obtain a lease, renews the lease zero or more times (as expected), but then continues using the IP address without renewing the lease further. The iPad allows the DHCP lease to expire, but it continues using the IP address after allowing the lease to expire.

Kudos to Princeton network administrators to taking a pro-active monitoring approach. Spending the time up-front to monitor the network can save countless hours on the back-end troubleshooting emergencies.

Princeton detected this issue because we take a very pro-active stance to monitor for certain kinds of common network problems, including this one. Our network monitoring includes comparing actual IP address usage to DHCP server lease assignments on a daily basis. This allows us to detect some devices using IP addresses not assigned for their use. This is a degree of monitoring that many sites do not perform. We also monitor our DHCP servers very closely for any problems they detect, including when they DHCP-leased IP addresses in-use when they should not be.
As a result, Princeton tends to learn about some kinds of bugs in DHCP client implementations sooner and more often than do many other sites.

Although much less severe, this issue is reminiscent of the early Apple iPhone issue experienced most notably by Duke University. That turned out to be caused by a Cisco bug which caused an ARP storm between wireless mobility peers. However, this iPad issue appears to clearly be a client-related.


Low Latency MAC Bug with 802.11n APs

During a recent VoWLAN deployment, I ran across sporadic voice performance during lab testing prior to production rollout. After double-checking all the recommended setting for a voice deployment, including physical layer RF, everything seemed to be correct.

A search for open caveats in the Cisco controller release notes for version revealed an interesting finding...

Looking up the bug on the Cisco bug toolkit reveals an issue with Low Latency MAC operation with 802.11n access points. This specific bug requests that Low Latency MAC should have no effect on 11n APs until the original bug is fixed. This was implemented as of currently in-development code version, and is really only a workaround.

CSCtc73527 Bug Details
Make Low Latency MAC a no op for 11n APs, till CSCsy66246 is addressed

An AP1252 or AP1142 may be found to be stubbornly transmitting
unicasts at the maximum supported rate. Despite the fact that the
transmissions are failing, the AP does not downshift.


Low Latency MAC ("Voice MAC Optimizations") configured.


Turn off Low Latency MAC.

Further Problem Description:

Low Latency MAC is not supported on the 802.11n APs. (See CSCsy66246.)
When this bug CSCtc73527 is Resolved, configuring LLC will have no effect
on the APs. Once CSCsy66246 is addressed, LLM will be supported.

The original bug details the issue with 11n APs failing to properly downshift to lower voice data rates (also referred to as 'nominal' data rates in Autonomous software) when attempting frame re-transmission. Note that this bug is still open at this time.

CSCsy66246 Bug Details
1250 doesn't downshift rates for retries when low latency mac is enabled
Symptom: The AP1250 doesn't downshift rates for retries when low latency mac is enabled. It will send 3 retransmissions, but the data rate for the retransmissions is the same data rate that the initial packet was sent at.

Condition: Using the AP1250 with low latency mac enabled.

Workaround: Do not enable low latency mac

To briefly explain the issue, Low Latency MAC is a feature which limits the attempted number of packet retransmissions to 3 attempts for packets in the voice queue to age them out quickly. Voice is a real-time protocol and frames become useless very quickly if delayed too long. They can disrupt a voice conversation and actually be counter-productive. It also controls the nominal data rates used to transmit packets in the voice queue, thereby reducing data rate shifting to a minimal set of data rates to ensure successful delivery of packets and avoid retransmissions.

So rather than continue to repeatedly attempt frame re-transmission, the network will only try 3 times at which point it will drop the frame. This will help clear up network and access point resources to deliver frames which still have validity for a voice call.

If you're deploying voice on 11n APs, be sure to disable Low Latency MAC... for now (and keep a close eye on the status of this bug ID). You can still safely enable EDCA, which controls the Adaptive Inter-Frame Spacing and Contention Window values in a QoS Basic Service Set, to any of the available settings without issue (WMM, Spectralink, Voice Optimized, Voice and Video).


Monday, April 5, 2010

Local RADIUS on Cisco Autonomous APs

Cisco autonomous access points may be configured as a local RADIUS server to provide AAA authentication services. This is typically done for a small wireless LAN which can't afford a centralized solution, to provide a backup authentication service, or to facilitate infrastructure connections between access points for bridging or WDS operation.

As a local RADIUS server, the access point performs LEAP, EAP-FAST, and MAC-based authentication for up to 50 client devices. The access point performs up to 5 authentications per second.

When you configure the local authenticator as a backup to your main servers, the access points periodically check the link to the main servers and stop using the local authenticator automatically when the link to the main servers is restored.

Deployment guidelines:
  • The local RADIUS server only listens on UDP ports 1812 and 1812, not the legacy ports 1645 and 1646
  • Do not use an AP that serves a large number of clients because performance may be degraded
  • Secure the AP because it contains sensitive credentials in the configuration file
Configuration of local RADIUS is fairly simple when keeping a few architecture points in mind as a reference point:
  1. The AP serves the role of the 802.1x authentication server once local RADIUS is enabled
  2. The AP also serves as the 802.1x authenticator (NAS), the same as when using a central RADIUS server
  3. The two roles must be tied together
  4. The same AAA new-model architecture and commands are used for configuration of RADIUS servers, including AAA groups, method-lists, and assignment of method-lists to SSIDs, WDS, etc.
Here is an example configuration to illustrate. The local AP has an IP address of and is configured to point to itself as for client authentication on the SSID "bridge".

aaa new-model
! Define a AAA group containing the local RADIUS server.
aaa group server radius rad_local
   server auth-port 1812 acct-port 1813
! Define a AAA login method-list.
! This list will be applied to SSIDs, WDS, etc.
aaa authentication login eap_local group rad_local
! Apply the method-list to an SSID if desired. This
! SSID example shows a bridge-link with this AP as the Root.
dot11 ssid bridge
   vlan 4
   authentication open eap eap_local
   authentication network-eap eap_local
   authentication key-management wpa
! Enable the local 802.1x authentiation server on the AP.
! Define what authentication methods (LEAP, EAP-FAST, MAC) are
! not allowed, what NAS clients are allowed, and create local
! users on the AP.

! Configure the local AP and any other APs authenticating clients
! as NAS entries.
radius-server local
   no authentication mac
   nas key nas-shared-secret
   user wds password wds-user-password
   user bridge password bridge-user-password
! Configure the local AP 802.1x authenticator. Create an entry
! for the local RADIUS server.
radius-server host auth-port 1812 acct-port 1813 key nas-shared-secret
The configuration of local RADIUS is straightforward as long as you remember that the AP is serving both 802.1x roles which must be configured independently.
Several additional features may be configured in the local RADIUS server, which are not illustrated in the example above:
  • User groups, which may restrict users to specific VLANs or SSIDs, define client re-authentication timers, and specify EAP-FAST PAC expiry periods.
  • MAC authentication is provided when user's are defined using the hexadecimal MAC address (without delimiters) as both the username and password.
  • EAP-FAST server identity (authority ID, server-key)
  • Manual PAC file generation and export for installation on client devices

Although defining a local RADIUS server is fairly easy, remember that it is not a scalable solution. However, it may have a place in your network design strategy for small sites, backup authentication service, or authentication of infrastructure components without relying on a centralized solution.

I have most-often seen local RADIUS deployed to ensure bridge links and WDS services remain in operation even though connection to central RADIUS servers is interrupted.


Cisco WDS Configuration

Configuring WDS is fairly straightforward, given an understanding of the concepts explained in my last post, Cisco WDS Overview.

First, configure one or multiple WDS master devices:
  1. Set the WDS client username and password (the master will authenticate itself as a client):
    wlccp ap username username password password

  2. Set the AAA method list to authenticate WDS client access points. The method list may include the local RADIUS server on the master access point, if enabled.
    wlccp authentication-server infrastructure aaa-method-list

  3. Set the AAA method list(s) to authenticate wireless clients. Multiple commands may be entered to specify different method lists depending on the authentication type used (EAP, LEAP), and SSIDs may optionally be specified under each list to restrict their application. If no SSIDs are specified under a list, then it applies to all SSIDs.
    wlccp authentication-server client { any | eap | leap } aaa-method-list
         [ ssid ssid-name ]

  4. Set the WDS master priority. The higher priority is elected as the active master. Values range from 1 - 255.
    wlccp wds priority value interface BVI1

  5. Optionally, set the wireless network management server of the WLSE:
    wlccp wnm ip address ip-address

  6. Optionally, set the WDS master to WDS only mode to prevent wireless client associations:
    wlccp wds mode wds-only

Second, configure the WDS client access points:
  1.  Set the WDS client username and password:
    wlccp ap username username password password

  2. Optionally, specify the WDS master address instead of waiting for multicast advertisements. If the specified WDS master does not respond, the WDS client reverts to listening for multicast advertisements.
    wlccp ap wds ip address ip-address

Verify WDS operation:
  •  Check the status of the WDS master, connected WDS client APs, and authenticated wireless clients (mobile nodes):
    show wlccp wds [ ap | mn ]

  • Check the status of the WDS client:
    show wlccp ap

Here is a simple WDS master configuration with two SSIDs. SSID "ccie" serves wireless clients using a central AAA server defined in the method list "eap_cisco". SSID "bridge" serves a non-root bridge using the local RADIUS server on the WDS master for authentication.

wlccp ap username wds password mysecret
wlccp authentication-server infrastructure eap_local
wlccp authentication-server client eap eap_cisco
  ssid ccie
wlccp authentication-server client leap eap_local
  ssid bridge
wlccp wds priority 250 interface BVI1
Verification of the WDS master:

Root#show dot11 associations

802.11 Client Stations on Dot11Radio0:

SSID [bridge] :

MAC Address    IP address  Device     Name    Parent State
0017.df96.0a50 11g-bridge Nonroot self   EAP-Assoc

Root#show wlccp wds ap
Nonroot  0017.df96.0a50 REGISTERED
Root     0016.c7d2.32be REGISTERED

Root#show wlccp wds mn
MAC-ADDR       IP-ADDR     Cur-AP         STATE
0017.df96.0a50 0016.c7d2.32be REGISTERED

Here is an example WDS client configuration:

wlccp ap username wds password mysecret

Verification on the WDS client:

Nonroot#show wlccp ap
WDS = 0016.c7d2.32be,
state = wlccp_ap_st_registered
IN Authenticator =
MN Authenticator =


Cisco WDS Overview

In this post, I will explain the purpose of Cisco Wireless Domain Services (WDS) and how wireless deployments can benefit from use of this feature. In a future post, I will detail how to configure WDS services on autonomous access points.

WDS is a service that runs on Cisco autonomous access points to provide coordinated management and control of wireless services across multiple standalone devices (APs). By participating in WDS, the access points have broader knowledge of their environment and can provide better service to clients than would otherwise be possible. Think of WDS as creating a common collective (... akin to The Borg for you Star Trek TNG fans!) amongst access points. This protocol builds upon the defunct Inter-Access Point Protocol (IAPP) which never took hold. The final evolution of this type of service can be seen in today's wireless controller based architectures.

WDS enables features to improve client performance and security of wireless LAN networks. The main features include:
  • Cisco Centralized Key Management (CCKM)
    CCKM is a Cisco proprietary protocol that enables clients and access points to cache and re-use keying material derived from a full 802.1x/EAP authentication. This enables clients to roam between access points faster without the need to perform a full re-authentication. This feature became especially useful as wireless LANs evolved from static WEP encryption to the much more robust WPA and eventually WPA2. CCKM requires CCXv2 (or newer) compatible clients, depending on the authentication type used.

  • Radio Management
    Access points forward radio management information such as rogue APs and client associations to the WDS master device. The WDS master device aggregates this information and forwards it to the Wireless LAN Solution Engine (WLSE) network management device for centralized logging and alerting. WDS also enables 802.11w management frame protection capability by providing a central point for key distribution and management across autonomous accesss points.

There are three components in the WDS architecuture:
  1. WDS Master
    The WDS master is the central control point for authenticating wireless clients, caching client key material, distributing MFP key material, reporting radio management information to an upstream network management station, and updating other APs participating in WDS. This service may be performed by an autonomous access point, Wireless LAN Services Module (WLSM) which is end-of-life, or by an ISR router. Only one device may be the active WDS master, but backup masters may exist.

  2. WDS Client
    Participates in WDS services and receives required information from the WDS master. Performed by autonomous access points.

  3. Wireless Network Manager (WNM)
    Performs centralized reporting and management. Usually performed by a Wireless LAN Solution Engine (WLSE).

Typically, autonomous access points are used as WDS masters due to the lack of available options with the WLSM and ISR router modules. When using an AP as the WDS master, a limitation of 30 WDS clients are supported if also serving wireless clients. If not serving wireless clients, then up to 60 WDS clients are allowed.

WDS communicates over the network using the Wireless LAN Context Control Protocol (WLCCP). This protocol uses the multicast destination address 01:40:96:ff:ff:c0 for advertisement and discovery of the WDS master. Once the master is discovered by WDS clients, then unicast UDP sessions are created over port 2887 to authenticate the WDS client and exchange information. Alternatively, the WDS master IP address may be configured statically in the WDS client if the master is not on the same subnet or if multicast routing is not enabled.

A few important features should be understood by network administrators prior to implementing WDS:
  • WDS master election is performed by comparing priority values. The device with the highest priority is elected master.
  • One or more backup WDS candidate devices may exist should the master fail.
  • WDS clients authenticate to the WDS master using LEAP. Therefore, LEAP must be enabled in the AAA server performing authentication for WDS devices.
  • All wireless client authentications are performed by the WDS master when active.
  • WDS clients will revert to standalone mode if the WDS master fails and CCKM fast roaming will not be available.
  • If a backup master exists, the WDS clients will re-join the new master and begin forwarding wireless client authentications again.
  • Network-EAP (LEAP) must be enabled on SSIDs performing CCKM fast roaming, even if wireless clients are authenticated using another EAP type.
In the next post, I will cover configuration of WDS on access points and verification of proper operation.


Thursday, April 1, 2010

Determining AP Transmit Power

One feature of the Cisco UWN that causes a lot of confusion for people is accurately determining lightweight access point power levels.

In the autonomous world this is fairly cut and dry, you configure the power level on each radio using the "power local dBm" interface command, which is then adjusted downward if necessary based on the channel and regulatory domain configured on the access point. You can then verify the power level in use with the "show controllers Dot11Radio" command:

Configured CCK Power: 20 dBm
Configured OFDM Power: 20 dBm
Active power levels by rate
     1.0 to 11.0 , 20 dBm
     6.0 to 54.0 , 17 dBm, changed due to regulatory maximum
  OffChnl Power: 20, Rate 1.0
Allowed CCK Power Levels: -1 2 5 8 11 14 17 20
Allowed OFDM Power Levels: -1 2 5 8 11 14 17 20
Allowed Client Power Levels: 2 5 8 11 14 17 20

Here, you can see the configured power level is 20 dBm, and the active power level is 20 dBm for 802.11b data rates and 17 dBm for 802.11g data rates. Also notice the output states that the power level for 802.11g data rates has been automatically changed due to the regulatory maximum. This AP is deployed in the U.S. and follows the -A regulatory domain restrictions.

However, on the Cisco UWN things become a bit harder to interpret. This is mainly caused because the power level assignment is performed using numeric level representations, not by absolute dBm value. Whether the power level is assigned dynamically through TPC (part of RRM) or statically by the administrator, it is always represented with numeric values of 1 through 8 (1 being highest power, 8 being lowest). Each power level reduction roughly corresponds to a 3 dB decrease (50% absolute power decrease), but this does not always hold true.

How do you determine the actual power level in use by a lightweight AP?

First, let's view the supported power levels on the AP. We can do this through the CLI with the command "show ap config { 802.11a | 802.11b } ap_name"

Tx Power
  Num Of Supported Power Levels ............. 8

  Tx Power Level 1 .......................... 17 dBm
  Tx Power Level 2 .......................... 15 dBm
  Tx Power Level 3 .......................... 14 dBm
  Tx Power Level 4 .......................... 11 dBm
  Tx Power Level 5 .......................... 8 dBm
  Tx Power Level 6 .......................... 5 dBm
  Tx Power Level 7 .......................... 2 dBm
  Tx Power Level 8 .......................... -1 dBm
  Tx Power Configuration .................... AUTOMATIC
  Current Tx Power Level .................... 2

Phy OFDM parameters
  Configuration ............................. AUTOMATIC
  Current Channel ........................... 36

This is a 1242 AP, and we can see the supported power levels are from 17 dBm down to -1 dBm, mapping to corresponding power levels 1 through 8, as expected. With this information, an administrator might expect the AP to transmit at 15 dBm when set to a power level of 2. This would be wrong!

Here's why. The supported power levels do not take into account other variables affecting tranmit power limits, such as current channel assignment and regulatory domain restrictions. This means that the power level values (1-8) will vary based on the maximum level allowed in the regulatory domain for the current operating channel. Therefore, any given power level value will not always equate to the same absolute power level in dBm across APs on different channels. Talk about making an administrator's life more difficult! This is more of a concern in the 5 GHz bands than the 2.4 GHz band, due the different intended purposes for various spectrum blocks in 5 GHz (outdoor vs. indoor, DFS requirements, etc.).

Continuing the example, the same command above also lists the current channel as 36. The FCC regulatory maximum transmit power for this channel is 11 dBm. This information can be found in the Cisco document "Channels and Maximum Power Settings for Cisco Aironet Lightweight Access Points, OL-11321-07". So, power level 1 will correspond to 11 dBm when using channel 36 in the U.S., power level 2 to 8 dBm, and so on down the list. (Note, that if only 5 power levels are supported, anything lower than power level 5 configured on the AP will apply the lowest power level supported)

How can we verify this information? We can see the adjusted active power level of the AP using a few CLI commands:

debug ap enable ap_name
debug ap command "show controller Dot11Radio1" ap_name

(Cisco Controller) debug ap enable ccielwap
(Cisco Controller) debug ap command "show controller d1" ccielwap

Mon Feb 22 08:19:34 2010: ccielwap: interface Dot11Radio1

Mon Feb 22 08:19:34 2010: ccielwap: Carrier Set: Americas (OFDM) (US) (-A)
Mon Feb 22 08:19:34 2010: ccielwap: Uniform Spreading Required: Yes
Mon Feb 22 08:19:34 2010: ccielwap: Configured Frequency: 5180 MHz Channel 36

Mon Feb 22 08:19:34 2010: ccielwap: Configured Power: 8 dBm
Mon Feb 22 08:19:34 2010: ccielwap: Active power levels by rate
Mon Feb 22 08:19:34 2010: ccielwap:      6.0 to 54.0 , 8 dBm
Mon Feb 22 08:19:34 2010: ccielwap:   OffChnl Power: 11, Rate 6.0
Mon Feb 22 08:19:34 2010: ccielwap: Allowed Power Levels: -1 2 5 8 11

Wow! How's that for complex. The easiest way that I have found to deal with this is to know your environment and your design principles. If you typically use a certain channel pattern layout or have limited the available channels in the DCA configuration, know the corresponding maximum power levels allowed for each channel. Perhaps make a quick cheat-sheet for reference. Many channels will be the same, but the few that are different will be important to remember.

Do you know of any other tips or tricks for determining power levels?