Windows XPSP3, Thoughts

Okay, so Microsoft has amazed me in light of their usual track record with software patches, and created a patch that doesn’t appear to blow anything up. Granted, all I use my Windows XP image for is the running of the Copilot remote assistance software, for those especially difficult support sessions. But, I can say that so far, Windows XP Service Pack 3 has broken nothing that I am aware of.

In fact, without reading the documentation for the patch, I would have almost no idea what this patch did accomplish aside from taking an inextricable amount of time to install. That being said, there is one cool feature that I am very happy to see. Windows Vista, being the giant ugly, resource hogging, unusable piece of bloat ware that it is, does have a really cool network level feature called internet black hole detection. Basically, internet black holes are created when less skilled network admins filter way too much of the useful signaling traffic the internet depends on to function properly. This is often done with the careful deployment of many, badly configured firewalls. Don’t get me started on firewalls, they accomplish about one tenth of the things people think they do, but I am getting sidetracked here. The end result of a network that has created a black hole in itself is that a very useful signaling system called Path Maximum Transmission Unit Discovery, or PMTUD is broken. The fallout from breaking this obscure, but tremendously important protocol is that many websites on said black holed network will be randomly inaccessible to many of us end users, who would like to visit those websites. Like I said, Windows Vista has a black hole detection and correction service written into the network level. With the installation of Service Pack 3, Windows XP gains this little feature.

In my opinion, this feature alone makes Service Pack 3 a good choice for immediate installation by most people. PMTUD issues are very frustrating for us, as they are the result of absolutely nothing wrong with our service, but we are usually charged with finding a workaround for them anyways. It is pretty hard for us to fix something that is broken on someone else’s network. But, the addition of this patch into a very mainstream operating system such as Windows XP, means that a general purpose workaround now exists that is readily available by default. The better solution would be to educate other networks about how black holes get created, and why they are so very, very bad. But, I suppose that will never happen. The next best thing is to work around them.

Static Electricity is just weird

Our Oregon Road tower has been driving us nuts for months. Odd things happen every time even a small weather event happens – UPS’s shut off for no apparent reason, power supplies blow out, switch ports dying, etc. I know… all of this sounds like either a power or grounding problem. But where and why?

The tower itself is a water storage tank. Steel. Full of water. Connected to miles of underground pipe. You can’t build anything with a better ground if you wanted to.

Power to the site is standard utility power – nothing special but the other tenants on the tower and the owner don’t seem to have any power problems. Why should we?

The weirdness started last summer (2007) when we installed a UPS (backup power) and a network switch at the tower. When a small storm would come by the UPS would shut down. Run over to the tower and everything looked normal but the UPS would be off. Push the power button and it comes right back on. After the 2nd time this happened we ordered a new UPS and installed it.

Due to some interference issues we replaced the backhaul radios feeding the tower. About 2 weeks later a small snowstorm came through and the site shut down again. When I get there the power supplies for the radios are dead and one of the switch ports is dead. At this point I’m wondering if there is something wrong with the new backhaul radios so we switch back to the old ones and test the new ones at the office – they work fine. Now what the heck does all this mean?

Fast forward to Friday night 3/21 when yet another snow storm comes through. Yep – Oregon Road is dead again. Run over there and I can’t believe what I’m seeing. Sparks. Really big sparks. Sparks coming from the network cables jumping to the UPS. Jumping to me when I get too close. Unplug the cables from the power injectors and sparks jump from the cable to the nearest ground or just crack between pins on the cables. Ack! Where the heck is all this electricity coming from? What’s different about this tower?

Time to backup a little bit. Our access points are a Motorola Canopy radio modified by Last Mile Gear (http://www.lastmilegear.com) with a better antenna and a stronger case called a Cyclone. The older Cyclones had a metal mounting bracket that was a pain to install but was all metal and grounded the radio case to the tower. The bracket design was improved about 2 years ago to one that is much easier to install but does not ground the radio case or the antenna.

We use shielded cable to connect the AP’s to the equipment at the base of the tower. We intentionally do not connect the shield at the top of the tower and ground the shield at the bottom. This design creates a Faraday cage inside the shield and prevents electrical noise from entering (or leaving) the cable. The shield is not intended to be a ground for the equipment. If you connect both ends of a shield you no longer have a shield but a conductor.

So what was creating all of the sparks? The only logical explanation is that the dry snow blowing past the antenna and case generates a static charge on the case. As the case is not grounded the charge builds up until it can jump across either the insulated mounting bracket or onto the shield of the Ethernet cable. The shield being closer that was were the charge was going, traveling down the cable, and discharging into the Ethernet switch at the base of the tower. Can blowing air generate much static? Well – I know it occurs on airplanes when flying in precipitation. Helicopters generate very high static charges on the airframe when flying in snow or dust. Apparently it also happens on isolated Access Points as well.

So why are the Cyclone Access Points (AP) not grounded? The idea was that lightning normally discharges from the ground to a cloud (despite how things look). A lightning bolt starts with both a charged path coming from the ground up toward a cloud that meets the bolt coming down from the cloud. The theory is that if you ground the AP you increase the risk that the leader will start at the AP resulting in a greater chance of damage. All good in theory – except that allowing the AP to accumulate static charges due to wind and snow is causing more damage than lightning itself.

We replace the AP with a new one and grounded it very well to the tower. At the base we installed good surge suppressors on both lines coming from the AP and BH with good grounding at the base. We replaced the switch with a new one since the old one was damaged.

I really hope this is the end of this mess. Now to go address the issue at all of the other towers…..

Updated 3/24/08

I had sent Brian Magnuson from Last Mile Gear an email regarding what we had found and asking about grounding the case given the static issues. Brian was kind enough to call me back this morning and we had a long discussion regarding grounding practices for the radios. Brian indicated he had not seen too many other cases where the Cyclone appeared to be collecting a large static charge. He did suggest grounding the AP using the shield in the STP CAT5 cable rather than leaving it open at one end (the common practice for shields). Brian also suggested putting Motorola 600SS’s (surge protectors) at both the top and bottom of the tower (unless you are using a CTM in which case put the 600SS’s only at the top).

I went back to the tower this afternoon and added the surge suppressors at the top of the tower. I had installed a ground wire to the case of the Cyclone on Friday night and decided to leave that in place as I did not have the proper shielded RJ45 connectors in order to ground the Cyclone using the drain wire. I may modify this the next time I need to climb the tower.

As far as the ground lug on the Cyclone case – Brian indicated that he didn’t think they were doing that yet but were going to be doing so on the units with timing onboard.   The reason for adding the ground was that the timing onboard units have surge suppression built into the case and therefore need a ground reference.   Not providing a ground to the case on the existing units was an intentional decision to try to avoid lightning issues.  In our situation where we are seeing strong static buildup it may be necessary to ground the case.