Upcoming Maintenance, AKA: Nuintari Will be Sleepy

The Domain Name System, often referred to simply as DNS is fundamentally important to internet communications. DNS is the system that translates easily remember names, such a www.amplex.net into something a computer can understand. Computers don’t understand English, or even the crazy subset of English that make up most domain names. When you type www.amplex.net into your browser, one of the first things your computer tries to do is figure out what that really is in a form it can understand. It asks one of your ISP’s DNS servers, Amplex has three. One of those DNS servers should get back to you, and inform your computer that www.amplex.net is in fact 64.246.100.105. This makes far more sense to your computer, and it will begin connecting to the server that houses www.amplex.net. This holds true for just about any site you wish to get to. Most DNS servers don’t actually know the answer to most sites you are going to try to find, but they know who to ask. Amplex’s DNS servers know everything about anything at amplex.net, as well as any other websites we happen to host. For anything else, we ask another set of DNS servers. Those servers might know the answer, or more likely, they also know who to ask. After many questions, you will usually get the information you seek. Our DNS servers will then hold onto that information for a little while, in case someone else also needs that information. This process is calling “caching.”

Unfortunately, almost all caching DNS servers have recently been observed to contain a software issue where a malicious third party could interfere with the data in the DNS server’s cache. So, when you try to reach www.whatever.com, your computer might be tricked into going to someplace nasty, with a virus waiting for you. Never fear, this was just a proof of concept, no known exploits exist for this yet, and most DNS software vendors, ours included had patches released this past weekend. Of the three Amplex DNS servers, I have successfully patched two of them without incident, the third one, the busiest of the three, and the one most likely to cause a disruption when I bring it down for a patch, is getting patched tonight, very late tonight. Like I said, we have three DNS servers, and most customers are configured to use the two closest to them for name resolution. But, the most centrally located, and therefore closest to the most people is the one I am working on tonight. basically means, if you are surfing the web tonight, you might notice, as I have to reboot the machine twice to properly apply this patch.

Long story short, expect a brief outage, starting at about 1am tonight, as I work my magic.

New Webserver!

I have just finished what I hope is the last bug tests for the new FreeBSD/Apache web server. This will replace the current web server, which has been very stable and very reliable, and I hope I am not changing that, as it is running the same base software as before. What is different is that my sanity will no longer be negatively effected by administration of this machine. The current web server requires me to log in and make changes to the configuration, by hand, every time someone needs a new site, or a change to a site. This is needlessly wasteful of my time, and slows down site implementation times. New sites take me well over ten minutes to manually enter and bring live. Site visitor tracking has to be manually configured as well, and I have a nasty habit of forgetting to set that up for new sites. There is also the issue that sites on the web server, do not always get entered into the billing system. Data that needs to be entered in two places and kept consistent is never a good idea.

So, the wonderful solution was to build a new machine that is completely tied into our billing system. You want a new site, I enter it into the billing system, which provisions the site for me. I only have to input a sitename, a username, and a password. You get a site that is ready to go, ftp access, and stats tracking, all automatically.

Now I have to start moving customer sites over to the new machine, a process that should about the next eternity. In addition to the new automated provisioning system, a few upgrades were an order while I was at it. Most notable being that PHP version 4 is no longer supported, so the new webserver has version 5 installed. This is almost guaranteed to break several websites, as the differences between versions 4 and 5 are immense. But, security patches for version 4 no longer exist, meaning any new bugs found in version 4, will never be fixed. Kind of a bad thing to leave PHP4 installations around at this time.

If you have a site on the current webserver, and want to migrate it to the new machine yourself. Please let me know. I can give you access to both machines at once, and you would be saving me a load of time.

Windows XPSP3, Thoughts

Okay, so Microsoft has amazed me in light of their usual track record with software patches, and created a patch that doesn’t appear to blow anything up. Granted, all I use my Windows XP image for is the running of the Copilot remote assistance software, for those especially difficult support sessions. But, I can say that so far, Windows XP Service Pack 3 has broken nothing that I am aware of.

In fact, without reading the documentation for the patch, I would have almost no idea what this patch did accomplish aside from taking an inextricable amount of time to install. That being said, there is one cool feature that I am very happy to see. Windows Vista, being the giant ugly, resource hogging, unusable piece of bloat ware that it is, does have a really cool network level feature called internet black hole detection. Basically, internet black holes are created when less skilled network admins filter way too much of the useful signaling traffic the internet depends on to function properly. This is often done with the careful deployment of many, badly configured firewalls. Don’t get me started on firewalls, they accomplish about one tenth of the things people think they do, but I am getting sidetracked here. The end result of a network that has created a black hole in itself is that a very useful signaling system called Path Maximum Transmission Unit Discovery, or PMTUD is broken. The fallout from breaking this obscure, but tremendously important protocol is that many websites on said black holed network will be randomly inaccessible to many of us end users, who would like to visit those websites. Like I said, Windows Vista has a black hole detection and correction service written into the network level. With the installation of Service Pack 3, Windows XP gains this little feature.

In my opinion, this feature alone makes Service Pack 3 a good choice for immediate installation by most people. PMTUD issues are very frustrating for us, as they are the result of absolutely nothing wrong with our service, but we are usually charged with finding a workaround for them anyways. It is pretty hard for us to fix something that is broken on someone else’s network. But, the addition of this patch into a very mainstream operating system such as Windows XP, means that a general purpose workaround now exists that is readily available by default. The better solution would be to educate other networks about how black holes get created, and why they are so very, very bad. But, I suppose that will never happen. The next best thing is to work around them.

Windows XP Service Pack 3

I can honestly say, I am so behind on caring about Microsoft Products, that I had no idea SP3 was even in the works, but it was, and it is here! Windows XP Service Pack 3 has been released as of today. As with all software patches, especially those labeled Microsoft *anything*, I am extremely wary of actually installing this puppy. Oh, I am sure it fixes many, many gloriously awful issues, and of course, causes 3 times as many new ones. I just can’t get excited about trading the bugs I know, for a whole host of new ones. But, I do have an ace in my sleeve today. I don’t run Windows XP as my primary workstation. I keep a copy of it running on a virtualization system called VMWare. VMware allows me to run a virtual computer, on my computer. So, I can have a copy of Windows XP, a copy of Windows Vista, and I can play with the latest Linux distributions, all without gumming up my workstation. It has lots of cool features too, like the ability to back itself up to a known safe state, so I feel pretty confident that installing this service pack won’t annoy me for very long. But I am rambling, I should I stop that.

Like I said, I am wary of any major Microsoft patches, and I strongly suggest you be wary as well. Better that I install SP3 on my virtual computer, and it blow up, rather than you install it on your PC, and it blow up. My virtual computer is very easy to fix, and I wouldn’t care very much if it wasn’t. I will post my impressions of the patch in a few days, after I have had several moments to see what breaks, what is cool, and of course, what makes me shake my head in utter disbelief.

Trouble Ticketing system

In order to better serve you, the customers, and to keep our own sanity, we are starting to use a trouble ticketing system. This is helpful because, as we have grown, we have discovered we cannot keep all of this stuff in a pile of notes on our desk, nor does it all fit in our brains all that well. So, we have centralized all the case notes in a very easily accessed system. The hope is, we will forget less, and when we have a caller with an ongoing issue that may span several days, one of us does not have to play the, “I wonder what the last tech already tried” game, we can just look at the case notes! I know, nothing terribly exciting, but it is important, and we hope it makes our support service faster and more effective.