FOR IMMEDIATE RELEASE
Hometown Traffic to Web Site
RHP Studios Apologizes for Errors beyond our control.
02/23/01 -- RHP Studios monitors all web sites in which we manage or have under our direct control. At approximately 12:43 PM on Friday, February 23, all traffic from 1 of the 8 subnets on the cable network here in Maysville dropped out of reaching any Verio web sites. It appeared to be router related. RHP Studios then checked from several other subnets and found the traffic to only affect the subnet in which our office is located.
We made several immediate phone calls to the local manager, Jeff Craycraft, and commercial tech support. Jeff Craycraft advised he was able to reach the RHP Studios domain and that I should not have any problems being on the same line. An over-the-phone comparison of trace routes revealed something odd...we were being routed differently. A call to HSA Commercial Support was made at approximately 4 P.M. I was advised at approximately 2330 hrs (11:30 P.M.) that they had identified a problem and was working to correct it.
I stayed up waiting on traffic to be restored and running more diagnostics from several shell accounts. I had several major updates to complete. During this time, the remaining subnets in Maysville also lost routing to Verio hosted web sites. A friend first advised me of this at approximately 0100 hrs on 02/24/01. I was told the problem appeared to be Verio's. I thought that it was odd that If you used a proxy server located off of HSA/UUNet, you could reach any Verio hosted web site. The question then, was Verio blocking only one routers traffic? If you routed through UUNET in Chicago, you were stopped cold every time. I also verified this from various shell accounts. I wanted to let the Maysville customers know this problem was not related to RHP Studios, but how could I notify the Maysville community of this without spamming them?
In further diagnostics, I found that everyone located anywhere in the world could reach RHP Studios and other Verio hosted web sites....except Maysville cable subscribers. How odd for this to occur less than 24 hours after RHP Studios new marketing/advertising campaign ran in the local Ledger-Independent Newspapers. I used a shell account in which I could redirect all email traffic sent to the RHP Studios domain and spent the remaining night setting up an encrypted channel between RHP Studios, the shell account, and my home computer so I would not be cut off from email on those who were able to make it to the web site.
At around 5 a.m. on Saturday, February 24, I finally gave up on waiting for the router problem to be fixed and went to bed.
DAY 2 - Not able to sleep, I woke up around 11:30 a.m. on Saturday, February 24 and checked my monitors for any signs of improvement. A quick call to Commercial Tech Support revealed they were "working on it". The web site was down to all visitors from Maysville who were on the HSA (High Speed Access) cable service or any other ISP's that were routed through 220.127.116.11 or 18.104.22.168. On the evening of February 24, I was told the problem was....."Verio is blocking all traffic from Maysville". Why would they do that? A quick call revealed this to not be an issue. Verio's. stated the issue was UUNET's router had a major problem and that I should send my ISP to a link on their web server for Verio's suggested fix. Verio asked me to send them the same traceroutes, MTR reports, dig, and other utilities I had ran in order to 'find the answers myself'. Verio was extremely supportive in attempting to get this resolved. I also attached a copy of a report sent to me from HSA Commercial Support, showing a technician was able to login to the Maysville gateway and ping both IP addresses in question. I thought maybe there was a problem with my firewall or some other unexplainable reason for not being able to reach the two sites I desperately needed to reach, or else the technician wasn't telling the truth. Why else could he be able to ping the IP addresses directly from the Maysville gateway, yet any traffic before the gateway had the sites as unreachable?
Within a few hours after sending the requested information, Verio responded back saying that all traffic to the web sites I mentioned hosted on Verio was OK, but a trace route from them to my own IP address located in Maysville...along with a few others I submitted "Showed Serious Connectivity Issues within your ISP's router on (alter.net). Please forward this information to them." The information contained in the email included this statement, "Verio does not accept announcements with a prefix length longer than a /20 in 22.214.171.124/6 and this is a fairly standard practice. Verio filtering policies can be found at <link removed for this article>. If you have any other issues or requests, please reply to this email." It was another long night, unable to sleep. This email was sent to commercial tech support at HSA who immediately responded with, "Yes, specifics!!! Thanks, this should help out our NOC guys a lot."
DAY 3 - I woke up on Sunday, February 25 around 10 a.m. Guess what, still no traffic from Maysville to any Verio hosted web sites. This remained through-out the day and night. I'm starting to get impatient at the obviously incompetent crew "working on the problem" - wherever the problem lies. Router issues can be somewhat a pain in the yang to fix, but 72 hours?? What more could HSA ask for....Verio supplied what they believed to be the exact problem, and even the servers/routers with the problem were identified.
DAY 4 - I had several things that needed to be completed today. I called Commercial Tech Support at 10 a.m. on Monday, February 26. I was told their "Ticket Tracking System was down and that I could not be given any information at that time". Also, I was told the two persons whom I had been in contact with this issue previously were not there at the moment, please try to call back later". And you can bet that's exactly what I planned on doing!
I called back again from my cell phone at 1400 hrs (2 P.M.) and spoke directly with the technician I had spoke to on previous two days. He assigned a ticket number to me of 360811. I was also told that one of the domains, FrenchQuarterInn.com was now reachable and they were still working on RHPStudios.com. I was not given any information directly related to who was at fault, or what was at fault. I was away from the office and didn't have a phone line or eth hub to jack into to verify this statement. I returned to the office at approximately 1700 hrs (5 P.M.). To my amazement, there had been 0 successful packets sent to FrenchQuarterInn.com or RHP Studios.com in the last 4,583 sent (1 packet sent every minute to every domain RHP Studios is affiliated with, or responsible for the management of). This comes to over 76 hours that Maysville traffic has not been able to reach any Verio hosted web sites, including FrenchQuarterInn or RHPStudios. I was also told by HSA commercial support that others from the Maysville area were complaining about not being able to reach Verio hosted web sites. When I asked if it was complaints about RHP Studios or French Quarter Inn, I was told no, that the caller also had a web site hosted on Verio (mental note to find out who this is.....). This obviously means that others in the Maysville area have web sites hosted on Verio other than me! (Interested in banner exchanges??) At approximately 0130 hrs on Tuesday, February 26 I went to bed. There was still no successful packets sent to either address.
DAY 5 - On Tuesday, February 27 I called HSA Commercial Support at 12:30 P.M. I was told that nothing had been entered into the ticket since February 26 at 1130 A.M. The technician told me to call back after 2 PM so that I could speak directly to the technician handling the original call. I called back, but nothing more was available. I finished the night wondering what the hell was taking so long.
DAY 6 - Several calls were made today without any luck or new information.
DAY 7 - Its March 1, and visitors to some of my managed sites still have Valentines Day themes. How convenient! Can I update via shell accounts? Yes, but don't really want to risk the "openness" of the traffic, using insecure communications and leaving the web sites wide open to anyone running a packet sniffer along the way. Some good news: today at approximately 1 P.M. I was told that my issue was being assigned "Critical Status" to the HSA Commercial Support NOC. How amazing that it only took a week of constant bitching and complaining (yes, that is a little sarcasm in my voice). I was also told that I would be given an update before the commercial support office closed at midnight.......I waited until 0300 hrs, just in case they were in some other time zone. They didn't call back.
DAY 8 - Well, today is Friday and I have a new ad running in the paper. My phone has been ringing off the hook all week......do you know your web site is down? Yes, the issue started a week ago, and is being worked on by HSA Commercial Support. I'm sure they will have the problem resolved sometime in this millennium. How long does it take to drive from Maysville to Chicago? Wonder if Ma-Bell would let me work on their router? Today I decided I was going to call Jeff Craycraft again and make one more check up with HSA Commercial Support, but then I wasn't going to call anyone else over this issue. Jeff Craycraft responded, "You are the only one complaining, so it must not be an issue. I will have my guys check into it though and make sure." Do you want the support number, I asked? "No" he stated. I hung up the phone, didn't care for his tone of voice, didn't care for his attitude, oh wait...he's a tech guy too, executive even....would hate to be in his shoes and have his job......bet he doesn't even ping the IP addresses, I thought to myself. (Do you know how many times my phone has rung in the last week....do you know your web site is down? Hey, call HSA Commercial Support I would say, and here is their phone number......xxx-xxx-xxxx I would repeat off the top of my head).
I did make one other call this morning, I called French Quarter Inn whom I had been giving updates on the status of the web site traffic, although as you can tell from this message....I haven't been able to give them much to go on. Today I made one request to French Quarter Inn. My exact words were, "If you get a chance, after trying without any luck for 8 days to get the web traffic issue resolved, could you please call HSA Commercial Support at xxx-xxx-xxxx and also Jeff Craycraft at xxx-xxxx and complain that you are unable to reach your web sites on Verio.
At approximately 1 P.M. I called back to French Quarter Inn to see if this call had been made. I was told by the person answering the phone that she wasn't sure, but she didn't think so.
At or near 5 p.m., traffic was restored. I couldn't tell exact time because I wasn't there, I could only estimate time because of the 1 packet ping issued since the web sites went online showed about 23 minutes of successful pings at 5:18 P.M. Hurray!!!! Now I have 8 days of backed up work to complete in a few hours........
Saturday March 03, 2001 at approximately 0810 hrs I received a call from Jeff Craycraft, his tone of voice was much more pleasant today. "RHP Studios, this is Ed Wiget." "Mr. Wiget, traffic to your web site and French Quarter Inn has been restored. It was a problem on Verio." "Yes Jeff, I noticed it was working yesterday evening....and I appreciate you calling.".
Today I spend updating web sites, business is back to normal....and I can finally sleep a full night.