Welcome back

[Mike]

We were away for a few hours, due to some equipment failure at the datacenter where our servers are located. Apologies for the downtime, but this one was simply down to a failed Automatic Transfer Switch and all we could do was twiddle out thumbs until a replacement could be wired in.

The replacement is a refurbished switch and the datacenter has announced they are going to be wiring in a new switch, as soon as they can get one on site. So we may have a brief bit of downtime coming up in the next couple days.

 

[RPM]

WOW I was having bad withdrawals.

 

[jcunning]

I thought I had forgotten how to spell tbucketeers.

 

[Mike]

Ron, I sent you a couple e-mails from another account. I'm not sure if you saw those or not.

Bottom line was an ATS failed, at Colo4.

This was the last bit of news we had received from Colo4, shortly after 5:30 PM, Eastern -

Our team and electricians are working diligently to get the temporary ATS installed, wired and tested to allow power to be restored. As the ATS involves high-voltage power, we are following the necessary steps to ensure the safety of our personnel and your equipment housed in our facility.

Based on current progress the electricians expect to start powering the equipment on between 6:15 – 7:00pm Central. This is our best estimated time currently. We have thoroughly tested and don’t anticipate any issues in powering up, but there is always the potential for unforeseen issues that could affect the ETA so we will keep you posted as we get progress reports. Our UPS vendor has checked every UPS, and the HVAC has checked every unit and found no issues. Our electrical contractor has also checked everything.

We realize how challenging and frustrating that it has been to not have an ETA for you or your customers, but we wanted to ensure we shared accurate and realistic information. We are working as fast as possible to get our customers back online and to ensure it is done safely and accurately. We will provide an update again within the hour.
While the team is working on the fix, I’ve answered some of the questions or comments that have been raised:

1. ATSs are pieces of equipment and can fail as equipment sometimes does, which is why we do 2N power in the facility in case the worst case scenario happens.

2. There is no problem with the electrical grid in Dallas or the heat in Dallas that caused the issue.

3. Our website and one switch were connected to two PDUs, but ultimately the same service entrance. This was a mistake that has been corrected.

4. Bypassing an ATS is not a simple fix, like putting on jumper cables. It is detailed and hard work. Given the size and power of the ATS, the safety of our people and our contractors must remain the highest priority.

5. Our guys are working hard. While we all prepare for emergencies, it is still quite difficult when one is in effect. We could have done a better job keeping you informed. We know our customers are also stressed.

6. The ATS could be repaired, but we have already made the decision to order a replacement. This is certainly not the cheapest route to take, but is the best solution for the long-term stability.

7. While the solution we have implemented is technically a temporary fix, we are taking great care and wiring as if it were permanent.

8. Colo4 does have A/B power for our routing gear. We identified one switch that was connected to A only which was a mistake. It was quickly corrected earlier today but did affect service for a few customers.

9. Some customers with A/B power had overloaded their circuits, which is a separate and individual versus network issue. (For example, if we offer A/B 20 amp feeds and the customer has 12 amps on each, if one trips, the other will not be able to handle the load.)

As you could imagine, this is the top priority for everyone in our facility. We will provide an update as quickly as possible.


I had two choices. Relax and wait for things to get fixed, or wring my hands and throw temper tantrums, like hundreds of other Colo4 customers were doing. I grabbed an iced tea and my Kindle and spent the afternoon reading. :shrug: The important thing is we are back.

 

[RPM]

Ron, I sent you a couple e-mails from another account. I'm not sure if you saw those or not.
:shrug: The important thing is we are back.

Yes I got them................I still was missing this site.

 

[Mike]

At several times in the past, I've considered different ways to communicate to our users when this site is down.

I did submit a tweet to Twitter, when we were starting to get some firmed-up repair time ETAs. I really don't know how many of our users actually use Twitter, so I've no way of knowing if that tweet was useful or not. If you are so inclined, you can follow us on Twitter.

I really don't know that many of you are using Facebook. As a result, I've never bothered setting up a Facebook page.

During the downtime, eighteen of my e-mail accounts were also out of business. And who should I send e-mails to, never knowing who gives a rat's rectum and who doesn't. I still had seven addresses up and running, but I doubt most of you actually know my name and would delete the e-mail out of hand. Many of you would likely see the e-mail get shunted into a Junk folder and I already know how few of you ever check those. As a quick aside, When all of those eighteen e-mail accounts lit back up, I had over 200 messages to wade through, so apologies to anyone who found the wait for a response off-putting.

I've considered setting up a WordPress blog on a separate domain and a separate server in another datacenter, so I could use that site to keep people informed if problems arise here. But I find people never manage to see informational topics on this site. So if I were to post a topic suggesting people bookmark someothersite.com to check when this site is down, maybe three of you would read it and bookmark the site. The rest would be in the dark. It would require purchasing a new domain name to do that with and I think we're already spending enough money as it is.

I had Ron's e-mail address handy and sent him some news, figuring he would be wondering what was happening. I know he's a lot of contacts and figured if anyone questioned him, he could at least pass along a bit of information.

Other than that, I really don't know a good way to keep you informed, in the few times the site has been down. Fortunately, the site is rarely ever down, so it's not as if it is a constant and ongoing issue. Yesterday's downtime eclipsed our cumulative total of downtime over the last 40+ months. And it was also the first time the server had been down in some 380 - 385 days.

 

[lincolnuT]

Mike,

I don't see the problem here. I came over here several times during the day and I knew you work working on the site or it was down........... hthedrama: ......I'll bet no one had as bad a day as you had without the site fix I get every morning.

So stop beating yourself up over a little downtime................ :kiltdance: your back up and running..........

Mike

 

[Mike]

So stop beating yourself up over a little downtime.

It's more a case of wishing I could develop some means of communication to explain reasons for downtime, than anything else. That way, if you visited here and here wasn't here when you arrived here, you could at least go somewhere to find an explanation for here not being here, when you were expecting it would be here.

We did take a traffic hit during the downtime. Traffic was down 30% from our average, daily traffic. :shrug2: It is what it is, I can't go back and change it nor am I losing any sleep over it. There is a silver lining in the cloud, however. Even losing what we did, our average, daily numbers indicate August will be yet another record-setting month, by a projected 8%.