ICG Link News

02.14.11

Valentines Day Outage

We are very sorry for the outage this morning. One of our systems administrators was working on a new system and when it came time to reboot it, he accidentally rebooted the entire live system. When the system came back, it didn't come back correctly and required a personal visit to the data center to correct the multiple problems. Since the main system was impacted, just about all other services were effected in some way and it took some time to get everything back on line again. We apologize for the error. There should be no further problems.

Regarding redundancy, the system that is currently in place was bleeding edge, state-of-the-art when we built it. It is fully redundant, but does require a manual intervention in the event of a catastrophic failure like we had today. Its replacement system with instant, hot redundancy is actually up and running and has been for several months as we have been ironing out any bugs. We only have to make the switch over to it which was planned for next weekend. An announcement about that will be forthcoming because the switch will require some down time in the middle of the night over the weekend.