Random Announcements
-
say the same when i buy shirts & jeans & hats & t-shirts & silver from you
-
Onnly bother reading if you are interested in what happened last night…...
Hello Giles Padmore,
This is a short announcement in advance of a more detailed one later today in response to a MAJOR network problem experienced last night and into today. Everything is now back up and online which was our first priority though we are still working on some load issues related to the backup LAN not being fully functional yet.
We will now be carrying out a detailed investigation before sending out a full RFO (Reason for Outage) within 24 hours. In summary, during some scheduled maintenance most of our enclosures suffered a switch fault which appears to be related to a rogue firmware version that rendered it useless. This required us to replace all of the switches and restore the config to each port within the switch to restore the network connections to each rack. Unfortunately there was a domino effect across other servers too causing problems with our internal infrastructure (eg support portal), master DNS and email cluster.
Our helpdesk and other methods of communication are fully online and can be used if anyone has any questions. However, we do ask that questions related to the issues here are help until after the RFO as the information available at this time is limited. Throughout the incident incoming emails have been getting queued which is still getting worked through.
This has been a horrendous outage and is being treated with absolute priority and urgency. Further details will follow as previously mentioned.
Please do not reply directly to this email as it is sent from an unmonitored email account. To contact us please open a support ticket in the appropriate dept: http://www.clook.info/helpdesk/
Kind regards,
Clook Internet
-
Wow, sounds like the folks at Clook Internet were pooping bricks. I know just enough IT geek-speak to recognize that they had a nasty crash going on…
Sorry it crushed your morning, too, Giles.
-
so chris can you translate into laymen terms, or more appropriate monday mush so i can understand what the hell he said in that post?
-
My understanding is basically this:
they were doing standard upkeep (you know, changing the oil, replacing air filters, etc.) when one little black box crapped out. They were scrambling to fix that one, but it caused other things to go bad, which caused even more things to go bad, and the next thing you know, they were sitting in the center lane of the LIE with a blown engine and a 4 mile traffic jam behind them at 5pm on a Friday. -
Probably not.:)
I'm sure you were not alone, though.
This kind of thing isn't supposed to happen. That's why they pay IT nerds lots of money. An unexpected crash for a few minutes is one thing, but bombing the system for 10-12 hours is a huge sh!tstorm. There are going to be a lot of angry meetings at Clook over the next few days, and a couple of people might even be updating their CVs…
-
thanks for the explanation chris
