Random Announcements
-
Onnly bother reading if you are interested in what happened last night…...
Hello Giles Padmore,
This is a short announcement in advance of a more detailed one later today in response to a MAJOR network problem experienced last night and into today. Everything is now back up and online which was our first priority though we are still working on some load issues related to the backup LAN not being fully functional yet.
We will now be carrying out a detailed investigation before sending out a full RFO (Reason for Outage) within 24 hours. In summary, during some scheduled maintenance most of our enclosures suffered a switch fault which appears to be related to a rogue firmware version that rendered it useless. This required us to replace all of the switches and restore the config to each port within the switch to restore the network connections to each rack. Unfortunately there was a domino effect across other servers too causing problems with our internal infrastructure (eg support portal), master DNS and email cluster.
Our helpdesk and other methods of communication are fully online and can be used if anyone has any questions. However, we do ask that questions related to the issues here are help until after the RFO as the information available at this time is limited. Throughout the incident incoming emails have been getting queued which is still getting worked through.
This has been a horrendous outage and is being treated with absolute priority and urgency. Further details will follow as previously mentioned.
Please do not reply directly to this email as it is sent from an unmonitored email account. To contact us please open a support ticket in the appropriate dept: http://www.clook.info/helpdesk/
Kind regards,
Clook Internet
-
Wow, sounds like the folks at Clook Internet were pooping bricks. I know just enough IT geek-speak to recognize that they had a nasty crash going on…
Sorry it crushed your morning, too, Giles.
-
so chris can you translate into laymen terms, or more appropriate monday mush so i can understand what the hell he said in that post?
-
My understanding is basically this:
they were doing standard upkeep (you know, changing the oil, replacing air filters, etc.) when one little black box crapped out. They were scrambling to fix that one, but it caused other things to go bad, which caused even more things to go bad, and the next thing you know, they were sitting in the center lane of the LIE with a blown engine and a 4 mile traffic jam behind them at 5pm on a Friday. -
Probably not.:)
I'm sure you were not alone, though.
This kind of thing isn't supposed to happen. That's why they pay IT nerds lots of money. An unexpected crash for a few minutes is one thing, but bombing the system for 10-12 hours is a huge sh!tstorm. There are going to be a lot of angry meetings at Clook over the next few days, and a couple of people might even be updating their CVs…
-
thanks for the explanation chris
-
These companies spent more money lobbying in the year prior, then they paid in taxes.. Crazy stuff
FedEx (0.9% effective tax rate)
Corning
DTE Energy
Honeywell International
Mattel
El Paso
Wells Fargo
Boeing
Navistar International
NextEra Energy
CMS Energy
Interpublic Group
Verizon Communications
Consolidated Edison
DuPoint
Duke Energy
Wisconsin Energy
Baxter International
Ryder System
Con-way
American Electric Power
Integrys Energy Group
Tenet Healthcare
CenterPoint Energy
NiSource
Computer Sciences
PG&E Corp
Paccar
General Electric
Pepco Holdings (-57.6% effective tax rate) -
a very good reason why i wear iron heart to work…. this happened recently at my job site:
