Global IT Outage

[Big DaveBig Dave is verified member.]

We're getting news here about some IT Outage that's going global. Affecting everything from banks, news organisations, supermarkets, casinos, airports, damn near everything. Apparently some update was pushed through that didn't comply and brought a blue "Recovery" screen to just about every linked Windows machine. Somebody's going to be in trouble. I haven't heard anything about the stock markets yet, but ours is already shut down for the day.

I do hope we're not going to run out of toilet paper again.

Sent from my motorola edge 20 fusion using Tapatalk

 

[Desertnate]

Looks like CloudStrike pushed a bad software update to Windows OS users. Not sure the update was on user machines on Win10/11, Server OS's or if it was a patch loaded by customers using Microsoft Azure cloud environment for app hosting. I haven't dug too deep into it.

From what I've seen, it has caused quite a bit of chaos, but an update fixing the problem has been released and stuff is starting to come back on line. I'm just glad I wasn't flying today. Looks like US airports were are a bit of a mess.

 

[Big DaveBig Dave is verified member.]

Yeah, lots of people standing around airports staring up at blue screens on the news here, also lots of abandoned shopping carts in supermarkets after people were told they have to pay by cash. Even the casinos were empty. They're saying globally the cost of this little outage will be $100 million plus.


Sent from my motorola edge 20 fusion using Tapatalk

 

[Karl_in_Chicago]

Pretty much anything with a dependence on M$ Azure - which, sadly, is nearly pretty much *everything* these days - has been impacted. I'm in Enterprise IT for a large financial svcs firm with a global presence (including Oz) and there's lots of stuff that's fubar'd. The benefits of outsourcing and "the cloud".

Come up to the Cloud, we'll get together, have a few laughs . . .
View attachment 77344

 

[Desertnate]

Pretty much anything with a dependence on M$ Azure - which, sadly, is nearly pretty much *everything* these days - has been impacted. I'm in Enterprise IT for a large financial svcs firm with a global presence (including Oz) and there's lots of stuff that's fubar'd. The benefits of outsourcing and "the cloud".

Come up to the Cloud, we'll get together, have a few laughs . . .
View attachment 77344

I'm in the government IT space and watching this with interest.

My oldest daughter is a supervisor in a major hospital and says the entire place is down right now. The worst part is that facility decided it would be a smart idea to put all of their IT service in the cloud as well, to include all voice/telephone, no local call managers at all. Not only do their IT apps not work, but their telephones are dead too, can't even make calls to other parts of the building either. Not sure who's bright idea that was.

No one ever things the "cloud" will go down, or the network that gets you to your cloud service...until it does.

 

[Karl_in_Chicago]

I'm in the government IT space and watching this with interest.

My oldest daughter is a supervisor in a major hospital and says the entire place is down right now. The worst part is that facility decided it would be a smart idea to put all of their IT service in the cloud as well, to include all voice/telephone, no local call managers at all. Not only do their IT apps not work, but their telephones are dead too, but can't even make calls to other parts of the building either. Not sure who's bright idea that was.

No one ever things the "cloud" will go down, or the network that gets you to your cloud service...until it does.

Yeah, I hear you. Even if a company's internal systems are on unaffected platforms it's connecting to and/or depending upon *other* companies that is still a potential point of failure. Just one example for my firm is that like most FinSvcs we have a large operation in Britain . . and British Telecom took it in the nuts. Building out their own telephone network obviously isn't a viable option so, oops, there it is. While regular review and scoring of service providers is not only a better practice but - in the case of "highly regulated" industries like finance, medical, etc. usually *required* - sometimes there just aren't any realistic options. In my case I always accept that absolutely worst case for me is that our failures might cost somebody money other firms might cost lives (hospitals, emergency services, . . .). Reading numerous reports of 911 systems being down, municipal police and fire systems being offline, etc. That's really bad. Of course I'm among the rapidly diminishing pool of mainframe dinosaurs so my systems are *relatively* immune (though *never* completely, complacence=arrogance) it's all the other itty-bitty systems that pull and push information from/to me that can make it irrelevant whether my systems are up or down. No point crowing about one's systems being up if nobody can get to them!

 

[opie2]

I'm in the government IT space and watching this with interest.

My oldest daughter is a supervisor in a major hospital and says the entire place is down right now. The worst part is that facility decided it would be a smart idea to put all of their IT service in the cloud as well, to include all voice/telephone, no local call managers at all. Not only do their IT apps not work, but their telephones are dead too, can't even make calls to other parts of the building either. Not sure who's bright idea that was.

No one ever things the "cloud" will go down, or the network that gets you to your cloud service...until it does.

What do you mean by putting all the stuff in the cloud? Does that mean like customer service stuff is all automated? Sorry im not an "IT" guy lol

Sent from my SM-S901U using Tapatalk

 

[Big DaveBig Dave is verified member.]

On the morning news just now, an IT Professor was saying the current fix is to go in and delete the latest update that downloaded, reset to the previous, and reboot, sounds simple, but it can't be done remotely, every workstation will have to be done individually. Some people are going to be really busy.

Sent from my motorola edge 20 fusion using Tapatalk

 

[Karl_in_Chicago]

On the morning news just now, an IT Professor was saying the current fix is to go in and delete the latest update that downloaded, reset to the previous, and reboot, sounds simple, but it can't be done remotely, every workstation will have to be done individually. Some people are going to be really busy.

Sent from my motorola edge 20 fusion using Tapatalk

Well he sounds like a frickin' moron. Like most of my coworkers I'm saddled with an employer-issued Windows PC (laptop, actually running Win 11). We didn't experience any desktop issues it was our suppliers/vendors/service providers and the Windows *servers* they use to deliver the contracted services (at least those that use CloudStrike, which is a LOT). It wasn't a M$ Windows update it was a *CloudStrike* update. None of us (me or people I work with) even HAVE CloudStrike on our machines so what he's saying isn't even possible or, obviously, an actual fix. Staff that use a service to complete or monitor trade settlements, for example, couldn't do so because even though their workstations were OK and the network connectivity was OK the end *service* was fubar'd.

There was an initial coupling of the CloudStrike outage with a M$ Azure outage but M$ has come back and said they were not related, it was just Azure being Azure.

 

[Big DaveBig Dave is verified member.]

Well he sounds like a frickin' moron. Like most of my coworkers I'm saddled with an employer-issued Windows PC (laptop, actually running Win 11). We didn't experience any desktop issues it was our suppliers/vendors/service providers and the Windows *servers* they use to deliver the contracted services (at least those that use CloudStrike, which is a LOT). It wasn't a M$ Windows update it was a *CloudStrike* update. None of us (me or people I work with) even HAVE CloudStrike on our machines so what he's saying isn't even possible or, obviously, an actual fix. Staff that use a service to complete or monitor trade settlements, for example, couldn't do so because even though their workstations were OK and the network connectivity was OK the end *service* was fubar'd.

There was an initial coupling of the CloudStrike outage with a M$ Azure outage but M$ has come back and said they were not related, it was just Azure being Azure.

Not saying anything, but it wasn't a he.

Sent from my motorola edge 20 fusion using Tapatalk

 

[Desertnate]

What do you mean by putting all the stuff in the cloud? Does that mean like customer service stuff is all automated? Sorry im not an "IT" guy lol

A joke I often hear is "cloud just means someone else's computer". Basically, instead of companies owning and running their own server farms to host their data and apps, they all pay companies like Amazon Web Services, Microsoft, Oracle, Google, and maybe a few others to do it for them. Works OK untill your data transport network goes down or your Could Provider has a big proplem and you at their mercy to restore service. It has it's advantages, but there are a couple places where a failure can leave you totally in the dark. Too many people don't plan properly for those failures and simply have faith (or naively believe) service will always be there. And then it isn't.....

Hopefully this outage and the massive cellular update on ATT's network a few months ago will have some organizations looking at how they do business and do a better job of disaster planning.

Well he sounds like a frickin' moron. Like most of my coworkers I'm saddled with an employer-issued Windows PC (laptop, actually running Win 11). We didn't experience any desktop issues it was our suppliers/vendors/service providers and the Windows *servers* they use to deliver the contracted services (at least those that use CloudStrike, which is a LOT). It wasn't a M$ Windows update it was a *CloudStrike* update. None of us (me or people I work with) even HAVE CloudStrike on our machines so what he's saying isn't even possible or, obviously, an actual fix. Staff that use a service to complete or monitor trade settlements, for example, couldn't do so because even though their workstations were OK and the network connectivity was OK the end *service* was fubar'd.

There was an initial coupling of the CloudStrike outage with a M$ Azure outage but M$ has come back and said they were not related, it was just Azure being Azure.

The professor was likely over-simplifying for a non-technical TV audience and did it to the point it became incorrect or misleading. I have to deal with this kind of situation all the time. At my daughter's hospital the updated patch was sent to their IT department and they had to go out and get rid of the old instance and replace it with the new one on the server side, wherever they were hosted. From what she was saying the IT department only had two people on staff who knew how to do this and it took them all day just to bring up the essential services to keep things running