!umich!blackhole
August 29, 2023 8:04 AM Subscribe
The University of Michigan, a community of over 120,000 people, has gone offline in response to a security incident.
On Sunday afternoon the IT department removed the entire school from the Internet. It is still offline and will likely be so for days. Indications are the problem is serious. Classes started Monday and the university is trying to operate without networking or most computer systems.
On Sunday afternoon the IT department removed the entire school from the Internet. It is still offline and will likely be so for days. Indications are the problem is serious. Classes started Monday and the university is trying to operate without networking or most computer systems.
The second link has a line about students not knowing where their classes were, and I'm like "why don't they just look in the course catalog" and then I was like I got one when I was in school it was a paper book like 50 pages long and then I was like yeah that was in 1985.
posted by Gorgik at 8:25 AM on August 29, 2023 [79 favorites]
posted by Gorgik at 8:25 AM on August 29, 2023 [79 favorites]
Must be either ransomware or a huge data breach, given the university's info-free statements.
posted by ryanrs at 8:27 AM on August 29, 2023 [29 favorites]
posted by ryanrs at 8:27 AM on August 29, 2023 [29 favorites]
1000 points to Gryffindor for the post title. Bang paths FTW.
posted by hanov3r at 8:31 AM on August 29, 2023 [36 favorites]
posted by hanov3r at 8:31 AM on August 29, 2023 [36 favorites]
The way I registered for classes freshman year was to show up at an appointed time with a hundred or so fellow students in the foyer outside the biggest room in the student center and then they would unlock the doors and we would run to departmental tables to get harried professors to sign us into their class. The next year (at a state university) we stood in what felt like a passport control line and prayed the academic advising staff at the counter would be generous with signing us into classes. Senior year we got phone registration. It felt like we were living in the future.
Senior year=1997
posted by thivaia at 8:38 AM on August 29, 2023 [30 favorites]
Senior year=1997
posted by thivaia at 8:38 AM on August 29, 2023 [30 favorites]
@thivaia same. We had to line up to hand in our paper forms and because of my major I NEEDED to get certain classes/sections to make it all work so usually was out there with a coffee lined up at 6am till they opened at 8 each semester.
posted by Captain_Science at 8:42 AM on August 29, 2023 [2 favorites]
posted by Captain_Science at 8:42 AM on August 29, 2023 [2 favorites]
I haphazardly venture that if this were ordinary ransomware or a ransomwarish data stealer, we'd see services coming back online already, piece by piece.
Since we're kind of not? I'm going to hazard that this is an Equifax scenario -- they just figured out that an adversary's been rummaging around in their networked stuff for quite a long time, and it's taking them a while even to figure out the extent of it.
I flinched when I heard. I cannot IMAGINE the disruption. Knocking on all the wood that this doesn't happen in my shop, where classes start next week...
posted by humbug at 8:46 AM on August 29, 2023 [16 favorites]
Since we're kind of not? I'm going to hazard that this is an Equifax scenario -- they just figured out that an adversary's been rummaging around in their networked stuff for quite a long time, and it's taking them a while even to figure out the extent of it.
I flinched when I heard. I cannot IMAGINE the disruption. Knocking on all the wood that this doesn't happen in my shop, where classes start next week...
posted by humbug at 8:46 AM on August 29, 2023 [16 favorites]
Glad someone appreciated the bang path :-) I think this startled me because as an Internet old timer, umich.edu was such a big part of the Internet. Back in the 60s-80s the university (and MERIT) was a premier networking innovator, including being the principal behind NSFNet which built the academic Internet. IT departments today are far removed from the CS research projects of those days but for them to have to go fully offline is really striking.
I'm also curious about how the community manages coming back to campus without any Internet. It seems as big a problem as not having running water or electricity. 35 years ago not having Internet would barely have been noticed. 20 years ago it would have been awkward but not a big deal. 10 years ago it would have been a significant hardship. What's going to happen now? Possibly it won't be quite so bad, since cloud hosting and cellular Internet means not everything is dependent on the campus network. I'll note the campus website is up and being updated, presumably it's hosted elsewhere.
The last time I can think of a whole university going offline was in 1988, in response to the Morris Worm. Going dark was considered something of a mistake in retrospect because it meant the folks working on the problem couldn't easily communicate with their peers at other universities.
posted by Nelson at 8:46 AM on August 29, 2023 [24 favorites]
I'm also curious about how the community manages coming back to campus without any Internet. It seems as big a problem as not having running water or electricity. 35 years ago not having Internet would barely have been noticed. 20 years ago it would have been awkward but not a big deal. 10 years ago it would have been a significant hardship. What's going to happen now? Possibly it won't be quite so bad, since cloud hosting and cellular Internet means not everything is dependent on the campus network. I'll note the campus website is up and being updated, presumably it's hosted elsewhere.
The last time I can think of a whole university going offline was in 1988, in response to the Morris Worm. Going dark was considered something of a mistake in retrospect because it meant the folks working on the problem couldn't easily communicate with their peers at other universities.
posted by Nelson at 8:46 AM on August 29, 2023 [24 favorites]
The second link has a line about students not knowing where their classes were,
I often work the exams at my alma mater. A couple of years ago the university’s intranet crashed for an hour or two during exams, right around the beginning of one of the four daily slots for exam starts.
I was working that day in one of the big gymnasium complexes containing three or four sizeable gyms with a total capacity of around two thousand students for exams. As it happened, that slot saw the exam for a popular course (maybe first year psychology?) and all of the people in the building would be writing the same examination.
There was panic, frustration, and generally emotion raw and stark from students and many exam administrators — how would people know they were in the right place? They might be in the wrong section of a gym or even the wrong room oh noes!!!
I can see the confusion for the students, who were stressed and may not have noticed that every single exam for the course was scheduled for the sane time in the same building and who, like true Gen Z sorts, write nothing down but assume it will be online. However, l even the other invigilators who had been doing this for years were — in some cases — freaked out that someone might go to Gym #1 and write precisely the same exam as they were scheduled to write in gym #3 at the same time. It was hard to persuade these folks that we could let the students in to sit down and if there turned out to be more students than there were exams, we could just send them across the hall to that room.
posted by ricochet biscuit at 8:48 AM on August 29, 2023 [7 favorites]
I often work the exams at my alma mater. A couple of years ago the university’s intranet crashed for an hour or two during exams, right around the beginning of one of the four daily slots for exam starts.
I was working that day in one of the big gymnasium complexes containing three or four sizeable gyms with a total capacity of around two thousand students for exams. As it happened, that slot saw the exam for a popular course (maybe first year psychology?) and all of the people in the building would be writing the same examination.
There was panic, frustration, and generally emotion raw and stark from students and many exam administrators — how would people know they were in the right place? They might be in the wrong section of a gym or even the wrong room oh noes!!!
I can see the confusion for the students, who were stressed and may not have noticed that every single exam for the course was scheduled for the sane time in the same building and who, like true Gen Z sorts, write nothing down but assume it will be online. However, l even the other invigilators who had been doing this for years were — in some cases — freaked out that someone might go to Gym #1 and write precisely the same exam as they were scheduled to write in gym #3 at the same time. It was hard to persuade these folks that we could let the students in to sit down and if there turned out to be more students than there were exams, we could just send them across the hall to that room.
posted by ricochet biscuit at 8:48 AM on August 29, 2023 [7 favorites]
Mark my words, Brutus Buckeye is behind all of this.
posted by Abehammerb Lincoln at 8:55 AM on August 29, 2023 [21 favorites]
posted by Abehammerb Lincoln at 8:55 AM on August 29, 2023 [21 favorites]
I'm also curious about how the community manages coming back to campus without any Internet.
I think it's impossible now. My son just started his freshman year and told me he needs to juggle a half dozen cloud-based applications to manage his education. Blackboard for assignments/grades, Starfish for counselors. A web app for tracking financial aid and tuition payments. Another one for adding or dropping classes (probably a HUGE problem right now). FAFSA crap. Office/Outlook for calendaring and email. A housing app. A dining app. A campus security app. It's insane.
posted by JoeZydeco at 8:57 AM on August 29, 2023 [28 favorites]
I think it's impossible now. My son just started his freshman year and told me he needs to juggle a half dozen cloud-based applications to manage his education. Blackboard for assignments/grades, Starfish for counselors. A web app for tracking financial aid and tuition payments. Another one for adding or dropping classes (probably a HUGE problem right now). FAFSA crap. Office/Outlook for calendaring and email. A housing app. A dining app. A campus security app. It's insane.
posted by JoeZydeco at 8:57 AM on August 29, 2023 [28 favorites]
I bet Ohio State is having a blast with this.
On the other hand, what else do you expect when you run a football outfit and dabble in tertiary education. So what if you helped develop the internet? Prob best to shop all that pesky IT out to a zillion shitty vendors at the lowest bid. So I expect it could happen to any Big State U next. I'm so sick of seeing our universities turn into business-oriented platforms for fundraising instead of public institutions of learning.
posted by SaltySalticid at 8:58 AM on August 29, 2023 [20 favorites]
On the other hand, what else do you expect when you run a football outfit and dabble in tertiary education. So what if you helped develop the internet? Prob best to shop all that pesky IT out to a zillion shitty vendors at the lowest bid. So I expect it could happen to any Big State U next. I'm so sick of seeing our universities turn into business-oriented platforms for fundraising instead of public institutions of learning.
posted by SaltySalticid at 8:58 AM on August 29, 2023 [20 favorites]
wow. this is really insane. I work for a large U (doing web tech, lol!!!) and I cannot IMAGINE the level of disruption this would cause, as we head into the new semester.
its an unpleasant reminder, even to us at a distance, of how incredibly reliant upon the internet, social media etc., we have all become...
posted by supermedusa at 9:06 AM on August 29, 2023 [7 favorites]
its an unpleasant reminder, even to us at a distance, of how incredibly reliant upon the internet, social media etc., we have all become...
posted by supermedusa at 9:06 AM on August 29, 2023 [7 favorites]
If you think the students, faculty and admin at U of M are having a bad day, please spare a thought for the underpaid drones who work for the call centres that answer their phones.
This is the week when university and college call centres are absolutely slammed and every second person calling is having a panic attack and is channeling that into spewing venom. If there is any way the students and faculty can figure out the number they can call, such as by looking at an e-mail sent to them a week ago, there will be a tsunami of calls. When the internet fails, people revert back to the ancient practice of trying to find a real person to speak to. And when they are trying to reach a real person they don't care which number they call so long as they can get a human. So they will be calling the schools IT help desk looking for a someone who can find them a map of campus. They will be calling the student pastoral help line. They will be calling the toll free recruitment number. They will be sitting on the line in the queue of whatever call centre they think has the best chance of getting them to a real person, snowballing from patience to panic to fury.
And frequently the person call will be a parent who screams in outrage at the person who answers that "My stupid son/daughter didn't look up this information in advance! We need this information NOW!" They are really screaming at the faintly mumbling voice behind them.
But on the positive side for the call centre workers, only one out of hundred of the callers will wait it out in the queue long enough to find a real person to scream at. They won't be able to get to even a fraction of them.
posted by Jane the Brown at 9:07 AM on August 29, 2023 [31 favorites]
This is the week when university and college call centres are absolutely slammed and every second person calling is having a panic attack and is channeling that into spewing venom. If there is any way the students and faculty can figure out the number they can call, such as by looking at an e-mail sent to them a week ago, there will be a tsunami of calls. When the internet fails, people revert back to the ancient practice of trying to find a real person to speak to. And when they are trying to reach a real person they don't care which number they call so long as they can get a human. So they will be calling the schools IT help desk looking for a someone who can find them a map of campus. They will be calling the student pastoral help line. They will be calling the toll free recruitment number. They will be sitting on the line in the queue of whatever call centre they think has the best chance of getting them to a real person, snowballing from patience to panic to fury.
And frequently the person call will be a parent who screams in outrage at the person who answers that "My stupid son/daughter didn't look up this information in advance! We need this information NOW!" They are really screaming at the faintly mumbling voice behind them.
But on the positive side for the call centre workers, only one out of hundred of the callers will wait it out in the queue long enough to find a real person to scream at. They won't be able to get to even a fraction of them.
posted by Jane the Brown at 9:07 AM on August 29, 2023 [31 favorites]
Back in my day (85-89) we submitted our requested class forms, somebody typed them all into a mainframe terminal, and the computer spit out your schedule. If you didn't get into Calc 201 at noon on MWF it stuck you wherever it could based on your other classes, and that was usually the class sessions no reasonable person ever requested (730 AM MWF or 5 PM MWF.)
posted by COD at 9:08 AM on August 29, 2023 [6 favorites]
posted by COD at 9:08 AM on August 29, 2023 [6 favorites]
I think there might be an active MeFite who works at UMich IT. It would be interesting to get an inside story.
posted by slogger at 9:08 AM on August 29, 2023 [7 favorites]
posted by slogger at 9:08 AM on August 29, 2023 [7 favorites]
yes, but how would they post it?
posted by ryanrs at 9:10 AM on August 29, 2023 [49 favorites]
posted by ryanrs at 9:10 AM on August 29, 2023 [49 favorites]
dabble in tertiary education
Really? It's almost certainly one of the top public universities in the nation academically.
posted by Abehammerb Lincoln at 9:22 AM on August 29, 2023 [23 favorites]
Really? It's almost certainly one of the top public universities in the nation academically.
posted by Abehammerb Lincoln at 9:22 AM on August 29, 2023 [23 favorites]
Guessing they were still on a domain and hadn't gotten to zero-trust yet. Also guessing they are learning a lot about their backup strategy right now. I'm on team Verge.io + Veeam + Wasabi at the moment, and wish them the best of luck.
posted by furtive at 9:23 AM on August 29, 2023 [1 favorite]
posted by furtive at 9:23 AM on August 29, 2023 [1 favorite]
yes, but how would they post it?Presumably from home or a cellular connection. However, all of my contacts at U-M IT are saying that the U is being very tight-lipped about this, even internally. From my experience there, this is consistent with how I would expect them to behave during/after a serious security incident and not really consistent with any other scenarios I can think of.
posted by Juffo-Wup at 9:23 AM on August 29, 2023 [14 favorites]
Presumably they're also very very busy. What a nightmare.
Although this gives some hope, from the update link end of Monday:
cloud services such as Google, Canvas, Zoom, Adobe Cloud, Dropbox, Slack and other systems are now back online and reachable when using off-campus and cellular networks.
Of course doesn't help if students are in university housing and have no home internet as a result.
posted by nat at 9:26 AM on August 29, 2023
Although this gives some hope, from the update link end of Monday:
cloud services such as Google, Canvas, Zoom, Adobe Cloud, Dropbox, Slack and other systems are now back online and reachable when using off-campus and cellular networks.
Of course doesn't help if students are in university housing and have no home internet as a result.
posted by nat at 9:26 AM on August 29, 2023
Please know the ITS teams are working tirelessly to resolve this matter as quickly as possible.
Oh, I bet they're pretty tired. My heart goes out to their sysadmin and ops staff. Whatever is going on here, it's almost certainly the worst day on the job any of them can remember.
posted by mhoye at 9:27 AM on August 29, 2023 [24 favorites]
Oh, I bet they're pretty tired. My heart goes out to their sysadmin and ops staff. Whatever is going on here, it's almost certainly the worst day on the job any of them can remember.
posted by mhoye at 9:27 AM on August 29, 2023 [24 favorites]
Although this gives some hope, from the update link end of Monday:
cloud services such as Google, Canvas, Zoom, Adobe Cloud, Dropbox, Slack and other systems are now back online and reachable when using off-campus and cellular networks.
Yeah, everything that's not on their network :-(
posted by furtive at 9:29 AM on August 29, 2023 [6 favorites]
cloud services such as Google, Canvas, Zoom, Adobe Cloud, Dropbox, Slack and other systems are now back online and reachable when using off-campus and cellular networks.
Yeah, everything that's not on their network :-(
posted by furtive at 9:29 AM on August 29, 2023 [6 favorites]
cloud services such as Google, Canvas, Zoom, Adobe Cloud, Dropbox, Slack and other systems are now back online and reachable when using off-campus and cellular networks.
During the pandemic, the state of New Mexico set up free, robust internet services accessible from large parking lots around the state so that students could continue to take classes, thus by-passing the "no internet at home" issues. I think it should become SOP for the DR plans for lots of critical institutions.
posted by Silvery Fish at 9:30 AM on August 29, 2023 [20 favorites]
During the pandemic, the state of New Mexico set up free, robust internet services accessible from large parking lots around the state so that students could continue to take classes, thus by-passing the "no internet at home" issues. I think it should become SOP for the DR plans for lots of critical institutions.
posted by Silvery Fish at 9:30 AM on August 29, 2023 [20 favorites]
UMich poached our excellent University Librarian a year or so ago.
Not gonna lie, I have been thinking "shouldn't have left us, Lisa."
posted by humbug at 9:30 AM on August 29, 2023 [18 favorites]
Not gonna lie, I have been thinking "shouldn't have left us, Lisa."
posted by humbug at 9:30 AM on August 29, 2023 [18 favorites]
I think this startled me because as an Internet old timer, umich.edu was such a big part of the Internet.
#samesies
I have an Internet friend I've "known" for just about 3 decades who I first "met" on USENET when she was support@umich.
posted by hanov3r at 9:30 AM on August 29, 2023 [8 favorites]
#samesies
I have an Internet friend I've "known" for just about 3 decades who I first "met" on USENET when she was support@umich.
posted by hanov3r at 9:30 AM on August 29, 2023 [8 favorites]
On the campus subreddit, an infosec-savvy bingo card.
posted by humbug at 9:31 AM on August 29, 2023 [10 favorites]
posted by humbug at 9:31 AM on August 29, 2023 [10 favorites]
Yeah, everything that's not on their network :-(And yet this is still an improvement over the situation earlier. I have a UMich-branded Google account which I was unable to access (from home) yesterday morning. While I was able to reach Google, Google was not able to reach the University's identity and access management servers and thus couldn't determine whether it should allow me to log in or not.
I don't know the details on how the auth flow for U-M's (many) other cloud services work, but I suspect many of them do rely on being able to reach the university's IAM servers.
posted by Juffo-Wup at 9:32 AM on August 29, 2023 [11 favorites]
I think there might be an active MeFite who works at UMich IT. It would be interesting to get an inside story.
Not if they enjoy being employed.
posted by meowzilla at 9:33 AM on August 29, 2023 [39 favorites]
Not if they enjoy being employed.
posted by meowzilla at 9:33 AM on August 29, 2023 [39 favorites]
Did an incoming fresher open up an email with the Subject line “Join the party!”
They warned me about that back in my day.
posted by The Pluto Gangsta at 9:35 AM on August 29, 2023 [2 favorites]
They warned me about that back in my day.
posted by The Pluto Gangsta at 9:35 AM on August 29, 2023 [2 favorites]
I agree that network connectivity is now a utility like power or water, at the same time I can remember a few times having college classes during campus-wide power outages so I'm not at all surprised that they are continuing.
Anyway the idea of going to the first day of class not knowing my schedule or where any of my classes are is so adjacent to a common nightmare of mine (where this occured during finals week) that I'm going to wrap myself in a blanket burrito and remind myself that it's been nearly 20 years since I graduated.
posted by muddgirl at 9:38 AM on August 29, 2023 [14 favorites]
Anyway the idea of going to the first day of class not knowing my schedule or where any of my classes are is so adjacent to a common nightmare of mine (where this occured during finals week) that I'm going to wrap myself in a blanket burrito and remind myself that it's been nearly 20 years since I graduated.
posted by muddgirl at 9:38 AM on August 29, 2023 [14 favorites]
I work at the other U of M in the Big Ten, and I'm sure most staff in IT also know a big fat nothing. Everything gets super tight lipped around security incidents. Not saying that's good or bad, but it sure hurts our relationships around campus. Sending love from one U of M to the other.
posted by advicepig at 9:41 AM on August 29, 2023 [14 favorites]
posted by advicepig at 9:41 AM on August 29, 2023 [14 favorites]
Plus, end of month means paychecks are due, so it's not only classes that are disrupted.
I worked at UMich for a couple of years a while back. There was a weird day when some malware was attached to a mandatory Windows update, or something along those lines, and it wrecked stuff for a day, but was sorted out relatively quickly. This seems much worse.
posted by wicked_sassy at 9:41 AM on August 29, 2023 [11 favorites]
I worked at UMich for a couple of years a while back. There was a weird day when some malware was attached to a mandatory Windows update, or something along those lines, and it wrecked stuff for a day, but was sorted out relatively quickly. This seems much worse.
posted by wicked_sassy at 9:41 AM on August 29, 2023 [11 favorites]
dabble in tertiary education... one of the top public universities in the nation academically.
Those two statements are not contradictory. If the were interested in scholarship they'd pay pay top dollar for top scholars. Instead they pay the most (by a loooooong shot) to the sportsball coach. That is what they prioritize. Education is a sideline, sports are the big show that brings in the big money.
Maybe you missed the memo. If your institution's top paycheck goes to a sports guy, they are a sports institution. Yes I understand this applies to many places that still manage to teach a few classes and do some nice research. Yes I have spent over 25 years in Big State U and I am a professional researcher and scholar who does know what they are talking about. No I will not be addressing further complaints about this framing.
posted by SaltySalticid at 9:47 AM on August 29, 2023 [44 favorites]
Those two statements are not contradictory. If the were interested in scholarship they'd pay pay top dollar for top scholars. Instead they pay the most (by a loooooong shot) to the sportsball coach. That is what they prioritize. Education is a sideline, sports are the big show that brings in the big money.
Maybe you missed the memo. If your institution's top paycheck goes to a sports guy, they are a sports institution. Yes I understand this applies to many places that still manage to teach a few classes and do some nice research. Yes I have spent over 25 years in Big State U and I am a professional researcher and scholar who does know what they are talking about. No I will not be addressing further complaints about this framing.
posted by SaltySalticid at 9:47 AM on August 29, 2023 [44 favorites]
I would guess that monthly payroll got sent to the payroll processing company last Thursday or Friday, since they generally require some turn-around time. Hopefully. Likely to impact biweekly payroll though.
posted by eviemath at 9:48 AM on August 29, 2023
posted by eviemath at 9:48 AM on August 29, 2023
housing app. A dining app. A campus security app. It's insane.
My high-schooler texted me this morning (first full day of classes): if you see me ask for an app called "smartpass" it's something i need for school
It's for requesting and approving and monitoring hall passes. Literally can't go to the bathroom without it being on the internets.
posted by Etrigan at 9:53 AM on August 29, 2023 [20 favorites]
My high-schooler texted me this morning (first full day of classes): if you see me ask for an app called "smartpass" it's something i need for school
It's for requesting and approving and monitoring hall passes. Literally can't go to the bathroom without it being on the internets.
posted by Etrigan at 9:53 AM on August 29, 2023 [20 favorites]
This reminds me of a fall semester in the early 2000s when a lot of colleges and universities experienced severe network problems when students returned to campus and their Windows computers became infected with a worm. This was before Windows was set up to automatically update on personally-owned computers by default so many people didn't update their computers or ignored the warning messages. I worked in network support for the residence halls. Just a few hours after the buildings were opened, we disconnected them from the Internet and the rest of campus; it was a moot decision at that point because the network switches and routers were completely overwhelmed by the traffic being generated by all of the infected computers. We decided burn a whole lot of CDs with the appropriate Windows patch and we printed a checklist for each residence hall floor or apartment complex. Before we would try to restore connectivity to a building or area, the resident assistants (undergraduate students who worked for the university in each residence hall or apartment complex) had to check off a substantial percentage of the students on their checklist indicating that the resident had installed the patch or did not need it (used a Mac, didn't have a computer, etc.).
It was a rough couple of days with a lot of grumpy students and many grumpy parents. But at least we knew what we were dealing with and had a good plan, both of which we could freely explain to everyone. To have to make the same decision on a much larger scale without being able to explain anything - ouch! Sending good thoughts and vibes to colleagues in Ann Arbor!
posted by ElKevbo at 10:03 AM on August 29, 2023 [9 favorites]
It was a rough couple of days with a lot of grumpy students and many grumpy parents. But at least we knew what we were dealing with and had a good plan, both of which we could freely explain to everyone. To have to make the same decision on a much larger scale without being able to explain anything - ouch! Sending good thoughts and vibes to colleagues in Ann Arbor!
posted by ElKevbo at 10:03 AM on August 29, 2023 [9 favorites]
~My high-schooler texted me this morning (first full day of classes): if you see me ask for an app called "smartpass" it's something i need for school
~It's for requesting and approving and monitoring hall passes. Literally can't go to the bathroom without it being on the internets.
To attend the upcoming Special Persons’ Day (aka Grandparents’ Day) at our granddaughter’s kindergarten, we had to pay to have background checks run on both of us. Once approved, we were issued e-passes (individually barcoded and with photo) through a site called SafeVisitor.
posted by Thorzdad at 10:18 AM on August 29, 2023 [7 favorites]
~It's for requesting and approving and monitoring hall passes. Literally can't go to the bathroom without it being on the internets.
To attend the upcoming Special Persons’ Day (aka Grandparents’ Day) at our granddaughter’s kindergarten, we had to pay to have background checks run on both of us. Once approved, we were issued e-passes (individually barcoded and with photo) through a site called SafeVisitor.
posted by Thorzdad at 10:18 AM on August 29, 2023 [7 favorites]
I hate every word of that comment after "kindergarten"
posted by gwint at 10:20 AM on August 29, 2023 [74 favorites]
posted by gwint at 10:20 AM on August 29, 2023 [74 favorites]
Prob best to shop all that pesky IT out to a zillion shitty vendors at the lowest bid.
I guarantee you the University of Michigan's IT is in-house.
If your institution's top paycheck goes to a sports guy, they are a sports institution.
Eh, even if they pay the football coach the most money, they're more likely a healthcare institution.
posted by mr_roboto at 10:22 AM on August 29, 2023 [7 favorites]
I guarantee you the University of Michigan's IT is in-house.
If your institution's top paycheck goes to a sports guy, they are a sports institution.
Eh, even if they pay the football coach the most money, they're more likely a healthcare institution.
posted by mr_roboto at 10:22 AM on August 29, 2023 [7 favorites]
@eviemath: "I would guess that monthly payroll got sent to the payroll processing company last Thursday or Friday, since they generally require some turn-around time. Hopefully. Likely to impact biweekly payroll though."
Heard from a friend of mine who works there (and has no more idea than we do WTF happened) that raise notifications went out on schedule. Here's hoping that means payroll's gonna be okay too.
posted by humbug at 10:24 AM on August 29, 2023 [1 favorite]
Heard from a friend of mine who works there (and has no more idea than we do WTF happened) that raise notifications went out on schedule. Here's hoping that means payroll's gonna be okay too.
posted by humbug at 10:24 AM on August 29, 2023 [1 favorite]
On a scale from 1 to 100, how bad is this shutdown.
posted by clavdivs at 10:37 AM on August 29, 2023 [3 favorites]
posted by clavdivs at 10:37 AM on August 29, 2023 [3 favorites]
To attend the upcoming Special Persons’ Day (aka Grandparents’ Day) at our granddaughter’s kindergarten, we had to pay to have background checks run on both of us. Once approved, we were issued e-passes (individually barcoded and with photo) through a site called SafeVisitor.
I hate every word of that comment after "kindergarten"
I absolutely love "background checks run on both of us." You SHOULD have to have a background check run to spend time with small children who aren't your own/related to you in some way. And even then, sometimes background checks might be a good idea.
posted by cooker girl at 10:38 AM on August 29, 2023 [2 favorites]
I hate every word of that comment after "kindergarten"
I absolutely love "background checks run on both of us." You SHOULD have to have a background check run to spend time with small children who aren't your own/related to you in some way. And even then, sometimes background checks might be a good idea.
posted by cooker girl at 10:38 AM on August 29, 2023 [2 favorites]
The last time I can think of a whole university going offline was in 1988, in response to the Morris Worm. Going dark was considered something of a mistake in retrospect because it meant the folks working on the problem couldn't easily communicate with their peers at other universities.
The other one I can remember that comes close is the ILOVEYOU/Melissa VBS worm.
I was on the front lines working in IT and support at a major University of California campus when the ILOVEYOU/Melissa email worm hit and it damn near took us completely offline, and that campus had some (at the time) very serious infrastructure and was part of the Abilene ("Internet 2") new backbone test network.
The campus was so IT driven for that era we were a test bed for new tech like gigabit ethernet, Lucent/Orinoco WiFi, some of the first WiFi mesh networks in the US and more. As I recall at the time there was only about a dozen other campuses and sites with multi-gigabit WAN pipes to other campuses and sites.
it wasn't a complete shut down because that was targeted at Outlook servers, services and clients but the traffic was so intense that admins couldn't even reliably log in to any known Outlook servers to turn them off or change settings to block hidden VBS attachments/extensions in the emails.
And getting the word out to all of the admins via phone was nigh impossible and took forever. We even had to go around and put up signs all over the place begging and pleading for students or staff to not even try checking their emails or not even click on any of the emails with the ILOVEYOU title and so on.
They had to go around and physically kill power to any/all Outlook servers and pull physical network cables and connections, which was a huge pain in the ass because there were Outlook servers all over the campus in individual departments and sub-colleges, and the traffic from the worm was so insane that even plain old Unix/Linux/GNU infrastructure was unreliable even when trying to telnet/SSH into something.
And it was almost impossible to stay ahead of ILOVEYOU because all it took was one person getting the email and executing the VBS attachment and spamming everyone in their contact list and restarting the whole infection all over again.
That whole situation was just incredible. As I recall they basically had to take every single Outlook server offline for something like 24-48 hours and physically go around to every Outlook server to apply the patches locally with a laptop with the Outlook server airgapped, apply the patches, filter the VBS attachment option, restore the servers with filtered/modified backups from tapes and add firewall rules to block anything VBS related to be extra-extra sure ILOVEYOU couldn't reinfect the whole campus.
And part of the problem is that there were staff and students on campus that also were on Outlook servers on other campuses, so even when our servers were downed and dark they'd be getting the ILOVEYOU emails from outside Outlook servers which would then spam their whole contact list and re-infect any missed Outlook servers and mailing/contact lists and it just kept going and going.
One thing I clearly remember was how many admins were running around like they were on fire and how as soon as they thought they had it under control it would pop up again and discover yet another unpatched Outlook server.
It took something like over 72 hours to get back to anything resembling normal traffic.
I can't even imagine what they're dealing with right now at UM to have to go completely dark at this later stage of internet as a commodity infrastructure. That's some serious shit.
posted by loquacious at 10:47 AM on August 29, 2023 [25 favorites]
The other one I can remember that comes close is the ILOVEYOU/Melissa VBS worm.
I was on the front lines working in IT and support at a major University of California campus when the ILOVEYOU/Melissa email worm hit and it damn near took us completely offline, and that campus had some (at the time) very serious infrastructure and was part of the Abilene ("Internet 2") new backbone test network.
The campus was so IT driven for that era we were a test bed for new tech like gigabit ethernet, Lucent/Orinoco WiFi, some of the first WiFi mesh networks in the US and more. As I recall at the time there was only about a dozen other campuses and sites with multi-gigabit WAN pipes to other campuses and sites.
it wasn't a complete shut down because that was targeted at Outlook servers, services and clients but the traffic was so intense that admins couldn't even reliably log in to any known Outlook servers to turn them off or change settings to block hidden VBS attachments/extensions in the emails.
And getting the word out to all of the admins via phone was nigh impossible and took forever. We even had to go around and put up signs all over the place begging and pleading for students or staff to not even try checking their emails or not even click on any of the emails with the ILOVEYOU title and so on.
They had to go around and physically kill power to any/all Outlook servers and pull physical network cables and connections, which was a huge pain in the ass because there were Outlook servers all over the campus in individual departments and sub-colleges, and the traffic from the worm was so insane that even plain old Unix/Linux/GNU infrastructure was unreliable even when trying to telnet/SSH into something.
And it was almost impossible to stay ahead of ILOVEYOU because all it took was one person getting the email and executing the VBS attachment and spamming everyone in their contact list and restarting the whole infection all over again.
That whole situation was just incredible. As I recall they basically had to take every single Outlook server offline for something like 24-48 hours and physically go around to every Outlook server to apply the patches locally with a laptop with the Outlook server airgapped, apply the patches, filter the VBS attachment option, restore the servers with filtered/modified backups from tapes and add firewall rules to block anything VBS related to be extra-extra sure ILOVEYOU couldn't reinfect the whole campus.
And part of the problem is that there were staff and students on campus that also were on Outlook servers on other campuses, so even when our servers were downed and dark they'd be getting the ILOVEYOU emails from outside Outlook servers which would then spam their whole contact list and re-infect any missed Outlook servers and mailing/contact lists and it just kept going and going.
One thing I clearly remember was how many admins were running around like they were on fire and how as soon as they thought they had it under control it would pop up again and discover yet another unpatched Outlook server.
It took something like over 72 hours to get back to anything resembling normal traffic.
I can't even imagine what they're dealing with right now at UM to have to go completely dark at this later stage of internet as a commodity infrastructure. That's some serious shit.
posted by loquacious at 10:47 AM on August 29, 2023 [25 favorites]
Libraries at UMich are currently almost completely inoperable; libraries have online catalogs, and Interlibrary Loan which requires connectivity, and all sorts of databases that require authentication. Here's an email that went out to the ILL community just now:
Hi all,
TL;DR: we are non-lender until some unspecified future time. Most of this is just anecdotes to boggle at.
EYM has had a massive IT disaster (perhaps not how they'd like me to phrase it, but OH WELL), that totally took out our first day of classes.
Yesterday we couldn't even log in to email to let folks know what was going on, and while some communication was possible if you'd managed to stay logged in over the weekend and off campus, anyone onsite (basically all of our team) was operating in a dark void.
We've gone non-lender for now, as the library in general and ILLiad in particular are taking a back seat to things like restoring distribution of financial aid and letting students actually find where their classes are being held. We currently have no real ETA on resumption of activities. We've been "severed" since Sunday.
I know you would figure it out when you saw us lowercase, but just in case you're waiting on some communication from us, we are basically dead in the water. I was running the first-day-of-classes escape rooms yesterday (because what else does a modern library do to say "welcome!" but lock you in a room?), and the sheer number of bewildered first-year students who couldn't comprehend that books cannot be located through mind-powers was staggering. One person very earnestly told me that someone had haphazardly shelved all the language books, for none of the authors were in alphabetical order.
Yours in disaster,
posted by rabbitrabbit at 10:58 AM on August 29, 2023 [67 favorites]
Hi all,
TL;DR: we are non-lender until some unspecified future time. Most of this is just anecdotes to boggle at.
EYM has had a massive IT disaster (perhaps not how they'd like me to phrase it, but OH WELL), that totally took out our first day of classes.
Yesterday we couldn't even log in to email to let folks know what was going on, and while some communication was possible if you'd managed to stay logged in over the weekend and off campus, anyone onsite (basically all of our team) was operating in a dark void.
We've gone non-lender for now, as the library in general and ILLiad in particular are taking a back seat to things like restoring distribution of financial aid and letting students actually find where their classes are being held. We currently have no real ETA on resumption of activities. We've been "severed" since Sunday.
I know you would figure it out when you saw us lowercase, but just in case you're waiting on some communication from us, we are basically dead in the water. I was running the first-day-of-classes escape rooms yesterday (because what else does a modern library do to say "welcome!" but lock you in a room?), and the sheer number of bewildered first-year students who couldn't comprehend that books cannot be located through mind-powers was staggering. One person very earnestly told me that someone had haphazardly shelved all the language books, for none of the authors were in alphabetical order.
Yours in disaster,
posted by rabbitrabbit at 10:58 AM on August 29, 2023 [67 favorites]
Been poking around recent reports on attacker dwell times -- the amount of time an attacker stays in a network once they've compromised it, before they get noticed and kicked out.
They appear to be shrinking overall, but a just-released Sophos report offered this intriguing tidbit (emphasis mine):
"In contrast, the median dwell time for all non-ransomware incidents has risen slightly from 11 days to 13 days. So, the ransomware gangs are moving faster than ever, but all other network intrusions, where the motivations and end goals aren’t always clear, tend to linger."
posted by humbug at 11:02 AM on August 29, 2023 [4 favorites]
They appear to be shrinking overall, but a just-released Sophos report offered this intriguing tidbit (emphasis mine):
"In contrast, the median dwell time for all non-ransomware incidents has risen slightly from 11 days to 13 days. So, the ransomware gangs are moving faster than ever, but all other network intrusions, where the motivations and end goals aren’t always clear, tend to linger."
posted by humbug at 11:02 AM on August 29, 2023 [4 favorites]
Prob best to shop all that pesky IT out to a zillion shitty vendors at the lowest bid.
I guarantee you the University of Michigan's IT is in-house.
Then what exactly does Merit do?
I guarantee you the University of Michigan's IT is in-house.
Then what exactly does Merit do?
Merit’s mission is “connecting organizations and building community.” We provide network, security and community services to member organizations that help make our society a better place to learn, discover, work and live – while upholding the principles of an open internet.posted by pwnguin at 11:04 AM on August 29, 2023 [2 favorites]
Then what exactly does Merit do?
Wow. That looks incredibly cool; I haven't heard of Merit before. Definitely not an IT vendor; it looks more like an internet 1.0 educational consortium that has morphed into a public network services/research org. The Moonshot approach to broadening access to public networks seems really promising; I don't know much about the field and I wonder how novel it is.
posted by mr_roboto at 11:11 AM on August 29, 2023 [3 favorites]
Wow. That looks incredibly cool; I haven't heard of Merit before. Definitely not an IT vendor; it looks more like an internet 1.0 educational consortium that has morphed into a public network services/research org. The Moonshot approach to broadening access to public networks seems really promising; I don't know much about the field and I wonder how novel it is.
posted by mr_roboto at 11:11 AM on August 29, 2023 [3 favorites]
Merit has been running netops for Michigan schools since 1966. It's about as far from "shitty vendor at the lowest bid" as it gets. It invented a lot of the Internet technology we use today.
posted by Nelson at 11:13 AM on August 29, 2023 [25 favorites]
posted by Nelson at 11:13 AM on August 29, 2023 [25 favorites]
I haven't hear of Merit before. Definitely not an IT vendor;
My introduction to them was when Kansas State Uni's IT dept outsourced Zimbra hosting to them and we had to rework various account management middleware.
posted by pwnguin at 11:14 AM on August 29, 2023
My introduction to them was when Kansas State Uni's IT dept outsourced Zimbra hosting to them and we had to rework various account management middleware.
posted by pwnguin at 11:14 AM on August 29, 2023
first-day-of-classes escape rooms
Is this a euphemism or a jokey way to refer to something else? That's a very insidery message, so I wasn't sure.
("EYM" is the Hatcher Graduate Library at UM. I think.)
posted by 41swans at 11:20 AM on August 29, 2023 [2 favorites]
Is this a euphemism or a jokey way to refer to something else? That's a very insidery message, so I wasn't sure.
("EYM" is the Hatcher Graduate Library at UM. I think.)
posted by 41swans at 11:20 AM on August 29, 2023 [2 favorites]
Our library does escape rooms where students solve puzzles about using and searching library resources. Basically, gamifying library instruction. Watching this situation with horror and profound sympathy for the poor staff and librarians at UMich. There but for the grace of God...
posted by indexy at 11:26 AM on August 29, 2023 [16 favorites]
posted by indexy at 11:26 AM on August 29, 2023 [16 favorites]
Universities and school systems are particularly vulnerable to cyberattacks due to their schedule.
During the summer break, many computers are shut down, because they're not in use and people want to save energy or the department is being physically moved or other reasons. This moves them out of the normal maintenance cycle of getting regular patches and auditing and configuration. There are tools to remotely turn on and maintain these systems, but for a variety of reasons they may not work - like the desktop has been physically unplugged from the wall, or the laptop was stored in a staff member's home and can't connect to the school network. And IT always has bigger problems than chasing down that one computer that no one can find and may not even exist.
Then the school year starts, all these computers are turned back on, and they're out of phase with all the other systems, with multiple unpatched vulnerabilities with exploits in the wild, a huge influx of new students and staff with new hardware, staff members who forgot their password, vendors who changed stuff back in July but no one noticed, people demanding that computers be fixed right now because next week is the first day of classes. This is in addition to all the usual problems that plague large bureaucratic organizations.
posted by meowzilla at 11:30 AM on August 29, 2023 [11 favorites]
During the summer break, many computers are shut down, because they're not in use and people want to save energy or the department is being physically moved or other reasons. This moves them out of the normal maintenance cycle of getting regular patches and auditing and configuration. There are tools to remotely turn on and maintain these systems, but for a variety of reasons they may not work - like the desktop has been physically unplugged from the wall, or the laptop was stored in a staff member's home and can't connect to the school network. And IT always has bigger problems than chasing down that one computer that no one can find and may not even exist.
Then the school year starts, all these computers are turned back on, and they're out of phase with all the other systems, with multiple unpatched vulnerabilities with exploits in the wild, a huge influx of new students and staff with new hardware, staff members who forgot their password, vendors who changed stuff back in July but no one noticed, people demanding that computers be fixed right now because next week is the first day of classes. This is in addition to all the usual problems that plague large bureaucratic organizations.
posted by meowzilla at 11:30 AM on August 29, 2023 [11 favorites]
20-odd years ago, the place I worked at got hit by SQL Slammer. None of the IT managed servers were vulnerable, but a lot of developers had SQL Server installed locally. SQL Server at the time was a pain to patch. There were 4 different patches to fix the vulnerability, and figuring out which of the 4 you needed was a dark art. I'd patch by uninstalling, downloading a patched version from MS, and reinstalling.
Anyway, we got hit bad, and some number of machines were spewing enough packets randomly that the live systems were effected, and IT cut the programming floor off from the rest of the organization. Meanwhile, to combat the virus, they sent out emails on how to combat the virus. They posted fliers by the elevators on how to get rid of it. They even made announcements on the floor. This was over a week, during which we basically sat there useless.
Eventually, they made floppies, and gave them out to a couple of the larger programming teams. We then went down each row of programmers, and personally ran the fix on every machine. Machine locked? Someone would wait for its owner to return. That fixed the problem.
Afterwards, they called everyone on the floor together for a couple of announcements. First, if you went through channels to get a local copy of SQL server, it'd get patched automagically. Second, if they had to do that again, the offending machines would just get hauled off and reimaged. If you had important stuff on there, so sorry.
That's my big experience with a virus.
posted by Spike Glee at 11:31 AM on August 29, 2023 [3 favorites]
Anyway, we got hit bad, and some number of machines were spewing enough packets randomly that the live systems were effected, and IT cut the programming floor off from the rest of the organization. Meanwhile, to combat the virus, they sent out emails on how to combat the virus. They posted fliers by the elevators on how to get rid of it. They even made announcements on the floor. This was over a week, during which we basically sat there useless.
Eventually, they made floppies, and gave them out to a couple of the larger programming teams. We then went down each row of programmers, and personally ran the fix on every machine. Machine locked? Someone would wait for its owner to return. That fixed the problem.
Afterwards, they called everyone on the floor together for a couple of announcements. First, if you went through channels to get a local copy of SQL server, it'd get patched automagically. Second, if they had to do that again, the offending machines would just get hauled off and reimaged. If you had important stuff on there, so sorry.
That's my big experience with a virus.
posted by Spike Glee at 11:31 AM on August 29, 2023 [3 favorites]
Apparently there are bananas in the Ann Arbor break room.
Jokes aside, I am in IT at a mid-size university and I almost sharted when I saw these headlines. Godspeed, you sysadmins; may your firewalls be tight and your restores error-free.
posted by wenestvedt at 11:35 AM on August 29, 2023 [33 favorites]
Jokes aside, I am in IT at a mid-size university and I almost sharted when I saw these headlines. Godspeed, you sysadmins; may your firewalls be tight and your restores error-free.
posted by wenestvedt at 11:35 AM on August 29, 2023 [33 favorites]
first-day-of-classes escape rooms
Is this a euphemism or a jokey way to refer to something else? That's a very insidery message, so I wasn't sure.
No no we literally do escape rooms and all kinds of other activities. Libraries can be fun!
Oh and EYM is simply UMich’s Interlibrary Loan code. Everyone who participates in ILL has one.
posted by rabbitrabbit at 11:39 AM on August 29, 2023 [11 favorites]
Is this a euphemism or a jokey way to refer to something else? That's a very insidery message, so I wasn't sure.
No no we literally do escape rooms and all kinds of other activities. Libraries can be fun!
Oh and EYM is simply UMich’s Interlibrary Loan code. Everyone who participates in ILL has one.
posted by rabbitrabbit at 11:39 AM on August 29, 2023 [11 favorites]
Yeah, UMich are Internet gray-beards, institutionally speaking. Typical SaaS outsourcing aside, I would trust them to know their shit better than most other schools.
(And that bang path in the title was some welcome levity on a day that has me contemplating our DR plan.)
posted by wenestvedt at 12:15 PM on August 29, 2023 [9 favorites]
(And that bang path in the title was some welcome levity on a day that has me contemplating our DR plan.)
posted by wenestvedt at 12:15 PM on August 29, 2023 [9 favorites]
I thought the standard response to ransomware attacks was “restore from backup like any other loss scenario, if you can’t, your admin is incompetent and should be fired”. I haven’t seen the Slashdot reaction, but I can’t imagine it’s far from the truth.
posted by dr_dank at 12:24 PM on August 29, 2023 [1 favorite]
posted by dr_dank at 12:24 PM on August 29, 2023 [1 favorite]
Most people understandably will be focusing their sympathies on the students and IT staff, but the administrative implications of this are mind boggling. If they can't fix it soon, the problems will begin multiplying as thousands of deadlines large and small start to pass.
posted by HotToddy at 12:25 PM on August 29, 2023 [7 favorites]
posted by HotToddy at 12:25 PM on August 29, 2023 [7 favorites]
That's why I don't think this is ransomware, dr_dank. If it were, we'd be seeing services coming back online one at a time.
My mind is going to Silent Librarian / Mabna Institute type stuff. Nation states infiltrating research universities to steal research. We know it's happened. (It's happened to the uni I work for, though not to my department and definitely not with this level of associated destruction.) We know UMich ain't patsies; they most likely wouldn't fall to just anyone. We know that this type of attack would reward a sneaky attacker good at persisting themself, hiding their tracks, and migrating from system to system... making them harder to eradicate completely.
I'm not prepared to assert that this absolutely is what's going on... but I'm sure not prepared to rule it out, either.
posted by humbug at 12:29 PM on August 29, 2023 [9 favorites]
My mind is going to Silent Librarian / Mabna Institute type stuff. Nation states infiltrating research universities to steal research. We know it's happened. (It's happened to the uni I work for, though not to my department and definitely not with this level of associated destruction.) We know UMich ain't patsies; they most likely wouldn't fall to just anyone. We know that this type of attack would reward a sneaky attacker good at persisting themself, hiding their tracks, and migrating from system to system... making them harder to eradicate completely.
I'm not prepared to assert that this absolutely is what's going on... but I'm sure not prepared to rule it out, either.
posted by humbug at 12:29 PM on August 29, 2023 [9 favorites]
My guess is a nation state infiltration as well. It happens all over the US: universities, companies. But I have absolutely no evidence, just a guess.
posted by Nelson at 12:40 PM on August 29, 2023 [1 favorite]
posted by Nelson at 12:40 PM on August 29, 2023 [1 favorite]
My infosec nerd connections pass on the rumor that it's this vulnerability being exploited, which would lend support to a ransomware-type attack.
posted by bgribble at 12:43 PM on August 29, 2023 [4 favorites]
posted by bgribble at 12:43 PM on August 29, 2023 [4 favorites]
My college, in Lansing, Michigan, went through this in March. This was just after we'd been briefly shut down after the shootings at Michigan State, which is just down the road in East Lansing. It was a semester of unprecedented disruption for our students. There were a lot of interesting and unexpected challenges. For instance, after we came back the administrator for my area said that they'd spent ages tracking down employee phone numbers to let everyone know about the shutdown, and not to try to use their work computers, because all that info had been made electronic after having existed in big binders. For some recent hires, they did things like dig through files of resumes to find contact into. Now it's all electronic AND the admins have big printed binders for doing that emergency outreach, should something like that ever happen again.
When we came back to campus, the computer people had set up a big station we all had to process through to reset our passwords and get safely logged back into our systems and work computers.
We have thousands of students. Doing all that with 120,000? Wow.
posted by Well I never at 12:44 PM on August 29, 2023 [8 favorites]
When we came back to campus, the computer people had set up a big station we all had to process through to reset our passwords and get safely logged back into our systems and work computers.
We have thousands of students. Doing all that with 120,000? Wow.
posted by Well I never at 12:44 PM on August 29, 2023 [8 favorites]
During the pandemic, the state of New Mexico set up free, robust internet services accessible from large parking lots around the state so that students could continue to take classes, thus by-passing the "no internet at home" issues. I think it should become SOP for the DR plans for lots of critical institutions.
One of problems we had to deal with when we got back after several days shut down when this happened to my college is that students had no way to know whether they were still expected to meet deadlines that occurred during the shutdown, and some profs expected them to though most did not. I work at a community college, and many of our students are low-income. They can check out TI-84 calculators, laptops, and even internet hotspots for the semester. One prof expected students to figure out that they could go the library, log into their web-based textbook/homework system, and keep up that way, but most of our students aren't that resourceful, or legit don't have the time and energy to tackle something like that on top of work, childcare responsibilities, or other life stresses.
posted by Well I never at 12:51 PM on August 29, 2023 [12 favorites]
One of problems we had to deal with when we got back after several days shut down when this happened to my college is that students had no way to know whether they were still expected to meet deadlines that occurred during the shutdown, and some profs expected them to though most did not. I work at a community college, and many of our students are low-income. They can check out TI-84 calculators, laptops, and even internet hotspots for the semester. One prof expected students to figure out that they could go the library, log into their web-based textbook/homework system, and keep up that way, but most of our students aren't that resourceful, or legit don't have the time and energy to tackle something like that on top of work, childcare responsibilities, or other life stresses.
posted by Well I never at 12:51 PM on August 29, 2023 [12 favorites]
Oof. Yeah, if the actual network infrastructure got hit, there's not much to do but take everything down. How long would it take to patch and verify? Network adminning is very not my thing.
posted by humbug at 12:51 PM on August 29, 2023
posted by humbug at 12:51 PM on August 29, 2023
No that you're wrong about having something set up like that. It was impossible to communicate efficiently with students during the shutdown, though. How do you make sure they all know about the backup computer set up site? How responsible do you hold them if they don't find out about it? It's a mess.
posted by Well I never at 12:52 PM on August 29, 2023
posted by Well I never at 12:52 PM on August 29, 2023
The Detroid Free Press is saying the FBI is investigating, which also points to a ransomware attack.
posted by Kattullus at 12:55 PM on August 29, 2023 [3 favorites]
posted by Kattullus at 12:55 PM on August 29, 2023 [3 favorites]
Mod note: Two comments deleted. Let's keep the conversation on topic.
posted by loup (staff) at 12:55 PM on August 29, 2023 [5 favorites]
posted by loup (staff) at 12:55 PM on August 29, 2023 [5 favorites]
....interestingly, U of Minnesota has had its own notification about a global security breach followed by a user claiming to have sensitive data from UMN databases last week. I.... am going to hope we're not equally at risk at the moment.
I should note, there's no way UMich's sysadmin stafff isn't in-house. While I know university IT often contracts out cloud services for particular services, in my experience almost all of the actual keeping-the-lights-running staff are in fact in-house. That looks to my inexpert eye like what's melting down over in Michigan today.
posted by sciatrix at 12:57 PM on August 29, 2023 [5 favorites]
I should note, there's no way UMich's sysadmin stafff isn't in-house. While I know university IT often contracts out cloud services for particular services, in my experience almost all of the actual keeping-the-lights-running staff are in fact in-house. That looks to my inexpert eye like what's melting down over in Michigan today.
posted by sciatrix at 12:57 PM on August 29, 2023 [5 favorites]
That's why I don't think this is ransomware, dr_dank. If it were, we'd be seeing services coming back online one at a time.
Not really. I happened to be working for these guys when they were hit with ransomware - even with cybersecurity specialists brought in from law enforcement, networking vendors and Microsoft - it still took 30 days to bring back the core services, then another 1-2 weeks for additional apps.
posted by rozcakj at 12:59 PM on August 29, 2023 [8 favorites]
Not really. I happened to be working for these guys when they were hit with ransomware - even with cybersecurity specialists brought in from law enforcement, networking vendors and Microsoft - it still took 30 days to bring back the core services, then another 1-2 weeks for additional apps.
posted by rozcakj at 12:59 PM on August 29, 2023 [8 favorites]
You rarely can say that the problem is just [one thing].
Once the attackers have administrative access into any computer, by virtue of being networked, the logins for anyone logging into that computer are compromised, and then any computer accessed with that login is now potentially compromised. There aren't signs on computers that say "not infected" and "infected". If they're turning off the entire network, it means that they don't even know the scope of the problem.
There's a whole class of attacks - Advanced Persistent Threats - which the attacker installs a backdoor and that quietly hangs around in the background until it specifically finds what it's looking for, like Department of Defense information or something. That's when it goes "loud" and sends out the collected data and that might be the first time anyone notices it.
posted by meowzilla at 1:24 PM on August 29, 2023 [3 favorites]
Once the attackers have administrative access into any computer, by virtue of being networked, the logins for anyone logging into that computer are compromised, and then any computer accessed with that login is now potentially compromised. There aren't signs on computers that say "not infected" and "infected". If they're turning off the entire network, it means that they don't even know the scope of the problem.
There's a whole class of attacks - Advanced Persistent Threats - which the attacker installs a backdoor and that quietly hangs around in the background until it specifically finds what it's looking for, like Department of Defense information or something. That's when it goes "loud" and sends out the collected data and that might be the first time anyone notices it.
posted by meowzilla at 1:24 PM on August 29, 2023 [3 favorites]
There has recently been a long series of network devices being compromised, which is really bad simply because it's the stuff you are counting on to protect you. The "secure file transfer appliances" that aren't secure, the "secure remote access devices" (i.e., Citrix) that get popped, the fifty-one vulns in D-Link gear that dropped yesterday (see https://www.zerodayinitiative.com/advisories/published/), etc., etc.
When the attacker co-opts the network itself, it's so hard to root them out and be confident they're gone and feel like you got everything.
--
During the pandemic, the state of New Mexico set up free, robust internet services accessible from large parking lots around the state....
During that same time, Rhode Island got its library parking lot lights wired up (courtesy of the boffins at OSHEAN) to extend the WiFi, so that everyone could park outside the library at night to do homework, order stuff, make OCVID test reservations, etc.: https://oslri.org/parking-lot-wi-fi/ It was a smart use of existing resources -- the wireless network, idle at night -- and served a significant need.
posted by wenestvedt at 1:28 PM on August 29, 2023 [22 favorites]
When the attacker co-opts the network itself, it's so hard to root them out and be confident they're gone and feel like you got everything.
--
During the pandemic, the state of New Mexico set up free, robust internet services accessible from large parking lots around the state....
During that same time, Rhode Island got its library parking lot lights wired up (courtesy of the boffins at OSHEAN) to extend the WiFi, so that everyone could park outside the library at night to do homework, order stuff, make OCVID test reservations, etc.: https://oslri.org/parking-lot-wi-fi/ It was a smart use of existing resources -- the wireless network, idle at night -- and served a significant need.
posted by wenestvedt at 1:28 PM on August 29, 2023 [22 favorites]
During the summer break, many computers are shut down, because they're not in use
In 2023, departmental IT mismanagement should not cause central IT to delete the internet connection. If I had to guess, something compromised the centralized authentication service in a way that they had to rebootstrap all trusted secrets.
🤞we get the full story out of this one as it seems likely to be far more complicated than outsiders will be able to deduce.
posted by pwnguin at 1:44 PM on August 29, 2023 [2 favorites]
In 2023, departmental IT mismanagement should not cause central IT to delete the internet connection. If I had to guess, something compromised the centralized authentication service in a way that they had to rebootstrap all trusted secrets.
🤞we get the full story out of this one as it seems likely to be far more complicated than outsiders will be able to deduce.
posted by pwnguin at 1:44 PM on August 29, 2023 [2 favorites]
My first year of university was that weird in-between time where they had a computer program that handled course scheduling, but you accessed it via touch tone phone. Call in and input course codes via your phone's keypad. In my second year they did away with it and let us do course selection on the web
posted by thecjm at 2:00 PM on August 29, 2023 [4 favorites]
posted by thecjm at 2:00 PM on August 29, 2023 [4 favorites]
dabble in tertiary education
Really? It's almost certainly one of the top public universities in the nation academically.
Yes, the University of Michigan is a world-class research-1 university, and has been for some time.
posted by doctornemo at 2:13 PM on August 29, 2023 [5 favorites]
Really? It's almost certainly one of the top public universities in the nation academically.
Yes, the University of Michigan is a world-class research-1 university, and has been for some time.
posted by doctornemo at 2:13 PM on August 29, 2023 [5 favorites]
(raises hand) UM alumnus here, three degrees from it and what feels like a lifetime in Ann Arbor.
This is a *huge* problem for the institution and its people. The university is playing its cards close to its vest, but I'm hearing multiple reports of federal agents now in the loop and even on campus.
(nostalgia lowers that hand) Man, I remember getting on the pre-web internet in Angell Hall back in the 1980s...
posted by doctornemo at 2:16 PM on August 29, 2023 [11 favorites]
This is a *huge* problem for the institution and its people. The university is playing its cards close to its vest, but I'm hearing multiple reports of federal agents now in the loop and even on campus.
(nostalgia lowers that hand) Man, I remember getting on the pre-web internet in Angell Hall back in the 1980s...
posted by doctornemo at 2:16 PM on August 29, 2023 [11 favorites]
Some updates:
from the Freep
CNN
MLive
Click on Detroit
posted by doctornemo at 2:49 PM on August 29, 2023 [1 favorite]
from the Freep
CNN
MLive
Click on Detroit
posted by doctornemo at 2:49 PM on August 29, 2023 [1 favorite]
From July: Rutgers, Stony Brook Among Colleges Warning of Data Exposure From MOVEit Hack.
posted by armacy at 2:53 PM on August 29, 2023 [1 favorite]
posted by armacy at 2:53 PM on August 29, 2023 [1 favorite]
>checks calendar
August 29th: Skynet Becomes Self-aware
posted by enfa at 3:14 PM on August 29, 2023 [18 favorites]
August 29th: Skynet Becomes Self-aware
posted by enfa at 3:14 PM on August 29, 2023 [18 favorites]
Here I am, selfishly wondering if these issues at my alma mater are causing the delays in the (opaque) background check for the job offer I've just accepted.
posted by timing at 3:18 PM on August 29, 2023 [5 favorites]
posted by timing at 3:18 PM on August 29, 2023 [5 favorites]
Doesn't the timing on first day of classes suggest they wanted to create chaos/get ransom more than access valuable research/defence secrets?
posted by Rumple at 4:39 PM on August 29, 2023
posted by Rumple at 4:39 PM on August 29, 2023
Is U of M connected to the NOIRlab telescopes that have been having all the cyberattack drama?
posted by janell at 5:02 PM on August 29, 2023
posted by janell at 5:02 PM on August 29, 2023
The way I registered for classes freshman year was to show up at an appointed time with a hundred or so fellow students in the foyer outside the biggest room in the student center
Oh, you had a foyer! In the student center! I stood in a cattle chute!
posted by GenjiandProust at 5:17 PM on August 29, 2023 [22 favorites]
Oh, you had a foyer! In the student center! I stood in a cattle chute!
posted by GenjiandProust at 5:17 PM on August 29, 2023 [22 favorites]
I remember the touch-tone system. When it opened it was DoS'ed for hours (days?) because everyone was trying to get the coveted Tuesday/Thursday schedule, so they could be hungover Friday through Monday. You just had to keep dialing and dialing, otherwise you end up with German at 8am M-F.
posted by credulous at 5:30 PM on August 29, 2023 [9 favorites]
posted by credulous at 5:30 PM on August 29, 2023 [9 favorites]
That’s how I ended up in a Saturday morning math class. Should have been illegal to do that to a freshman.
posted by PussKillian at 5:45 PM on August 29, 2023 [5 favorites]
posted by PussKillian at 5:45 PM on August 29, 2023 [5 favorites]
The Detroid Free Press is saying the FBI is investigating, which also points to a ransomware attack.
The FBI is very active in all cybercrime areas; they've been briefing the company I work at going back at least 15 years warning about IP theft and other risks.
(While the incident confirms cybercrime is serious, this is actually a bit of an ongoing sore spot for me, as it becomes the main area of IT our Board actually cares about. I had a computer last week that literally took an hour to boot up, but the Board won't hear about that.)
posted by mark k at 5:53 PM on August 29, 2023 [4 favorites]
The FBI is very active in all cybercrime areas; they've been briefing the company I work at going back at least 15 years warning about IP theft and other risks.
(While the incident confirms cybercrime is serious, this is actually a bit of an ongoing sore spot for me, as it becomes the main area of IT our Board actually cares about. I had a computer last week that literally took an hour to boot up, but the Board won't hear about that.)
posted by mark k at 5:53 PM on August 29, 2023 [4 favorites]
My infosec nerd connections pass on the rumor that it's this vulnerability being exploited, which would lend support to a ransomware-type attack.
I have to admit I just hovered the link and saw the string "Citrix" and my hands shut down and refused to click through to anything involved with Citrix.
Because as someone who used to support Citrix? Fuck Citrix! Fuck anything to do with Citrix! To mis-appropriate Don DeLillo - Citrix invented the concept of remote frameworks. Citrix deserves whatever it gets. This alone warrants their doom.
*hisses and makes warding gestures*
posted by loquacious at 6:06 PM on August 29, 2023 [15 favorites]
I have to admit I just hovered the link and saw the string "Citrix" and my hands shut down and refused to click through to anything involved with Citrix.
Because as someone who used to support Citrix? Fuck Citrix! Fuck anything to do with Citrix! To mis-appropriate Don DeLillo - Citrix invented the concept of remote frameworks. Citrix deserves whatever it gets. This alone warrants their doom.
*hisses and makes warding gestures*
posted by loquacious at 6:06 PM on August 29, 2023 [15 favorites]
The FBI is very active in all cybercrime area
Indeed, the FBI announced today it has succesfully taken down the Qakbot ransomware bot. Probably not related? But it'd be funny if the takedown was to protect the rest of the internet rather than umich.
posted by pwnguin at 6:29 PM on August 29, 2023 [2 favorites]
Indeed, the FBI announced today it has succesfully taken down the Qakbot ransomware bot. Probably not related? But it'd be funny if the takedown was to protect the rest of the internet rather than umich.
posted by pwnguin at 6:29 PM on August 29, 2023 [2 favorites]
Citrix exists at the intersection of capitalism and Cthulu, and I absolutely hated supporting it; but at least the basic Terminal Services component that it long ago sold to Microsoft for integration into the OS is good.
They definitely make everything more expensive than it needs to be, which can lead to bad security because there isn't money in the budget for the recommended solutions, or to upgrade to the latest versions, or to implement and support tighter policies. Plus, nothing ever works the way they say it's supposed to.
(I don't know if they quite invented the concept of remote frameworks. What about Xwindows? NetWare had an early network multiuser environment based on DESQview. Or, rsh even.)
posted by snuffleupagus at 6:45 PM on August 29, 2023 [5 favorites]
They definitely make everything more expensive than it needs to be, which can lead to bad security because there isn't money in the budget for the recommended solutions, or to upgrade to the latest versions, or to implement and support tighter policies. Plus, nothing ever works the way they say it's supposed to.
(I don't know if they quite invented the concept of remote frameworks. What about Xwindows? NetWare had an early network multiuser environment based on DESQview. Or, rsh even.)
posted by snuffleupagus at 6:45 PM on August 29, 2023 [5 favorites]
I used to work for UMich and all I can say is the folks who work in IT there are some of THE most dedicated, hardworking, kind, creative individuals I've met. When I wanted to run an event where kids could play Assassins Creed Origins to learn about Ancient Egypt, they got me a gaming computer and figured out how to hook it up to our ancient projector! Every ticket I submitted was responded to in a timely and professional manner. No question I had was treated like it was unimportant or stupid. I'm sending alllllllll the good vibes their way.
posted by Mouse Army at 6:59 PM on August 29, 2023 [28 favorites]
posted by Mouse Army at 6:59 PM on August 29, 2023 [28 favorites]
When I was at Washington State University in the mid-1970s, we had computerized scheduling. We stood in line (alphabetically!) to submit our requests on paper and waited three days (most of which was probably spent keypunching the requests) for the Immensely Big Machine behind the glass walls to spit out the schedules. Then began the great rush to add and drop classes...
posted by lhauser at 7:08 PM on August 29, 2023 [1 favorite]
posted by lhauser at 7:08 PM on August 29, 2023 [1 favorite]
please spare a thought for the underpaid drones who work for the call centres that answer their phones.
I AM that person. We got 400 phone calls because some departments thought it was a great idea to make it look like there was more availability than there was. Reading this makes me think things I would get my comment deleted for. I'm knocking on wood that my work isn't next, but we'd be the perfect hack target and it's the 2020's.
posted by jenfullmoon at 7:13 PM on August 29, 2023 [9 favorites]
I AM that person. We got 400 phone calls because some departments thought it was a great idea to make it look like there was more availability than there was. Reading this makes me think things I would get my comment deleted for. I'm knocking on wood that my work isn't next, but we'd be the perfect hack target and it's the 2020's.
posted by jenfullmoon at 7:13 PM on August 29, 2023 [9 favorites]
Forgot to mention when my employer was struck by ransomware early last year, our (literal) next door neighbors at the FBI came knocking at our door very quickly, and were very helpful.
posted by lhauser at 7:13 PM on August 29, 2023 [1 favorite]
posted by lhauser at 7:13 PM on August 29, 2023 [1 favorite]
My uninformed guess: they found some widespread cryptolocker infection before it triggered. That would be an excellent reason to take down the network, and slowly bring things back online once they’ve been verified clean.
posted by sbutler at 7:34 PM on August 29, 2023 [2 favorites]
posted by sbutler at 7:34 PM on August 29, 2023 [2 favorites]
This reminds me of a fall semester in the early 2000s when a lot of colleges and universities experienced severe network problems when students returned to campus and their Windows computers became infected with a worm. This was before Windows was set up to automatically update on personally-owned computers by default so many people didn't update their computers […] We decided burn a whole lot of CDs with the appropriate Windows patch […]
I remember this too (doing dorm peer computer help). Before we had such CDs, there were incidents of computers getting infected during the 15 minutes it took to download the Windows update.
posted by brendano at 7:45 PM on August 29, 2023 [2 favorites]
first-day-of-classes escape rooms
Honestly, when i was doing undergrad computer lab support I really wanted to BOFH and LART about one in three undergrads until I realized they were already dealing with their own private, personal escape rooms every day.
I still wish my boss let me follow through replacing all of the "NO FOOD OR DRINKS IN THE LABS" signage with full color posters of real world bacterial cultures and samples from our computer lab keyboards and other high contact surfaces.
Of all the places on campus to use your unwashed hands to cram vending machine treats into your piehole, the public computer labs are the absolute worst place to do it. Even the mechanical engineering bathrooms would be cleaner, because at least those have the ability to be cleaned and the staff to do it.
posted by loquacious at 8:16 PM on August 29, 2023 [7 favorites]
Honestly, when i was doing undergrad computer lab support I really wanted to BOFH and LART about one in three undergrads until I realized they were already dealing with their own private, personal escape rooms every day.
I still wish my boss let me follow through replacing all of the "NO FOOD OR DRINKS IN THE LABS" signage with full color posters of real world bacterial cultures and samples from our computer lab keyboards and other high contact surfaces.
Of all the places on campus to use your unwashed hands to cram vending machine treats into your piehole, the public computer labs are the absolute worst place to do it. Even the mechanical engineering bathrooms would be cleaner, because at least those have the ability to be cleaned and the staff to do it.
posted by loquacious at 8:16 PM on August 29, 2023 [7 favorites]
(I don't know if they quite invented the concept of remote frameworks. What about Xwindows? NetWare had an early network multiuser environment based on DESQview. Or, rsh even.)
It is forbidden to speak of Novell NetWare versions 4.11 and earlier. Later versions of NetWare don't exist and are forbidden. I would elaborate but it is forbidden to speak of Novell NetWare. Please refer to the forbidden Novell NetWare documentation for any further questions about why it is forbidden to speak of Novell NetWare. Please note that you will need to install the forbidden IPX/SPX protocols to access the forbidden documentation that will forbid you to speak of Novell NetWare.
If you find yourself installing SuSE, OES, or even OS/2 to access the forbidden NetWare documentation or forbidden IPX/SPX protocols to speak of NetWare, please note that this is forbidden and you may be funged, folded or even spindled for even considering or attempting to speak of NetWare, which is forbidden.
Please note that reading this comment about Novell NetWare is forbidden and you may experience being folded or spindled if you read the entirety of this comment. By reading the entirety of this comment you agree that speaking of Novel NetWare is forbidden and you agree to be folded or spindled at this time or any time in the past or future.
posted by loquacious at 8:36 PM on August 29, 2023 [28 favorites]
It is forbidden to speak of Novell NetWare versions 4.11 and earlier. Later versions of NetWare don't exist and are forbidden. I would elaborate but it is forbidden to speak of Novell NetWare. Please refer to the forbidden Novell NetWare documentation for any further questions about why it is forbidden to speak of Novell NetWare. Please note that you will need to install the forbidden IPX/SPX protocols to access the forbidden documentation that will forbid you to speak of Novell NetWare.
If you find yourself installing SuSE, OES, or even OS/2 to access the forbidden NetWare documentation or forbidden IPX/SPX protocols to speak of NetWare, please note that this is forbidden and you may be funged, folded or even spindled for even considering or attempting to speak of NetWare, which is forbidden.
Please note that reading this comment about Novell NetWare is forbidden and you may experience being folded or spindled if you read the entirety of this comment. By reading the entirety of this comment you agree that speaking of Novel NetWare is forbidden and you agree to be folded or spindled at this time or any time in the past or future.
posted by loquacious at 8:36 PM on August 29, 2023 [28 favorites]
Doesn't the timing on first day of classes suggest they wanted to create chaos/get ransom more than access valuable research/defence secrets?
You are getting cause and effect backwards.
The university turned off their own network in order to prevent secret information from leaving the network. Also, to prevent additional computers from getting attacked.
All the university's computers / networking hardware are quarantined both from the outside world and probably from each other, because none of them are in a trusted state.
posted by meowzilla at 9:33 PM on August 29, 2023 [3 favorites]
You are getting cause and effect backwards.
The university turned off their own network in order to prevent secret information from leaving the network. Also, to prevent additional computers from getting attacked.
All the university's computers / networking hardware are quarantined both from the outside world and probably from each other, because none of them are in a trusted state.
posted by meowzilla at 9:33 PM on August 29, 2023 [3 favorites]
Could universities contact news or reddit or something to announce resources in an emergency?
I realize it isn't simple because nothing is simple-- for example authenticating whether information is actually coming from the university, but still....
posted by Nancy Lebovitz at 9:52 PM on August 29, 2023
I realize it isn't simple because nothing is simple-- for example authenticating whether information is actually coming from the university, but still....
posted by Nancy Lebovitz at 9:52 PM on August 29, 2023
I went looking for the UMich Reddit and found this. Honestly, it wasn't nearly as panicky and crazeballs there as I would have expected.
posted by jenfullmoon at 10:54 PM on August 29, 2023 [1 favorite]
posted by jenfullmoon at 10:54 PM on August 29, 2023 [1 favorite]
wasn't nearly as panicky and crazeballs there as I would have expected.
Mint mobile offers an unlimited data plan for 15 bucks a month. Not a huge deal, even after it flips back to regular rates. Monday and Tuesday are basically just "here's a syllabus, dont cheat" day anyways, so faculty barely have to change their instruction approach beyond maybe figuring out how to connect their laptop to the AV.
Also: didn't most of these kids just spend 2 years of their short life staring at a computer screen all day? "Network is down" sounds almost like a vacation in comparison.
posted by pwnguin at 11:48 PM on August 29, 2023 [4 favorites]
Mint mobile offers an unlimited data plan for 15 bucks a month. Not a huge deal, even after it flips back to regular rates. Monday and Tuesday are basically just "here's a syllabus, dont cheat" day anyways, so faculty barely have to change their instruction approach beyond maybe figuring out how to connect their laptop to the AV.
Also: didn't most of these kids just spend 2 years of their short life staring at a computer screen all day? "Network is down" sounds almost like a vacation in comparison.
posted by pwnguin at 11:48 PM on August 29, 2023 [4 favorites]
I really feel this because I moved into a new apartment yesterday and my wife, who I've been in a relationship with since the first release of Netscape Navigator, said "Setting up the internet is not a priority".
posted by srboisvert at 3:51 AM on August 30, 2023 [7 favorites]
posted by srboisvert at 3:51 AM on August 30, 2023 [7 favorites]
Most people understandably will be focusing their sympathies on the students and IT staff, but the administrative implications of this are mind boggling. If they can't fix it soon, the problems will begin multiplying as thousands of deadlines large and small start to pass.
The vast majority of deadlines at universities are just Calvinball. Many of the external deadlines are also Calvinball. It might just be Calvinball all the way up and down [I'm married to a dept. chair whose life revolves almost entirely around navigating the rules of Calvinball. Sometimes she's Calvin and sometimes she Calvined.]
posted by srboisvert at 4:26 AM on August 30, 2023 [16 favorites]
The vast majority of deadlines at universities are just Calvinball. Many of the external deadlines are also Calvinball. It might just be Calvinball all the way up and down [I'm married to a dept. chair whose life revolves almost entirely around navigating the rules of Calvinball. Sometimes she's Calvin and sometimes she Calvined.]
posted by srboisvert at 4:26 AM on August 30, 2023 [16 favorites]
Could universities contact news or reddit or something to announce resources in an emergency?
It has a radio station. WCBN 88.3 FM. 1,400 watts. I think they've got the campus covered. Town too.
posted by snuffleupagus at 5:41 AM on August 30, 2023 [3 favorites]
It has a radio station. WCBN 88.3 FM. 1,400 watts. I think they've got the campus covered. Town too.
posted by snuffleupagus at 5:41 AM on August 30, 2023 [3 favorites]
I was just thinking of radio, and then wondered if kids these days listen to radio.
posted by MtDewd at 5:58 AM on August 30, 2023 [7 favorites]
posted by MtDewd at 5:58 AM on August 30, 2023 [7 favorites]
It is forbidden to speak of Novell NetWare versions 4.11 and earlier...
I'm having flashbacks to my first software testing gig and dealing with NetWare - because it was forbidden (aka near impossible) to clone the installs. So every time we had a project that required NetWare the servers had to be set up manually. I became quite the expert in the lab.
posted by theBigRedKittyPurrs at 6:00 AM on August 30, 2023 [1 favorite]
I'm having flashbacks to my first software testing gig and dealing with NetWare - because it was forbidden (aka near impossible) to clone the installs. So every time we had a project that required NetWare the servers had to be set up manually. I became quite the expert in the lab.
posted by theBigRedKittyPurrs at 6:00 AM on August 30, 2023 [1 favorite]
Much #hugops to the UMich folks - I have worked on "front page news" outages and it's just not fun.
posted by rmd1023 at 6:50 AM on August 30, 2023 [4 favorites]
posted by rmd1023 at 6:50 AM on August 30, 2023 [4 favorites]
They're back on line!
From the UMich Announcements feed as of 10:30am Aug 30:
From the UMich Announcements feed as of 10:30am Aug 30:
NOTICE: Internet Restored on All U-M Campusesposted by heatherlogan at 7:59 AM on August 30, 2023 [24 favorites]
Dear U-M community,
Thank you for your patience during our recent service disruption. I am happy to inform you that internet connectivity and WiFi has been restored on all U-M campuses. You should be able to connect as normal from any device.
We expect some issues with select U-M systems and services in the short term, and not all of our remediation efforts are complete. However, they will be resolved over the next several days.
We will be posting announcements about any service interruptions on the ITS status page. Please contact our Service Center for technical assistance if needed.
The investigative work into the security issue continues, and we are not able to share any information that might compromise the investigation. We appreciate your understanding as we continue to move through the investigative process.
In true Wolverine fashion, faculty, staff and students rose to the occasion and met the challenge to ensure continuity of our mission.
We extend a special thank you to the Information and Technology Services team who all worked tirelessly to address this challenge. We all appreciate all you have done, and continue to do, to maintain the safety of our enterprise systems.
Sincerely,
Santa J. Ono
President
Ravi Pendse PhD
Vice President for Information Technology and Chief Information Officer
University of Michigan
Hail! Go Blue!
My heart goes out to the IT department. Great people
posted by JoeXIII007 at 8:55 AM on August 30, 2023 [5 favorites]
My heart goes out to the IT department. Great people
posted by JoeXIII007 at 8:55 AM on August 30, 2023 [5 favorites]
Eh, even if they pay the football coach the most money, they're more likely a healthcare institution.
The U of Michigan healthcare system has just bought out, or is buying out, Lansing's main healthcare company, a thing so sprawling and huge that there's a joke about the whole area becoming one big hospital.
They're not just a healthcare institution, they seem to be on their way to becoming the healthcare institution.
posted by Well I never at 9:24 AM on August 30, 2023 [4 favorites]
The U of Michigan healthcare system has just bought out, or is buying out, Lansing's main healthcare company, a thing so sprawling and huge that there's a joke about the whole area becoming one big hospital.
They're not just a healthcare institution, they seem to be on their way to becoming the healthcare institution.
posted by Well I never at 9:24 AM on August 30, 2023 [4 favorites]
Monday and Tuesday are basically just "here's a syllabus, dont cheat" day anyways, so faculty barely have to change their instruction approach beyond maybe figuring out how to connect their laptop to the AV.
Unless they couldn't even get to their syllabus, which happened to one UM prof I spoke to.
posted by Preserver at 9:33 AM on August 30, 2023 [4 favorites]
Unless they couldn't even get to their syllabus, which happened to one UM prof I spoke to.
posted by Preserver at 9:33 AM on August 30, 2023 [4 favorites]
...faculty barely have to change their instruction approach beyond maybe figuring out how to connect their laptop to the AV.
Faculty can't take/report attendance without Banner+Starfish.
Students & faculty can't use those clickers with the cloud app when WiFi is down.
How about a cool "smart board" that uses collaborative edit/mark-up features? It probably integrates with AzureAD, which in turn requires WiFi.
Some projectors will happily display the slide deck, if it's on a thumb drive (and you can reach the thing!). Here's hoping there isn't any data loss-prevention software that blocks the professors from copying their slides down from OneDrive/Google!
After class ends, meal swipes should be stored in the POS terminal, but many of those larger systems (e.g., Transact, CBORD) want to traverse the network in order to print out order slips in the kitchens.
Schools with centralized room-planning and -reservation systems need WiFi for the door locks to accept new reservations. Most will fail back to offline mode and only accept cached credentials. Hope you didn't want to sign out an open study room for a group work session!
WiFi-enabled dorm locks will also fall back to cached creds, but can't report attempts by unauthorized users, nor have their allow lists updated (if a resident is added/dropped) when WiFi is down.
Outsourced tuition payment-processing might work (and cache the transaction), but it also might just break until the campus is back online. Don't wait until the last minute to pay your bills, because Banner won't see your payment and will block you.
International students' visa status has to be reported frequently and promptly, or else their visa gets zapped. Hope they didn't show up at the last minute!
OMG, there is so much outsourced software in higher ed, and so much of it suuuucks.
posted by wenestvedt at 11:12 AM on August 30, 2023 [10 favorites]
Faculty can't take/report attendance without Banner+Starfish.
Students & faculty can't use those clickers with the cloud app when WiFi is down.
How about a cool "smart board" that uses collaborative edit/mark-up features? It probably integrates with AzureAD, which in turn requires WiFi.
Some projectors will happily display the slide deck, if it's on a thumb drive (and you can reach the thing!). Here's hoping there isn't any data loss-prevention software that blocks the professors from copying their slides down from OneDrive/Google!
After class ends, meal swipes should be stored in the POS terminal, but many of those larger systems (e.g., Transact, CBORD) want to traverse the network in order to print out order slips in the kitchens.
Schools with centralized room-planning and -reservation systems need WiFi for the door locks to accept new reservations. Most will fail back to offline mode and only accept cached credentials. Hope you didn't want to sign out an open study room for a group work session!
WiFi-enabled dorm locks will also fall back to cached creds, but can't report attempts by unauthorized users, nor have their allow lists updated (if a resident is added/dropped) when WiFi is down.
Outsourced tuition payment-processing might work (and cache the transaction), but it also might just break until the campus is back online. Don't wait until the last minute to pay your bills, because Banner won't see your payment and will block you.
International students' visa status has to be reported frequently and promptly, or else their visa gets zapped. Hope they didn't show up at the last minute!
OMG, there is so much outsourced software in higher ed, and so much of it suuuucks.
posted by wenestvedt at 11:12 AM on August 30, 2023 [10 favorites]
Faculty can't take/report attendance without Banner+Starfish.
Serious question: did they do that when the network was functional? In my (admittedly dated) experience, instructional tech is like two extremely online professors who didn't get the "your promotion is based on research" memo, and everyone else uses whiteboards or a slide deck. Yes, date of last attendance is to be reported to the feds. No idea what faculty do when asked -- make something plausible up?
Nobody uses the very expensive smartboard, and the clickers get used once per lecture, presumably as a stealth attendance check, and only in the big classes. The educational benefit to their function isn't much different than a simple "show of hands" question.
want to traverse the network in order to print out order slips in the kitchens.
I mean, my point was largely about instruction, but: last time I was on a campus I wrote my order down on a slip, and that worked okay. And if worst case scenario, the kitchen is closed, campus cafeterias aren't the only source of food for 130k people. Obviously it sucks if the school pre-charged you for meals they can't provide.
Like, yes, Banner is an outdated piece of tech that powers this absurdly totalitarian system. All these things suck, especially the visa thing. But there should be some adaptive capacity lying around at a place in the running for most highly educated places in the world.
posted by pwnguin at 12:05 PM on August 30, 2023 [2 favorites]
Serious question: did they do that when the network was functional? In my (admittedly dated) experience, instructional tech is like two extremely online professors who didn't get the "your promotion is based on research" memo, and everyone else uses whiteboards or a slide deck. Yes, date of last attendance is to be reported to the feds. No idea what faculty do when asked -- make something plausible up?
Nobody uses the very expensive smartboard, and the clickers get used once per lecture, presumably as a stealth attendance check, and only in the big classes. The educational benefit to their function isn't much different than a simple "show of hands" question.
want to traverse the network in order to print out order slips in the kitchens.
I mean, my point was largely about instruction, but: last time I was on a campus I wrote my order down on a slip, and that worked okay. And if worst case scenario, the kitchen is closed, campus cafeterias aren't the only source of food for 130k people. Obviously it sucks if the school pre-charged you for meals they can't provide.
Like, yes, Banner is an outdated piece of tech that powers this absurdly totalitarian system. All these things suck, especially the visa thing. But there should be some adaptive capacity lying around at a place in the running for most highly educated places in the world.
posted by pwnguin at 12:05 PM on August 30, 2023 [2 favorites]
OMG, there is so much outsourced software in higher ed, and so much of it suuuucks.
Don't even get me started on the "homegrown vs. outsourced" argument. Frankly, both stink in different ways. Outsourced programs frequently suck, homegrowns involve dealing with the idiosyncrasies of the programmers.
But there should be some adaptive capacity lying around at a place in the running for most highly educated places in the world.
With what money, though?
posted by jenfullmoon at 12:20 PM on August 30, 2023 [4 favorites]
Don't even get me started on the "homegrown vs. outsourced" argument. Frankly, both stink in different ways. Outsourced programs frequently suck, homegrowns involve dealing with the idiosyncrasies of the programmers.
But there should be some adaptive capacity lying around at a place in the running for most highly educated places in the world.
With what money, though?
posted by jenfullmoon at 12:20 PM on August 30, 2023 [4 favorites]
Schools with centralized room-planning and -reservation systems need WiFi for the door locks to accept new reservations. Most will fail back to offline mode and only accept cached credentials. Hope you didn't want to sign out an open study room for a group work session!
My school wanted to go all swipe card on buildings and take our (tech support) physical keys away. My boss wasn't having it, and luckily (for us) the whole Facebook Locks Themselves Out Of Their Own Buildings debacle happened, and we were able to sway them.
I trust these online systems to stick around and function as far as I can throw them. There's not reason we should have to rely solely on the internet to get into our buildings.
posted by gc at 12:24 PM on August 30, 2023 [5 favorites]
My school wanted to go all swipe card on buildings and take our (tech support) physical keys away. My boss wasn't having it, and luckily (for us) the whole Facebook Locks Themselves Out Of Their Own Buildings debacle happened, and we were able to sway them.
I trust these online systems to stick around and function as far as I can throw them. There's not reason we should have to rely solely on the internet to get into our buildings.
posted by gc at 12:24 PM on August 30, 2023 [5 favorites]
"Take attendance"? What is this concept?
Also, I have never once seen a "smart board" in a university classroom. Those toys are for kid-school districts with more money than sense.
posted by heatherlogan at 1:13 PM on August 30, 2023 [3 favorites]
Also, I have never once seen a "smart board" in a university classroom. Those toys are for kid-school districts with more money than sense.
posted by heatherlogan at 1:13 PM on August 30, 2023 [3 favorites]
two extremely online professors who didn't get the "your promotion is based on research" memo
Hi, I'm the extremely online professor, and I took attendance today on the back of a map on a clipboard.
posted by erolls at 1:33 PM on August 30, 2023 [8 favorites]
Hi, I'm the extremely online professor, and I took attendance today on the back of a map on a clipboard.
posted by erolls at 1:33 PM on August 30, 2023 [8 favorites]
An application called Starfish correlates a number of factors -- attendance, low scores, disciplinary events, and more -- to try to spot students having problems before they get too serious to fix, and the student flunks out or quits or worse.
My understanding is that it really does help, but it requires that it have the data. So teachers have to take attendance and get it into Banner.
Folks who have been out of the .edu IT game for long might be surprised at the things that are required now. *shrug* There's a lot that's non-negotiable.
posted by wenestvedt at 1:47 PM on August 30, 2023 [5 favorites]
My understanding is that it really does help, but it requires that it have the data. So teachers have to take attendance and get it into Banner.
Folks who have been out of the .edu IT game for long might be surprised at the things that are required now. *shrug* There's a lot that's non-negotiable.
posted by wenestvedt at 1:47 PM on August 30, 2023 [5 favorites]
Santa Ono?? Did we cross timelines and nobody told me?
posted by slogger at 4:46 PM on August 30, 2023 [1 favorite]
posted by slogger at 4:46 PM on August 30, 2023 [1 favorite]
The UMich outage also took out the Michigan Radio (NPR) app home page with it. Podcasts, like cockroaches in a nuclear winter, were still fine.
posted by axiom at 2:03 PM on August 31, 2023 [3 favorites]
posted by axiom at 2:03 PM on August 31, 2023 [3 favorites]
I have never once seen a "smart board" in a university classroom.
No, they exist in many a college and university classroom.
I'm teaching in one tonight with three.
posted by doctornemo at 2:12 PM on August 31, 2023 [5 favorites]
No, they exist in many a college and university classroom.
I'm teaching in one tonight with three.
posted by doctornemo at 2:12 PM on August 31, 2023 [5 favorites]
I was at the newly-refreshed building science building at Providence College last weekend, and their spaces -- from small rooms to lecture halls -- have glass walls for dry-erase markers, and multiple large monitors that anyone can AirPlay/cast video or audio to, plus cameras and microphones for recording or streaming.
Rooms can all be reserved remotely so that they get used frequently for classes, study groups, or movie parties.
The opportunities for digital collaboration are amazing, and most of them would be b0rked without cloud access and local resources.
posted by wenestvedt at 6:29 PM on August 31, 2023 [4 favorites]
Rooms can all be reserved remotely so that they get used frequently for classes, study groups, or movie parties.
The opportunities for digital collaboration are amazing, and most of them would be b0rked without cloud access and local resources.
posted by wenestvedt at 6:29 PM on August 31, 2023 [4 favorites]
Right? Even the simple docucam (my beloved) is infinitely better than a chalkboard because I can actually write legibly with it and, if I want to, record my scribbled drawings and my own scrawled notes so that students can use them as study materials. And display it in real time on the giant monitors so that my students can read what I'm writing even if they have shitty vision, like me.
I have never used a smart board in a college classroom, but I have used all kinds of incredibly useful digitally integrated teaching resources... many of which do require cloud integration and/or local access in order to make use of them.
Frankly, that Starfish app sounds like genius, too. I used to have to just rely on whether TAs teaching labs were following students carefully enough to independently bring up noticed patterns in absences, which is real easy to miss. Having data on who is missing and when and why sounds really helpful, especially if you can cross-link it across classes. A lot of struggling students can turn their shit around and do a lot better if a) someone notices them struggling and b) proactively reaches out to check in and ask kind questions. There is a lot of shame and a lot of self-judgement for most of them, and they usually don't know what kinds of resources are available to them to help them with whatever is causing them to flounder. When someone notices and reaches out without judging them just to check what's up, often you can do some fairly simple stuff to fix it: everything from letting the kid with massive migraines come fifteen minutes to a quieter, darker class to giving the kid with election-related panic attacks some skills to manage them to even just saying: I don't judge you for having a bad time, and while I want you to succeed, I am not taking it personally if you don't for a little while. What can I do to help?
I don't want to pull some fucking inspiration porn The Help bullshit or whatever, they're baby adults and they get to make their own decisions and sometimes the shit they're struggling with is bigger than I can fix or than structures generally can fix. But just taking the time to notice and ask what the problems are can make so much of a difference.
posted by sciatrix at 8:27 AM on September 1, 2023 [6 favorites]
I have never used a smart board in a college classroom, but I have used all kinds of incredibly useful digitally integrated teaching resources... many of which do require cloud integration and/or local access in order to make use of them.
Frankly, that Starfish app sounds like genius, too. I used to have to just rely on whether TAs teaching labs were following students carefully enough to independently bring up noticed patterns in absences, which is real easy to miss. Having data on who is missing and when and why sounds really helpful, especially if you can cross-link it across classes. A lot of struggling students can turn their shit around and do a lot better if a) someone notices them struggling and b) proactively reaches out to check in and ask kind questions. There is a lot of shame and a lot of self-judgement for most of them, and they usually don't know what kinds of resources are available to them to help them with whatever is causing them to flounder. When someone notices and reaches out without judging them just to check what's up, often you can do some fairly simple stuff to fix it: everything from letting the kid with massive migraines come fifteen minutes to a quieter, darker class to giving the kid with election-related panic attacks some skills to manage them to even just saying: I don't judge you for having a bad time, and while I want you to succeed, I am not taking it personally if you don't for a little while. What can I do to help?
I don't want to pull some fucking inspiration porn The Help bullshit or whatever, they're baby adults and they get to make their own decisions and sometimes the shit they're struggling with is bigger than I can fix or than structures generally can fix. But just taking the time to notice and ask what the problems are can make so much of a difference.
posted by sciatrix at 8:27 AM on September 1, 2023 [6 favorites]
sciatrix: Frankly, that Starfish app sounds like genius, too. ... But just taking the time to notice and ask what the problems are can make so much of a difference.
What's really great is that the software (and others like it -- no endorsement &c.) can access records across multiple areas, and synthesize a view of the student's condition that no one faculty/staff could get. And because it's software, it's not impatient, nor is it going home at night with a head full of other people's problems, and it doesn't judge.
We have a whole office for accommodations, and another for counseling, plus advisors and tutors. Just knowing where to put that support is super helpful.
Look, I tell my own kids, "They're called 'teachers' and not 'fail-ers' because they want to teach you and not to get you thrown out. Let them help you! Go to office hours! Make small talk and connect, because they are interesting people who love to share what they know."
posted by wenestvedt at 9:57 AM on September 1, 2023 [3 favorites]
What's really great is that the software (and others like it -- no endorsement &c.) can access records across multiple areas, and synthesize a view of the student's condition that no one faculty/staff could get. And because it's software, it's not impatient, nor is it going home at night with a head full of other people's problems, and it doesn't judge.
We have a whole office for accommodations, and another for counseling, plus advisors and tutors. Just knowing where to put that support is super helpful.
Look, I tell my own kids, "They're called 'teachers' and not 'fail-ers' because they want to teach you and not to get you thrown out. Let them help you! Go to office hours! Make small talk and connect, because they are interesting people who love to share what they know."
posted by wenestvedt at 9:57 AM on September 1, 2023 [3 favorites]
Right! And otherwise I am fairly sure that that information is FERPA-protected, so I'm pretty sure that you couldn't get the whole-courseload info legally even if you had infinite time, motivation, and energy. Which I did not, and do not, and could not have. Having a database collate it into one place and not share it except when a pattern flags a pervasive problem is really quite clever.
I cannot underscore enough how much things like this are helpful for structurally assisting students and helping student facilities do their jobs in the best possible sense of the term.
posted by sciatrix at 9:33 PM on September 1, 2023 [2 favorites]
I cannot underscore enough how much things like this are helpful for structurally assisting students and helping student facilities do their jobs in the best possible sense of the term.
posted by sciatrix at 9:33 PM on September 1, 2023 [2 favorites]
[cough] my gig is involved with measuring the metrics of what makes a college "successful". The reason these schools with $$ are investing in these retention efforts is because this is one of the few that they can throw some money at to improve, and not have to confront any of the larger systemic issues.
posted by zenon at 1:07 PM on September 3, 2023 [3 favorites]
posted by zenon at 1:07 PM on September 3, 2023 [3 favorites]
I've been low-key looking for updates on this story and have seen none. No further explanation of what happened, perhaps an investigation is ongoing. They did require everyone reset their passwords but gave people a couple of weeks to do it.
posted by Nelson at 2:25 PM on September 19, 2023 [6 favorites]
posted by Nelson at 2:25 PM on September 19, 2023 [6 favorites]
« Older "I just published a wildly over-researched... | Free Online Browser tools: Big list of free In... Newer »
This thread has been archived and is closed to new comments
posted by AzraelBrown at 8:22 AM on August 29, 2023 [8 favorites]