British Library releases ransomware incident report
March 8, 2024 6:34 AM Subscribe
The British Library explains what happened and why recovery took so long. Part of the problem: "Our major software systems cannot be brought back in their pre-attack form, either because they are no longer supported by the vendor or because they will not function on the new secure infrastructure that is currently being rolled out."
As a librarian myself... I find this chilling. Frankly, from what I know, more libraries than not are running outdated software somewhere, not-wholly-uncommonly in core work areas like the Integrated Library System or (as with the BL) the guts of digital-collections systems.
I hope the profession starts some serious discussions right now about the security and operations risks inherent to not migrating. I sure plan to raise some hell, as opportunity permits.
Previously.
As a librarian myself... I find this chilling. Frankly, from what I know, more libraries than not are running outdated software somewhere, not-wholly-uncommonly in core work areas like the Integrated Library System or (as with the BL) the guts of digital-collections systems.
I hope the profession starts some serious discussions right now about the security and operations risks inherent to not migrating. I sure plan to raise some hell, as opportunity permits.
Previously.
Eh. That, and we don't have the resources to modernize, often. Because people don't think we need it. We're just libraries, after all, and libraries totally stopped evolving in 1950, right?
Michael Taggart on Mastodon: "Ransomware also tends to be the bill collector for technical debt." Amazingly well-put.
posted by humbug at 6:45 AM on March 8 [45 favorites]
Michael Taggart on Mastodon: "Ransomware also tends to be the bill collector for technical debt." Amazingly well-put.
posted by humbug at 6:45 AM on March 8 [45 favorites]
I hope the profession starts some serious discussions right now about the security and operations risks inherent to not migrating.
I seriously doubt those discussions aren’t happening. The risks are well known. But, as always, the issue will always come down to finances. Migrating/updating can be hugely expensive, as is maintenance. And, as our wonderfully interconnected world becomes more and more perilous, the updating will also chew-up more and more financial resources.
Libraries, like most public entities, operate on extremely tight (and, in US anyway, ever-decreasing unless you’re the police) budgets. At what point does the cost of software and systems maintenance and updating equal or outstrip funding for the core mission?
It’s super easy to castigate libraries for not spending whatever it takes to upgrade, but if the money isn’t there to do it properly and keep it updated, what are they to do? The BL is likely in a much better financial position to have been keeping systems up-to-date, but most libraries certainly aren’t. It’s somewhat cruel and very unfair to point at libraries and castigate them for not spending whatever it takes to stay updated.
posted by Thorzdad at 6:52 AM on March 8 [22 favorites]
I seriously doubt those discussions aren’t happening. The risks are well known. But, as always, the issue will always come down to finances. Migrating/updating can be hugely expensive, as is maintenance. And, as our wonderfully interconnected world becomes more and more perilous, the updating will also chew-up more and more financial resources.
Libraries, like most public entities, operate on extremely tight (and, in US anyway, ever-decreasing unless you’re the police) budgets. At what point does the cost of software and systems maintenance and updating equal or outstrip funding for the core mission?
It’s super easy to castigate libraries for not spending whatever it takes to upgrade, but if the money isn’t there to do it properly and keep it updated, what are they to do? The BL is likely in a much better financial position to have been keeping systems up-to-date, but most libraries certainly aren’t. It’s somewhat cruel and very unfair to point at libraries and castigate them for not spending whatever it takes to stay updated.
posted by Thorzdad at 6:52 AM on March 8 [22 favorites]
In addition to funding libraries, maybe also put pressure on corporations for minimum support requirements? Even if all they do is define the support term as X years, that gives the libraries a way to budget for needed upgrades.
posted by CheeseDigestsAll at 6:55 AM on March 8 [10 favorites]
posted by CheeseDigestsAll at 6:55 AM on March 8 [10 favorites]
I actually do doubt those discussions are happening everywhere they should be, Thorzdad. While resources are indeed a serious constraint, and one I'm wearily familiar with from my own librarian career, part of the problem is a fairly widespread and largely unchallenged culture in librarianship of foot-dragging when it comes to any software change that changes familiar workflows.
The BL is now the paradigm of why that can't be acceptable. That's what I think needs communicating and what I want to communicate (and oh boy am I ever planning my next five library-conference presentations right now).
As for "cruel and very unfair," I'm totes willing to have that discussion with librarianship -- it's been cruel and very unfair to me over the years. I think it should maybe back off and be glad I'm still trying, in my small way, to help it.
posted by humbug at 7:01 AM on March 8 [8 favorites]
The BL is now the paradigm of why that can't be acceptable. That's what I think needs communicating and what I want to communicate (and oh boy am I ever planning my next five library-conference presentations right now).
As for "cruel and very unfair," I'm totes willing to have that discussion with librarianship -- it's been cruel and very unfair to me over the years. I think it should maybe back off and be glad I'm still trying, in my small way, to help it.
posted by humbug at 7:01 AM on March 8 [8 favorites]
Thank you for posting this. I'm a humanities librarian an ocean away from the BL, and the outage has been a problem even for me once every few weeks, when I'm trying to help a patron locate something, and there's... no British Library online. I wish them luck moving forward, and all best for the financial and technical support they need to finish rebuilding, shift platforms, and adjust their security.
posted by cupcakeninja at 7:21 AM on March 8 [6 favorites]
posted by cupcakeninja at 7:21 AM on March 8 [6 favorites]
Our library website, was victim of an attack and our site was down a year. And it can totally be laid at the feet of our IT admin at the time who thought that we didn't need to pay for software upgrades, it would look better to the library admin if we were saving money. Thank god he left with the library director who thought monetary savings were the most important thing a library could do. Between them they nearly destroyed the library.
posted by evilDoug at 7:41 AM on March 8 [16 favorites]
posted by evilDoug at 7:41 AM on March 8 [16 favorites]
These legacy systems will in many cases need to be migrated to new versions, substantially modified, or even rebuilt from the ground up, either because they are unsupported and therefore cannot be repurchased or restored, or because they simply will not operate on modern servers or with modern security controls.Why not restore the existing servers from a whole system backup? Elsewhere in the report they talk about the attackers "destroying" servers, which could possibly mean actual destruction of hardware. But I guess no disk image level backups or backups that don't properly restore is more likely.
It's notable to me that there are three mentions of backups in the entire report, of which one is talking about the ransomware attacker's "backup" and one is their plan to implement "a robust and resilient backup service"
posted by joeyh at 7:42 AM on March 8 [1 favorite]
I'm relieved to hear that their digitized content is apparently safe, as they have digitized a number of rare texts and made them freely available through their website, outside of the paid subscription databases. (Or, well, they did make them freely available.)
posted by thomas j wise at 8:04 AM on March 8 [2 favorites]
posted by thomas j wise at 8:04 AM on March 8 [2 favorites]
[a momentary pause while I put on my NOTIS-LMS hat]
These legacy systems will in many cases need to be migrated to new versions, substantially modified, or even rebuilt from the ground up, either because they are unsupported and therefore cannot be repurchased or restored, or because they simply will not operate on modern servers or with modern security controls.
This is not entirely true, but it is effectively true because the entire library world got sucked into believing the "modern server" hype back in the 90s and migrated away from the most bulletproof LMS server environment -- OS/390. Years back a 390 server sat in the cube next to mine, and it ran a library database that had existed since the late 60s, in addition to a modern web server, a backup system, and a printing system that was so hideously fast it was actually a major fire hazard (really, that printer was fucking terrifying when it went full speed).
The MBA bosses replaced it with a Windows system that got hacked like four times in the first six months and one of my tasks was to physically reboot it every morning "just in case". But it did have a fancy-looking UI, and a CD drive that never got used!
Yes, NOTIS had weird quirks (the file layout diagrams took up hundreds of pages, because every single library in the world was convinced that their unique way of handling MARC data was the only correct way even though everyone was supposed to be doing MARC21 anyway), but it was essentially indestructible barring intentional malfeasance by staff with physical access to the machine (and, even then, unless they knew what they were doing odds are you could recover due to the ridiculous fault-tolerance built into both the hardware and software).
posted by aramaic at 8:49 AM on March 8 [16 favorites]
These legacy systems will in many cases need to be migrated to new versions, substantially modified, or even rebuilt from the ground up, either because they are unsupported and therefore cannot be repurchased or restored, or because they simply will not operate on modern servers or with modern security controls.
This is not entirely true, but it is effectively true because the entire library world got sucked into believing the "modern server" hype back in the 90s and migrated away from the most bulletproof LMS server environment -- OS/390. Years back a 390 server sat in the cube next to mine, and it ran a library database that had existed since the late 60s, in addition to a modern web server, a backup system, and a printing system that was so hideously fast it was actually a major fire hazard (really, that printer was fucking terrifying when it went full speed).
The MBA bosses replaced it with a Windows system that got hacked like four times in the first six months and one of my tasks was to physically reboot it every morning "just in case". But it did have a fancy-looking UI, and a CD drive that never got used!
Yes, NOTIS had weird quirks (the file layout diagrams took up hundreds of pages, because every single library in the world was convinced that their unique way of handling MARC data was the only correct way even though everyone was supposed to be doing MARC21 anyway), but it was essentially indestructible barring intentional malfeasance by staff with physical access to the machine (and, even then, unless they knew what they were doing odds are you could recover due to the ridiculous fault-tolerance built into both the hardware and software).
posted by aramaic at 8:49 AM on March 8 [16 favorites]
You and me both, thomas j wise. This could have been a National Museum of Brazil situation, and it's absolutely fantastic that it wasn't. (As a digital preservationist, I admit I'm preening just a tiny bit about that. Our workflows, they do actually work.) The BL deserves a lot of credit for treating irreplaceable digital materials with due care and caution. That ain't easy and it sure ain't cheap.
joeyh: Yeah, I would guess no bootable backups. It's also possible that the oldest stuff was running on the oldest hardware such that there are hardware-level incompatibilities with trying to bring it back on modern infrastructure. I know a library that didn't migrate a couple of services off Solaris until it was dead, buried, and quite thoroughly rotted. I can envision those services being unrestorable if they'd been attacked before the migration.
posted by humbug at 8:51 AM on March 8 [1 favorite]
joeyh: Yeah, I would guess no bootable backups. It's also possible that the oldest stuff was running on the oldest hardware such that there are hardware-level incompatibilities with trying to bring it back on modern infrastructure. I know a library that didn't migrate a couple of services off Solaris until it was dead, buried, and quite thoroughly rotted. I can envision those services being unrestorable if they'd been attacked before the migration.
posted by humbug at 8:51 AM on March 8 [1 favorite]
20 minutes ago: Russian hackers breached key Microsoft systems
Russian state-backed hackers gained access to some of Microsoft’s core software systems in a hack first disclosed in January, the company said Friday, revealing a more extensive and serious intrusion into Microsoft’s systems than previously known.posted by gwint at 8:56 AM on March 8 [4 favorites]
Microsoft believes that the hackers have in recent weeks used information stolen from Microsoft’s corporate email systems to access “some of the company’s source code repositories and internal systems,” the tech firm said in a filing with the US Securities and Exchange Commission.
Source code is coveted by corporations — and spies trying to breach them — because it is the secret nuts and bolts of a software program that make it function.
Hackers with access to source code can use it for follow-on attacks on other systems.
joeyh, I'm surprised that it's not in the text, but the timeline image does include "Confirmation that all onsite backups had been compromised and encrypted" on October 30th.
It would certainly be interesting to know what the rest of the backup strategy was, given that they eventually recovered the digital collections. At least they off-sited those.
Given the impact scope I'd imagine that all the server backups were on-site and on-disk. If you do that, it's nearly guaranteed that the attackers will eat your backups, too.
It may make IT people look like retrogrouches, but it's really freaking hard to encrypt a tape in a safe. Cloud immutables are maybe even better if you have the money and bandwidth.
posted by CHoldredge at 9:00 AM on March 8 [5 favorites]
It would certainly be interesting to know what the rest of the backup strategy was, given that they eventually recovered the digital collections. At least they off-sited those.
Given the impact scope I'd imagine that all the server backups were on-site and on-disk. If you do that, it's nearly guaranteed that the attackers will eat your backups, too.
It may make IT people look like retrogrouches, but it's really freaking hard to encrypt a tape in a safe. Cloud immutables are maybe even better if you have the money and bandwidth.
posted by CHoldredge at 9:00 AM on March 8 [5 favorites]
Me, looking at several dozen Microsoft TFA emails from an account I almost never use: You don't say.
posted by McBearclaw at 9:01 AM on March 8 [2 favorites]
posted by McBearclaw at 9:01 AM on March 8 [2 favorites]
I hope the profession starts some serious discussions right now about the security and operations risks inherent to not migrating.
I was at a conference yesterday for core information security people at higher education institutions in New England -- so yes, Tufts, MIT, and Harvard all had people in the audience and speaking.
It was mentioned again that any of us -- even these Best And Brightest -- can have systems that were bleeding edge when created, but not constantly kept modern. Once enough of that technical debt accumulates, it's only a matter of time until it breaks.
Technical debt is real debt!
posted by wenestvedt at 9:03 AM on March 8 [4 favorites]
I was at a conference yesterday for core information security people at higher education institutions in New England -- so yes, Tufts, MIT, and Harvard all had people in the audience and speaking.
It was mentioned again that any of us -- even these Best And Brightest -- can have systems that were bleeding edge when created, but not constantly kept modern. Once enough of that technical debt accumulates, it's only a matter of time until it breaks.
Technical debt is real debt!
posted by wenestvedt at 9:03 AM on March 8 [4 favorites]
It's fantastic to see this level of detail in public. So often ransomware intrusions like this are swept under the rug with nothing disclosed to avoid embarrassing the institution. There's a lot to digest here, I confess I've only read the executive summary.
"Why not just restore from backup?" is a pretty naive question. Information systems like a major library are incredibly complex with zillions of different computers, datastores, and software systems. As the report indicates a lot of them are obsolete and hard to restore. Also if the backup itself got corrupted by the ransomware, that's a really painful failure that's difficult to recover from. I don't think the British Library is particularly negligent in this stuff; most IT systems are vulnerable like this.
There's a couple of IT system trends out there that should help protect sites from attacks like this in the future. Ransomware resilience is a real design goal these days.
One is repeatable builds / automated installation. From simple things like an Ansible script that stands up an EC2 instance for some server function to fancy Nix-based functional system descriptions. Devops has trended towards making it easy to replicate and migrate services which makes recovery easier. It's a lot of work moving to this way of doing things though.
The other is append-only backup datastores. Systems where you can write a backup but then never delete or modify it, so it can't then be subverted. I've been using Restic as backup software lately and it has explicit support for it. It's still a work in progress; the backup datastore itself has to support it, not just the backup service. But it's a great direction for development.
posted by Nelson at 9:19 AM on March 8 [7 favorites]
"Why not just restore from backup?" is a pretty naive question. Information systems like a major library are incredibly complex with zillions of different computers, datastores, and software systems. As the report indicates a lot of them are obsolete and hard to restore. Also if the backup itself got corrupted by the ransomware, that's a really painful failure that's difficult to recover from. I don't think the British Library is particularly negligent in this stuff; most IT systems are vulnerable like this.
There's a couple of IT system trends out there that should help protect sites from attacks like this in the future. Ransomware resilience is a real design goal these days.
One is repeatable builds / automated installation. From simple things like an Ansible script that stands up an EC2 instance for some server function to fancy Nix-based functional system descriptions. Devops has trended towards making it easy to replicate and migrate services which makes recovery easier. It's a lot of work moving to this way of doing things though.
The other is append-only backup datastores. Systems where you can write a backup but then never delete or modify it, so it can't then be subverted. I've been using Restic as backup software lately and it has explicit support for it. It's still a work in progress; the backup datastore itself has to support it, not just the backup service. But it's a great direction for development.
posted by Nelson at 9:19 AM on March 8 [7 favorites]
I have a little gang of ex-library worker friends. One of them said the other day that since libraries don't make money, they don't GET money.
posted by jenfullmoon at 9:20 AM on March 8 [4 favorites]
posted by jenfullmoon at 9:20 AM on March 8 [4 favorites]
a Windows system that got hacked like four times in the first six months
During my tenure at Google (early 2000s), there was a period when they simply stopped buying new Windows systems because they'd be hacked during the setup & installation process.
posted by CheeseDigestsAll at 9:39 AM on March 8 [3 favorites]
During my tenure at Google (early 2000s), there was a period when they simply stopped buying new Windows systems because they'd be hacked during the setup & installation process.
posted by CheeseDigestsAll at 9:39 AM on March 8 [3 favorites]
> public institutions are often really um…penny wise, pound foolish about software, especially hidden, unglamorous, transactional software.
Eh. That, and we don't have the resources to modernize, often. Because people don't think we need it. We're just libraries, after all, and libraries totally stopped evolving in 1950, right?
Emphasized for MF TRUTH.
I've been hanging around my local library branch a lot more - it was A Way To Get Books when I was unemployed, and it's a place to hang out on the evenings when my roommate has telehealth appointments. Mefite SaharaRose is a friend and is also a librarian in Brooklyn's system (I'll put up the batsignal to see if she can come weigh in). A few months ago I was hearing from both SaharaRose and from my local branch about how the city was cutting some funding this year, and as a result, lots of branches are having to cut back their hours because they are stretched so thin they simply can't operate. Brooklyn's whole library system is doing away with Sunday hours - which sucks for the people who work 6 days a week.
And to add insult to injury - the libraries weren't the only ones getting their funding cut. Funding for school lunches was cut (because, apparently, the kids are "eating too much"). And lest you think that it was just schools and libraries getting funding cut, Mayor Adams was also planning to cut funding to the NYPD and FDNY. The cause of the budget gap, Mayor Adams says, is the city having to cope with caring for the refugees being bused up here by Ron DeSantis and Greg Abbott.
Speaking of those refugees, though - you know where a lot of them are getting access to different vital services, like legal advice, job training and language lessons? The fucking libraries. Social workers and legal aid society reps have walk-up hours in the libraries near the various migrant shelters, because that's the only place many of the migrants can find them. I'm at my local branch every Tuesday night, and a volunteer has been working with a couple of French speakers, teaching them English; I overhear them (it's a small space) every night when I'm there, as she patiently tries to teach them conversational English.
When the whole fucking library system is being used as a political pawn and it's being expected to be part of the social support system for the city's residents, and the mayor of your city is playing a game of financial chicken with the federal government, libraries have no CHOICE but to be "penny wise, pound foolish" when it comes to upgrading the computer system because they've got to make choices like "do we pay Microsoft to upgrade our network for only one year, or do we use that same money for the extra lawyer for the 800 Ukranians who just moved into the shelter last week?"
posted by EmpressCallipygos at 9:41 AM on March 8 [10 favorites]
Eh. That, and we don't have the resources to modernize, often. Because people don't think we need it. We're just libraries, after all, and libraries totally stopped evolving in 1950, right?
Emphasized for MF TRUTH.
I've been hanging around my local library branch a lot more - it was A Way To Get Books when I was unemployed, and it's a place to hang out on the evenings when my roommate has telehealth appointments. Mefite SaharaRose is a friend and is also a librarian in Brooklyn's system (I'll put up the batsignal to see if she can come weigh in). A few months ago I was hearing from both SaharaRose and from my local branch about how the city was cutting some funding this year, and as a result, lots of branches are having to cut back their hours because they are stretched so thin they simply can't operate. Brooklyn's whole library system is doing away with Sunday hours - which sucks for the people who work 6 days a week.
And to add insult to injury - the libraries weren't the only ones getting their funding cut. Funding for school lunches was cut (because, apparently, the kids are "eating too much"). And lest you think that it was just schools and libraries getting funding cut, Mayor Adams was also planning to cut funding to the NYPD and FDNY. The cause of the budget gap, Mayor Adams says, is the city having to cope with caring for the refugees being bused up here by Ron DeSantis and Greg Abbott.
Speaking of those refugees, though - you know where a lot of them are getting access to different vital services, like legal advice, job training and language lessons? The fucking libraries. Social workers and legal aid society reps have walk-up hours in the libraries near the various migrant shelters, because that's the only place many of the migrants can find them. I'm at my local branch every Tuesday night, and a volunteer has been working with a couple of French speakers, teaching them English; I overhear them (it's a small space) every night when I'm there, as she patiently tries to teach them conversational English.
When the whole fucking library system is being used as a political pawn and it's being expected to be part of the social support system for the city's residents, and the mayor of your city is playing a game of financial chicken with the federal government, libraries have no CHOICE but to be "penny wise, pound foolish" when it comes to upgrading the computer system because they've got to make choices like "do we pay Microsoft to upgrade our network for only one year, or do we use that same money for the extra lawyer for the 800 Ukranians who just moved into the shelter last week?"
posted by EmpressCallipygos at 9:41 AM on March 8 [10 favorites]
""Why not just restore from backup?" is a pretty naive question. "
Asking naive questions in ones area of expertise, what was that called again?
posted by joeyh at 10:03 AM on March 8 [1 favorite]
Asking naive questions in ones area of expertise, what was that called again?
posted by joeyh at 10:03 AM on March 8 [1 favorite]
Asking naive questions in ones area of expertise, what was that called again?
I call it a "best practice," actually, and also "keeping myself honest" and "how everyone learns." :7)
posted by wenestvedt at 10:38 AM on March 8 [10 favorites]
I call it a "best practice," actually, and also "keeping myself honest" and "how everyone learns." :7)
posted by wenestvedt at 10:38 AM on March 8 [10 favorites]
Ugh. I used to work in a university IT department and a ransomware attack was one of the things that really scared me. I knew we were vulnerable somewhere. It wasn't just a lack of money that made it harder for us to update and reconfigure our systems to be able to withstand an attack like this, it was a lack of time and expertise as well. We had backups that I thought maybe would have been good enough but we almost certainly missed something. You need a plan for how you're going to respond to a situation like this and ideally you need to practice that plan regularly. Similarly you need to periodically test your backups and test your backup processes. All of this is hard to find time for when you don't have enough staff and way too much work as it is just keeping the systems going. And, to make it worse, nobody wants to work in university IT for low pay when they could make more money in private industry so we had a hard time finding and keeping good people.
Also, it took the university a long time to realize that, like it or not, technology was a critical part of the organization and they had to stop thinking of it as something they could stick in a back closet and forget about.
I've only read the executive summary so far but I note that they say that they had copies of their digital collections but were unable to restore it because they didn't have viable infrastructure to restore it to. That reminds me of a story I heard from an IT guy at a research organization. The researchers were doing something that produced enormous amounts of data each day so they had several racks of drives to store that data. They also had offsite backups for the data. One day there was a water leak and one of the rack of drives was destroyed. They then realized that their backups were basically useless because a) they didn't have anywhere to restore it to and b) it would have taken something like a year to restore the data because the speed of the connection to the offsite backup location wasn't good enough. This stuff is really hard!
posted by fansler at 10:40 AM on March 8 [8 favorites]
Also, it took the university a long time to realize that, like it or not, technology was a critical part of the organization and they had to stop thinking of it as something they could stick in a back closet and forget about.
I've only read the executive summary so far but I note that they say that they had copies of their digital collections but were unable to restore it because they didn't have viable infrastructure to restore it to. That reminds me of a story I heard from an IT guy at a research organization. The researchers were doing something that produced enormous amounts of data each day so they had several racks of drives to store that data. They also had offsite backups for the data. One day there was a water leak and one of the rack of drives was destroyed. They then realized that their backups were basically useless because a) they didn't have anywhere to restore it to and b) it would have taken something like a year to restore the data because the speed of the connection to the offsite backup location wasn't good enough. This stuff is really hard!
posted by fansler at 10:40 AM on March 8 [8 favorites]
Adding to the chorus of librarians and folks who work in cultural institutions to say: This stuff is really hard. Underfunded cultural institutions, responsible for maintaining a huge repository of culturally important data... versus ransomware attack that could anytime from anywhere.
Especially to what @Humbug said:
Michael Taggart on Mastodon: "Ransomware also tends to be the bill collector for technical debt." Amazingly well-put.
And nothing like an institution that's been underfunded and more underfunded year over year, plus stretched thin by competing demands to showcase and maintain cultural relics, do community outreach... etc etc... and add in the expenses and challenge of maintaining tech infrastructure. (I promise I'm still talking about the British Library... and not any other city library experiencing this conundrum. Cough-cough NYC, Mayor Adams and his budget scissors. Bastard.)
New plan: when we see a nonprofit get hit by a ransomware attack, let's follow the money and follow the politics, and see where the technical debt has been racked up by chronic underfunding.
posted by SaharaRose at 11:01 AM on March 8 [6 favorites]
Especially to what @Humbug said:
Michael Taggart on Mastodon: "Ransomware also tends to be the bill collector for technical debt." Amazingly well-put.
And nothing like an institution that's been underfunded and more underfunded year over year, plus stretched thin by competing demands to showcase and maintain cultural relics, do community outreach... etc etc... and add in the expenses and challenge of maintaining tech infrastructure. (I promise I'm still talking about the British Library... and not any other city library experiencing this conundrum. Cough-cough NYC, Mayor Adams and his budget scissors. Bastard.)
New plan: when we see a nonprofit get hit by a ransomware attack, let's follow the money and follow the politics, and see where the technical debt has been racked up by chronic underfunding.
posted by SaharaRose at 11:01 AM on March 8 [6 favorites]
Yesterday one of the speakers raised a point that I found interesting: many organizations have an Incident Response Plan that handles things from Low to High severity...but they lack a plan for the truly harrowing problems.
During these crises, you have the least time for persuasion, explanation, and consideration -- and the greatest at stake in terms of organizational risk.
It's beforehand that you need to meet with the president and the general counsel, so that they know who you are when you call to ask to shut off the network or pay a ransomware gang. If it requires four hours of meetings for them to listen to you (if they even do!), then you probably missed your chance to save anything from the disaster.
What makes this kind of planning hard is that you have to consider the real Nightmare Scenarios, and also be bluntly honest about your exposure. And no one wants to admit to the CEO that some things simply can't be defended!
posted by wenestvedt at 11:03 AM on March 8 [5 favorites]
During these crises, you have the least time for persuasion, explanation, and consideration -- and the greatest at stake in terms of organizational risk.
It's beforehand that you need to meet with the president and the general counsel, so that they know who you are when you call to ask to shut off the network or pay a ransomware gang. If it requires four hours of meetings for them to listen to you (if they even do!), then you probably missed your chance to save anything from the disaster.
What makes this kind of planning hard is that you have to consider the real Nightmare Scenarios, and also be bluntly honest about your exposure. And no one wants to admit to the CEO that some things simply can't be defended!
posted by wenestvedt at 11:03 AM on March 8 [5 favorites]
I spent years working for a firm that had a team specialising in Koha, the open source ILS. One of the things I learned was that selling a new system is hard because every librarian of sufficiently long tenure has experienced one, possibly two migrations that were absolutely nightmarish, and they never ever want to do it again. Despite the fact that the team had a track record of smooth error-free cutovers with references, nobody ever really believed it until it happened. I am not surprised at all to learn that a large and long-lived institution had many systems never upgraded.
(Also there has been massive consolidation in the sector, with a few players buying up all the others. They offer support for the competitor systems they bought for a while but not forever, they want you to move to the one thing they are going forward with. Which means if there is functionality you can't live without, then you'll keep running your old system unsupported and hope).
posted by i_am_joe's_spleen at 11:52 AM on March 8 [4 favorites]
(Also there has been massive consolidation in the sector, with a few players buying up all the others. They offer support for the competitor systems they bought for a while but not forever, they want you to move to the one thing they are going forward with. Which means if there is functionality you can't live without, then you'll keep running your old system unsupported and hope).
posted by i_am_joe's_spleen at 11:52 AM on March 8 [4 favorites]
an untested continuity of ops plan is a good luck charm.
posted by j_curiouser at 12:59 PM on March 8 [1 favorite]
posted by j_curiouser at 12:59 PM on March 8 [1 favorite]
One day there was a water leak and one of the rack of drives was destroyed. They then realized that their backups were basically useless because a) they didn't have anywhere to restore it to and b) it would have taken something like a year to restore the data because the speed of the connection to the offsite backup location wasn't good enough. This stuff is really hard!
No matter how good your backup system is there is a adaptec SCSI card with a misconfigured out of date driver somewhere in your legacy hardware that will ruin your day/month/year. Rain is just being extra.
posted by srboisvert at 2:42 PM on March 8 [5 favorites]
No matter how good your backup system is there is a adaptec SCSI card with a misconfigured out of date driver somewhere in your legacy hardware that will ruin your day/month/year. Rain is just being extra.
posted by srboisvert at 2:42 PM on March 8 [5 favorites]
I spent 22 years working at a not for profit library software company, and i can tell you confirm two previous points
- librarians hate new software systems. Hate hate hate.
- they are sincerely doing the best they can with the funding they have.
Ok, three
- many of those libraries are running on custom, unsupported ILS’s because there is some fundamental function that they can’t live without.
posted by das_2099 at 4:40 PM on March 8 [8 favorites]
- librarians hate new software systems. Hate hate hate.
- they are sincerely doing the best they can with the funding they have.
Ok, three
- many of those libraries are running on custom, unsupported ILS’s because there is some fundamental function that they can’t live without.
posted by das_2099 at 4:40 PM on March 8 [8 favorites]
New plan: when we see a nonprofit get hit by a ransomware attack, let's follow the money and follow the politics, and see where the technical debt has been racked up by chronic underfunding.
But ‘where’ is everywhere. This is a universal problem. It’s every library. The ones you ‘see’ are just the ones that pulled the black marble. And there’s no mystery where the money is going, conservatives (in the US) are openly attacking and defunding libraries. So it’s either open and blatant political positioning or short-sighted administrative misprioritization. There are institutions with libraries with basically infinite amounts of money (eg Harvard). Are they all hardened? Probably not, but if not, it’s because an administrator doesn’t think it’s necessary, not because they don’t have enough money.
This isn’t a mystery. A quick google search for ‘library defunded’ will bring up a depressingly long list of news stories.
posted by bq at 8:20 AM on March 9 [4 favorites]
But ‘where’ is everywhere. This is a universal problem. It’s every library. The ones you ‘see’ are just the ones that pulled the black marble. And there’s no mystery where the money is going, conservatives (in the US) are openly attacking and defunding libraries. So it’s either open and blatant political positioning or short-sighted administrative misprioritization. There are institutions with libraries with basically infinite amounts of money (eg Harvard). Are they all hardened? Probably not, but if not, it’s because an administrator doesn’t think it’s necessary, not because they don’t have enough money.
This isn’t a mystery. A quick google search for ‘library defunded’ will bring up a depressingly long list of news stories.
posted by bq at 8:20 AM on March 9 [4 favorites]
Pretttttttty sure that's the point SaharaRose (who is another of MeFi's librarians) was making.
posted by humbug at 11:05 AM on March 9 [1 favorite]
posted by humbug at 11:05 AM on March 9 [1 favorite]
> librarians hate new software systems. Hate hate hate.
My mother was a corporate librarian starting in the 70s. She would actually name that as one of the things she liked about the job: that there was a big change in the technology just about every time she otherwise started getting bored, so there was always new things to learn.
Of course, the technology was changing a lot faster back then, and came with more obvious improvements for your trouble. It's also the case that "corporate librarian" is a very different niche than the typical librarian job. I'm not sure how much it even still exists. She retired when the company she worked for closed down its library.
posted by vibratory manner of working at 4:01 PM on March 9 [3 favorites]
My mother was a corporate librarian starting in the 70s. She would actually name that as one of the things she liked about the job: that there was a big change in the technology just about every time she otherwise started getting bored, so there was always new things to learn.
Of course, the technology was changing a lot faster back then, and came with more obvious improvements for your trouble. It's also the case that "corporate librarian" is a very different niche than the typical librarian job. I'm not sure how much it even still exists. She retired when the company she worked for closed down its library.
posted by vibratory manner of working at 4:01 PM on March 9 [3 favorites]
Corporate librarians still exist. I did a job with the American association of law librarians last year. Some were academic but most corporate.
posted by bq at 6:31 PM on March 9 [2 favorites]
posted by bq at 6:31 PM on March 9 [2 favorites]
Speaking as an IT worker who has been with some cash starved outfits and some cash flush outfits, I can say that footdragging on essential updates is endemic to organizations. There are some rare exceptions, but for the most part management and the users are virulently opposed to software updates and humbug is correct on the basis of that objection: they don't want to change workflows.
And I get that! No one likes to have their workflow changed, you commit it all to muscle memory and just blur along fast and smooth without any need for thought.
But we're not cruel and just making changes for grins. It's all necessary.
posted by sotonohito at 2:46 PM on March 10 [2 favorites]
And I get that! No one likes to have their workflow changed, you commit it all to muscle memory and just blur along fast and smooth without any need for thought.
But we're not cruel and just making changes for grins. It's all necessary.
posted by sotonohito at 2:46 PM on March 10 [2 favorites]
A double complication here is that demand for electronic resources (e-books and streaming libraries) went up sharply since COVID -- and many electronic items cost more than paper.
The publishers make super bad deals for e-books, like, "this can be checked out twelve times, but not concurrently, and then it expires. Also, it costs three times as much as retail. And if it's not used all twelve times in two years, it also expires."
So libraries are paying more than they would for comparable circulation of a printed item, and circulation is higher. Those costs add up quickly. And when a library is level-funded, those e-book deals can take precedence over upgrading the ILS, sadly.
(ObDisc: am IT guy and library trustee, and often I despair at what I see.)
posted by wenestvedt at 9:05 AM on March 11 [3 favorites]
The publishers make super bad deals for e-books, like, "this can be checked out twelve times, but not concurrently, and then it expires. Also, it costs three times as much as retail. And if it's not used all twelve times in two years, it also expires."
So libraries are paying more than they would for comparable circulation of a printed item, and circulation is higher. Those costs add up quickly. And when a library is level-funded, those e-book deals can take precedence over upgrading the ILS, sadly.
(ObDisc: am IT guy and library trustee, and often I despair at what I see.)
posted by wenestvedt at 9:05 AM on March 11 [3 favorites]
In my experience, I agree with the root cause being budget. Is there some foot dragging and moaning around changes? Yeah, for sure. No one likes change. But at least so far in my library, changes we can afford to make are made.
When budgets see barely maintenance growth year after year, or are cut outright, things don't get done. And this can be the cost of the solution itself (e.g.: software, licenses, external support), or just having enough staff to implement the solution (e.g.: does the library have internal IT? do they access the IT services of a parent org? do they have someone to champion and maintain a solution?).
posted by eekernohan at 11:49 AM on March 11 [2 favorites]
When budgets see barely maintenance growth year after year, or are cut outright, things don't get done. And this can be the cost of the solution itself (e.g.: software, licenses, external support), or just having enough staff to implement the solution (e.g.: does the library have internal IT? do they access the IT services of a parent org? do they have someone to champion and maintain a solution?).
posted by eekernohan at 11:49 AM on March 11 [2 favorites]
« Older R.I.P. Akira Toriyama, 1955-2024 | How do Dudes Pee? Newer »
This thread has been archived and is closed to new comments
posted by warriorqueen at 6:41 AM on March 8 [2 favorites]