Canada’s vital infrastructure disrupted by major telecoms outage
July 11, 2022 1:06 PM Subscribe
On July 8, Canada experienced a nation-wide telecommunications outage that brought many essential services to a halt, including emergency services, government services, and banking. Rogers Communications, which was responsible for the outage, is one of only a handful of telecom companies in Canada, and one of three that dominate the market. Rogers is currently pursuing a merger with Shaw Communications; Canada’s Competition Bureau has opposed the deal, which would further narrow Canada’s telecoms oligopoly. Meanwhile, Vass Bednar, executive director of McMaster University’s Master of Public Policy program, says it’s time to consider other options such as nationalizing telecoms infrastructure.
Justin Draper: Canada is three oligopolies in a trench coat
Justin Draper: Canada is three oligopolies in a trench coat
As a Canadian who was painfully without internet access for all of July 8 and doesn't like our lack of real competition in telecom, I'm sympathetic to doing something about this problem - but nationalizing the infrastructure doesn't make it immune to failure.
I don't believe there is any public, private, non-profit, angelic, or demonic entity that can get BGP right 100% of the time.
posted by allegedly at 1:19 PM on July 11, 2022 [7 favorites]
I don't believe there is any public, private, non-profit, angelic, or demonic entity that can get BGP right 100% of the time.
posted by allegedly at 1:19 PM on July 11, 2022 [7 favorites]
This a mere two weeks after a beaver took out internet and cell service for northern BC (and of course all debit transactions). I'm kinda surprised there wasn't a run on banks this morning with people pulling out cash for future emergencies.
posted by Mitheral at 1:22 PM on July 11, 2022 [4 favorites]
posted by Mitheral at 1:22 PM on July 11, 2022 [4 favorites]
allegedly, was it BGP again? It's always BGP. Freaking BGP.
posted by scolbath at 1:30 PM on July 11, 2022 [8 favorites]
posted by scolbath at 1:30 PM on July 11, 2022 [8 favorites]
Government IT work is always contracted out. The work would go to CGI, or IBM, or some such. And we know how that would turn out, so...while I'm definitely pro-nationalization in general, I don't think it would improve things.
Now I get to tell my tale of Friday woe. We left to drive from Edmonton to Grande Prairie on Friday afternoon (5 hour-ish drive). I wasn't terribly worried because I had heard that credit was working and ATMs were working. Well - all our banking is with TD, and for reasons I don't understand, the TD Visa wouldn't work anywhere. Non-TD ATMs didn't work. We tried to find a TD ATM, but being in rural Alberta, there wasn't one for another 2 or 3 hours. A kind person paid for my order at Tim's. We were only able to get $30 worth of gas, which was all the cash we had. When we finally landed at our motel, of course the card didn't work and this is when I began to panic. 9pm, two hungry kids, no money, no place to stay. Our last ditch effort was the TD bank ATM - success! But I was honestly beginning to panic.
I think I'm actually more pissed off at the banks than at Rogers. Why wasn't there clear communication from someone - anyone - about which payment options were working? I was checking reddit threads trying to figure it out, which was ridiculous. It should have been made clear that TD customers would have to go to a TD branch ATM (not even the TD-branded ATM inside the gas station worked). TD should have said that the TD Visa was not working. Someone could have come out and said - today is not a safe day to travel long distances with children unless you have a lot of cash in hand.
posted by kitcat at 1:42 PM on July 11, 2022 [17 favorites]
Now I get to tell my tale of Friday woe. We left to drive from Edmonton to Grande Prairie on Friday afternoon (5 hour-ish drive). I wasn't terribly worried because I had heard that credit was working and ATMs were working. Well - all our banking is with TD, and for reasons I don't understand, the TD Visa wouldn't work anywhere. Non-TD ATMs didn't work. We tried to find a TD ATM, but being in rural Alberta, there wasn't one for another 2 or 3 hours. A kind person paid for my order at Tim's. We were only able to get $30 worth of gas, which was all the cash we had. When we finally landed at our motel, of course the card didn't work and this is when I began to panic. 9pm, two hungry kids, no money, no place to stay. Our last ditch effort was the TD bank ATM - success! But I was honestly beginning to panic.
I think I'm actually more pissed off at the banks than at Rogers. Why wasn't there clear communication from someone - anyone - about which payment options were working? I was checking reddit threads trying to figure it out, which was ridiculous. It should have been made clear that TD customers would have to go to a TD branch ATM (not even the TD-branded ATM inside the gas station worked). TD should have said that the TD Visa was not working. Someone could have come out and said - today is not a safe day to travel long distances with children unless you have a lot of cash in hand.
posted by kitcat at 1:42 PM on July 11, 2022 [17 favorites]
Government IT work is always contracted out.
sure but it doesn’t have to be. you could hire people with experience and give them a budget to staff up a national isp. like the addiction to outside contractors is just a symptom of the same neoliberal policy regimes that also encourage oligopolies in markets. it’s not the only option
posted by dis_integration at 1:45 PM on July 11, 2022 [45 favorites]
sure but it doesn’t have to be. you could hire people with experience and give them a budget to staff up a national isp. like the addiction to outside contractors is just a symptom of the same neoliberal policy regimes that also encourage oligopolies in markets. it’s not the only option
posted by dis_integration at 1:45 PM on July 11, 2022 [45 favorites]
you could hire people with experience and give them a budget to staff up a national is
I mean, we should definitely do this. I'm just so cynical.
posted by kitcat at 1:55 PM on July 11, 2022 [4 favorites]
I mean, we should definitely do this. I'm just so cynical.
posted by kitcat at 1:55 PM on July 11, 2022 [4 favorites]
To add detail to the story this wasn't a typical (can't work for a bit outage) since both wireless and wired connections failed. That meant for entire areas businesses couldn't take card payments and also people couldn't take cash out of the ATM.
More critically 911 wasn't working for many people.
posted by artificialard at 2:10 PM on July 11, 2022 [5 favorites]
More critically 911 wasn't working for many people.
posted by artificialard at 2:10 PM on July 11, 2022 [5 favorites]
Cloudflare blog on the outage
Rogers hasn't said much about the root cause to confirm or deny, but the symptom that happened when the outage started was the sudden withdrawal of basically all BGP routes for the entire Rogers network.
posted by allegedly at 2:12 PM on July 11, 2022 [8 favorites]
Rogers hasn't said much about the root cause to confirm or deny, but the symptom that happened when the outage started was the sudden withdrawal of basically all BGP routes for the entire Rogers network.
posted by allegedly at 2:12 PM on July 11, 2022 [8 favorites]
Michael Geist has a great post on this:
Responding to the Rogers Outage: Time to Get Serious About Competition, Consumer Rights and Communications Regulation
Geist is the Canada Research Chair in Internet and E-commerce Law at the University of Ottawa.
posted by New Frontier at 2:15 PM on July 11, 2022 [9 favorites]
Responding to the Rogers Outage: Time to Get Serious About Competition, Consumer Rights and Communications Regulation
Geist is the Canada Research Chair in Internet and E-commerce Law at the University of Ottawa.
posted by New Frontier at 2:15 PM on July 11, 2022 [9 favorites]
Metafilter: demonic entity that can get BGP right 100% of the time.
posted by armoir from antproof case at 2:16 PM on July 11, 2022 [2 favorites]
posted by armoir from antproof case at 2:16 PM on July 11, 2022 [2 favorites]
The public explanations for this have just been mystifying: "a maintenance update in our core network" is what the Rogers CEO said triggered the failure. Either this just isn't true, or their "core network" is structured such that a "maintenance update" can take down everything Rogers sells in the entire country. It doesn't pass the smell test.
Anyways, I had stuff I was supposed to be doing, but we just checked out and went to a friend's cottage where there is no internet anyways, and my (Teksavvy, a reseller buying Rogers lines) internet was back by the time we returned to the world. Luckily we had enough cash for some beer at the depanneur on the way there (because every other way to pay was down!).
posted by advil at 2:16 PM on July 11, 2022 [2 favorites]
Anyways, I had stuff I was supposed to be doing, but we just checked out and went to a friend's cottage where there is no internet anyways, and my (Teksavvy, a reseller buying Rogers lines) internet was back by the time we returned to the world. Luckily we had enough cash for some beer at the depanneur on the way there (because every other way to pay was down!).
posted by advil at 2:16 PM on July 11, 2022 [2 favorites]
More critically 911 wasn't working for many people.
Even if it is working and the ambulance gets to your apartment building, what do you do when the buzzer in the lobby does not go to an intercom in your front hall but to a phone that -- sadness! -- has stopped working?
Luckily we had enough cash for some beer at the depanneur on the way there (because every other way to pay was down!).
Fortunately some small business customers are better prepared than a company with $14 billion in revenue every year.
Rogers proudly announces $10 service charge for “Unplug and Relax” nationwide outage [SLBeaverton]
I used to have my telecom needs addressed by Rogers. Used to.
posted by ricochet biscuit at 2:23 PM on July 11, 2022 [5 favorites]
Even if it is working and the ambulance gets to your apartment building, what do you do when the buzzer in the lobby does not go to an intercom in your front hall but to a phone that -- sadness! -- has stopped working?
Luckily we had enough cash for some beer at the depanneur on the way there (because every other way to pay was down!).
Fortunately some small business customers are better prepared than a company with $14 billion in revenue every year.
Rogers proudly announces $10 service charge for “Unplug and Relax” nationwide outage [SLBeaverton]
I used to have my telecom needs addressed by Rogers. Used to.
posted by ricochet biscuit at 2:23 PM on July 11, 2022 [5 favorites]
If it was BGP, which sounds plausible, this is the same problem that took down everything for Facebook last October (discussed here with some useful explanations).
posted by figurant at 2:27 PM on July 11, 2022 [3 favorites]
posted by figurant at 2:27 PM on July 11, 2022 [3 favorites]
There was also this stone-cold Beaverton headline: Rogers announces all customers affected by nationwide outage will receive free copy of Ted Rogers’ autobiography
Rogers once dimed a guy to the cops for joking about throwing a Ted Rogers statue in Lake Ontario.
But yeah, the head of the CRTC used to work for Rogers and Telus, so the cherry on the oligopoly shit sundae is regulatory capture.
posted by mandolin conspiracy at 2:28 PM on July 11, 2022 [10 favorites]
Rogers once dimed a guy to the cops for joking about throwing a Ted Rogers statue in Lake Ontario.
But yeah, the head of the CRTC used to work for Rogers and Telus, so the cherry on the oligopoly shit sundae is regulatory capture.
posted by mandolin conspiracy at 2:28 PM on July 11, 2022 [10 favorites]
Some of my friends on Rogers-provided Teksavvy internet connections still don't have service.
Canada is an odd little country. It would prefer homegrown price-gouging than any competition that might have a hint of not-Canadianness to it. The current chair of its telecoms regulator is the former SVP of government relations for TELUS, one of the three companies that own 90% of all communications infrastructure in Canada. There will be no change arising from this, because that would be against Peace, Order and Bloody Expensive Cell Phone Plans.
Rogers used to be a good Quaker company, too.
posted by scruss at 2:37 PM on July 11, 2022 [6 favorites]
Canada is an odd little country. It would prefer homegrown price-gouging than any competition that might have a hint of not-Canadianness to it. The current chair of its telecoms regulator is the former SVP of government relations for TELUS, one of the three companies that own 90% of all communications infrastructure in Canada. There will be no change arising from this, because that would be against Peace, Order and Bloody Expensive Cell Phone Plans.
Rogers used to be a good Quaker company, too.
posted by scruss at 2:37 PM on July 11, 2022 [6 favorites]
Someone could have come out and said - today is not a safe day to travel long distances with children unless you have a lot of cash in hand.
I think you hit a perfect intersection of telecoms oligopolies and banking oligopolies. My primary bank is BMO, and I use a BMO mastercard and this entire outage basically didn't affect me at all (I also live in Edmonton for context). I only know one person who uses Rogers for their cell service, apparently, and it was only that one guy who had an annoying day. Otherwise it was pretty nice patio weather on Whyte ave.
It was a reminder of how I've spent the last 2+ years living a largely cashless existence (physical cash was the first thing to go for the pandemic) and I just happened to luck out on which of the 3 big telecom companies and 5(?) banks I use. Otherwise I have maybe a twoonie to my name.
posted by selenized at 2:40 PM on July 11, 2022 [1 favorite]
I think you hit a perfect intersection of telecoms oligopolies and banking oligopolies. My primary bank is BMO, and I use a BMO mastercard and this entire outage basically didn't affect me at all (I also live in Edmonton for context). I only know one person who uses Rogers for their cell service, apparently, and it was only that one guy who had an annoying day. Otherwise it was pretty nice patio weather on Whyte ave.
It was a reminder of how I've spent the last 2+ years living a largely cashless existence (physical cash was the first thing to go for the pandemic) and I just happened to luck out on which of the 3 big telecom companies and 5(?) banks I use. Otherwise I have maybe a twoonie to my name.
posted by selenized at 2:40 PM on July 11, 2022 [1 favorite]
My own internet and cell service wasn't interrupted by the outage but I went to Ikea to purchase a new mattress and they couldn't take my debit card so I had to walk to a TD bank machine, withdrawal cash, and walk back. It was a pain but I cannot imagine how much money that single Ikea was losing that day. People were getting to the checkout with their carts full of goods and then just abandoning them and leaving when they were told about the outage. It was insane. Multiply that by the number of businesses affected and it's mind-boggling.
Bell and Rogers are two of the worst companies in the country and it angers me that so many people voluntarily use their services.
posted by dobbs at 2:41 PM on July 11, 2022 [1 favorite]
Bell and Rogers are two of the worst companies in the country and it angers me that so many people voluntarily use their services.
posted by dobbs at 2:41 PM on July 11, 2022 [1 favorite]
This is pretty face-palmy...
Canadian payments system Interac says adding backup network supplier after Rogers outage
Canadian online payments service Interac said on Monday it was adding another network provider to its system after an outage at Rogers Communications last week left millions of Canadians locked out of online payments.
"We are adding a supplier (besides Rogers) to strengthen our existing network redundancy so Canadians can continue to rely on Interac daily," Interac told Reuters in a statement.
Context: Interac is the network that acts as the intermediary between banks and merchants for debit and credit transactions in Canada. And also is the network for requesting or sending e-transfers. So...yeahhh.
posted by mandolin conspiracy at 2:46 PM on July 11, 2022 [8 favorites]
Canadian payments system Interac says adding backup network supplier after Rogers outage
Canadian online payments service Interac said on Monday it was adding another network provider to its system after an outage at Rogers Communications last week left millions of Canadians locked out of online payments.
"We are adding a supplier (besides Rogers) to strengthen our existing network redundancy so Canadians can continue to rely on Interac daily," Interac told Reuters in a statement.
Context: Interac is the network that acts as the intermediary between banks and merchants for debit and credit transactions in Canada. And also is the network for requesting or sending e-transfers. So...yeahhh.
posted by mandolin conspiracy at 2:46 PM on July 11, 2022 [8 favorites]
Not having redundant connectivity for your payments network is that kind of malpractice that might make some customers consider their legal options...
posted by i_am_joe's_spleen at 2:50 PM on July 11, 2022 [14 favorites]
posted by i_am_joe's_spleen at 2:50 PM on July 11, 2022 [14 favorites]
More critically 911 wasn't working for many people.
Were folks able to fall back to landlines for critical services?
It sounds like this wasn't the first time, either. To the extent that having functioning telecommunications infrastructure is obviously vital to national security and the wellbeing of the public, I am surprised at pushback on taking over that role from those who had been gifted this responsibility and failed (again).
posted by They sucked his brains out! at 3:01 PM on July 11, 2022
Were folks able to fall back to landlines for critical services?
It sounds like this wasn't the first time, either. To the extent that having functioning telecommunications infrastructure is obviously vital to national security and the wellbeing of the public, I am surprised at pushback on taking over that role from those who had been gifted this responsibility and failed (again).
posted by They sucked his brains out! at 3:01 PM on July 11, 2022
Either this just isn't true, or their "core network" is structured such that a "maintenance update" can take down everything Rogers sells in the entire country. It doesn't pass the smell test.
If their 'maintenance' update, instead of updating their published BGP routes instead erased them, which is what sounds like happened, it absolutely would take Rogers offline entirely.
It's not an amazing analogy, but if your IP address was your house address, BGP is the constantly updating map of how to get there, partly based on traffic conditions. By the sounds of it, Rogers basically told the rest of the internet 'we no longer exist, please forget about how to find us or any of our customers', and since that is in fact a core feature of BGP to promptly stop traffic getting sent down dead ends - that's exactly what they did.
So now you've a) got to fix the problem in the first place and b) wait for the rest of the internet to realise you do now exist again to send you website traffic (you ask for a webpage, the server has to know where to send the answer to the next place that will get it closer to you), which takes time to propagate out. BGP is an *extremely* powerful tool that basically allows a network of networks the size of the current internet managed by so many different companies to even function at all. But you fuckup, and it makes a hell of a mess. Worse, it doesn't even have to be your fuckup, someone else's fuckup can take your ISP or other large network down if they start saying that your traffic should go to them instead - the whole thing largely runs on trust that BGP admins know What They Are Doing, as a number of companies have found out the hard way. It's amazing any of it works at all, to be honest.
TL;DR - BGP is the thing network admins have nightmares about. That, and DNS.
Another big fuckup here was that the payment networks didn't appear to be redundant, i.e. use more than one internet provider. THAT'S just poor planning.
posted by Absolutely No You-Know-What at 3:09 PM on July 11, 2022 [17 favorites]
If their 'maintenance' update, instead of updating their published BGP routes instead erased them, which is what sounds like happened, it absolutely would take Rogers offline entirely.
It's not an amazing analogy, but if your IP address was your house address, BGP is the constantly updating map of how to get there, partly based on traffic conditions. By the sounds of it, Rogers basically told the rest of the internet 'we no longer exist, please forget about how to find us or any of our customers', and since that is in fact a core feature of BGP to promptly stop traffic getting sent down dead ends - that's exactly what they did.
So now you've a) got to fix the problem in the first place and b) wait for the rest of the internet to realise you do now exist again to send you website traffic (you ask for a webpage, the server has to know where to send the answer to the next place that will get it closer to you), which takes time to propagate out. BGP is an *extremely* powerful tool that basically allows a network of networks the size of the current internet managed by so many different companies to even function at all. But you fuckup, and it makes a hell of a mess. Worse, it doesn't even have to be your fuckup, someone else's fuckup can take your ISP or other large network down if they start saying that your traffic should go to them instead - the whole thing largely runs on trust that BGP admins know What They Are Doing, as a number of companies have found out the hard way. It's amazing any of it works at all, to be honest.
TL;DR - BGP is the thing network admins have nightmares about. That, and DNS.
Another big fuckup here was that the payment networks didn't appear to be redundant, i.e. use more than one internet provider. THAT'S just poor planning.
posted by Absolutely No You-Know-What at 3:09 PM on July 11, 2022 [17 favorites]
A youtube video showing the routing links disappearing, in real time, as Rogers drops off the Internet. I imagine that this is a scary video for internet router admins!
Major Rogers BGP Outage 2022/07/08
From a reddit discussion of the video:
"I’m not a network expert but BGP is one of the most common layer 3 (routing) protocols. Each one of those numbered circles are routers. Each line that connects them is a path that data can take to get from point A to point B. As more and more of the “connections” fail there is literally no way for data to get across the Rogers network."
I don't know any details of how BGP (Border Gateway Protocol) works, but this thrashing of links and gradual loss of connection is an interesting view. I think the moving lines are attempting to reroute the connection between two router nodes, and a flashing connection line shows it being dropped. By the end, only two routers are connected together, no other connection links remain.
posted by jjj606 at 3:11 PM on July 11, 2022 [10 favorites]
Major Rogers BGP Outage 2022/07/08
From a reddit discussion of the video:
"I’m not a network expert but BGP is one of the most common layer 3 (routing) protocols. Each one of those numbered circles are routers. Each line that connects them is a path that data can take to get from point A to point B. As more and more of the “connections” fail there is literally no way for data to get across the Rogers network."
I don't know any details of how BGP (Border Gateway Protocol) works, but this thrashing of links and gradual loss of connection is an interesting view. I think the moving lines are attempting to reroute the connection between two router nodes, and a flashing connection line shows it being dropped. By the end, only two routers are connected together, no other connection links remain.
posted by jjj606 at 3:11 PM on July 11, 2022 [10 favorites]
It sounds like this wasn't the first time, either.
The Rogering will continue until morale improves.
posted by mandolin conspiracy at 3:11 PM on July 11, 2022 [28 favorites]
The Rogering will continue until morale improves.
posted by mandolin conspiracy at 3:11 PM on July 11, 2022 [28 favorites]
Context: Interac is the network that acts as the intermediary between banks and merchants for debit and credit transactions in Canada. And also is the network for requesting or sending e-transfers. So...yeahhh.
Interac is also the company that charges you a service charge every time you spend your own money via electronic payment. I noticed that during the financial crisis that accompanied the lockdown when many organizations were suspending invoicing as millions saw their jobs evaporate, Interac had nothing to say on the windfall they were seeing.
As I mentioned elsewhere on the site, in March I was briefly a customer of my former employer and learned that they now no longer accept cash (indeed, there is no cash in the building save what people might have in their pockets). I wonder if last Friday’s events will occasion a rethinking of that policy.
posted by ricochet biscuit at 3:12 PM on July 11, 2022 [3 favorites]
Interac is also the company that charges you a service charge every time you spend your own money via electronic payment. I noticed that during the financial crisis that accompanied the lockdown when many organizations were suspending invoicing as millions saw their jobs evaporate, Interac had nothing to say on the windfall they were seeing.
As I mentioned elsewhere on the site, in March I was briefly a customer of my former employer and learned that they now no longer accept cash (indeed, there is no cash in the building save what people might have in their pockets). I wonder if last Friday’s events will occasion a rethinking of that policy.
posted by ricochet biscuit at 3:12 PM on July 11, 2022 [3 favorites]
(Teksavvy, a reseller buying Rogers lines)
I have Teksavvy, too, and it didn't go down at all for me. Maybe Teksavvy resells different providers in different places?
Speakout Wireless, on the other hand, which resells Rogers wireless, was down all day for me.
I assume that my Bell home phone worked, but I don't think I picked it up at any point during the day.
posted by clawsoon at 3:13 PM on July 11, 2022
I have Teksavvy, too, and it didn't go down at all for me. Maybe Teksavvy resells different providers in different places?
Speakout Wireless, on the other hand, which resells Rogers wireless, was down all day for me.
I assume that my Bell home phone worked, but I don't think I picked it up at any point during the day.
posted by clawsoon at 3:13 PM on July 11, 2022
I liked whomever said that the Rogers network is basically made from Twizzlers.
My main personal concern is that I couldn't withdraw any money at the ATM of the bank I use. I mean, if ScotiaBank owns Tangerine, I should be able to use my g-d debit card to get cash out in an emergency. I couldn't!
posted by Kitteh at 3:14 PM on July 11, 2022 [1 favorite]
My main personal concern is that I couldn't withdraw any money at the ATM of the bank I use. I mean, if ScotiaBank owns Tangerine, I should be able to use my g-d debit card to get cash out in an emergency. I couldn't!
posted by Kitteh at 3:14 PM on July 11, 2022 [1 favorite]
I mean, if ScotiaBank owns Tangerine, I should be able to use my g-d debit card to get cash out in an emergency. I couldn't!
Look, if they wanted to invest some of their record profits in redundancy, that would be absolutely...
...oh, right.
I'm just so cynical.
Right? It's hard not to be when this is a regulated industry that could be upended overnight if the political will existed. Anything makes more sense than the current state of things.
This gets me thinking about things like Tbaytel - it's owned by the city of Thunder Bay, and pays out an annual dividend that goes to the city. Last year it was a little over $21 million. I mean, better a municipality than the goddamn Rogers Family Control Trust (pdf).
posted by mandolin conspiracy at 3:22 PM on July 11, 2022 [7 favorites]
Look, if they wanted to invest some of their record profits in redundancy, that would be absolutely...
...oh, right.
I'm just so cynical.
Right? It's hard not to be when this is a regulated industry that could be upended overnight if the political will existed. Anything makes more sense than the current state of things.
This gets me thinking about things like Tbaytel - it's owned by the city of Thunder Bay, and pays out an annual dividend that goes to the city. Last year it was a little over $21 million. I mean, better a municipality than the goddamn Rogers Family Control Trust (pdf).
posted by mandolin conspiracy at 3:22 PM on July 11, 2022 [7 favorites]
A youtube video showing the routing links disappearing, in real time, as Rogers drops off the Internet. I imagine that this is a scary video for internet router admins!
Yup, seeing that is definitely the sort of thing you know is causing a brown trouser day for someone. You just hope it ain't you.
It's seeing all the routes that go to and via AS812 - which is Rogers. At the start of the video, there are a lot of links between them and other network providers (the other numbers), as you'd expect for a major ISP - you want to send traffic by the right route (a big direct connection is best, but sometimes you're better off sending stuff via a big longer route than a congested direct one). And all those connections start dropping, one by one (the flashing). As one goes down, it's showing the re-routing in real time - uhoh, can't go that way anymore, lets go this way instead. As more go down, you see multiple routes getting moved around the rapidly diminishing options (it takes time for updates to propagate out). Then some routes simply become impossible, and at the end, Rogers isn't connected to anyone. Oops.
posted by Absolutely No You-Know-What at 3:24 PM on July 11, 2022 [3 favorites]
Yup, seeing that is definitely the sort of thing you know is causing a brown trouser day for someone. You just hope it ain't you.
It's seeing all the routes that go to and via AS812 - which is Rogers. At the start of the video, there are a lot of links between them and other network providers (the other numbers), as you'd expect for a major ISP - you want to send traffic by the right route (a big direct connection is best, but sometimes you're better off sending stuff via a big longer route than a congested direct one). And all those connections start dropping, one by one (the flashing). As one goes down, it's showing the re-routing in real time - uhoh, can't go that way anymore, lets go this way instead. As more go down, you see multiple routes getting moved around the rapidly diminishing options (it takes time for updates to propagate out). Then some routes simply become impossible, and at the end, Rogers isn't connected to anyone. Oops.
posted by Absolutely No You-Know-What at 3:24 PM on July 11, 2022 [3 favorites]
Oh, and - the mayor of Canada's largest city (from 2021): Toronto mayor defends role in Rogers family trust amid boardroom drama
Also the mayor of Toronto's largest city, from a TV Ontario journalist: "Can't help but note that in May, Toronto City Council's executive (led and handpicked by John Tory) decided that Toronto definitely didn't need any kind of publicly-owned alternative to the telecom oligopoly."
posted by mandolin conspiracy at 3:28 PM on July 11, 2022 [7 favorites]
Also the mayor of Toronto's largest city, from a TV Ontario journalist: "Can't help but note that in May, Toronto City Council's executive (led and handpicked by John Tory) decided that Toronto definitely didn't need any kind of publicly-owned alternative to the telecom oligopoly."
posted by mandolin conspiracy at 3:28 PM on July 11, 2022 [7 favorites]
nationalizing the infrastructure doesn't make it immune to failure.
Nationalize then decentralize.
posted by mhoye at 3:48 PM on July 11, 2022 [1 favorite]
Nationalize then decentralize.
posted by mhoye at 3:48 PM on July 11, 2022 [1 favorite]
sure but it doesn’t have to be. you could hire people with experience and give them a budget to staff up a national isp
I’m not sure right now is the moment to suggest that the federal government adds to the services it provides. They can barely deliver passports, process unemployment insurance & etc in reasonable time, there’s just some air of nonchalance about the non delivery of basic services that doesn’t make me want for them to be in charge of my internet.
And in principle I think internet access is now just as crucial as water or electricity and should either be a public service or closely regulated (price & all) to be as close to one as possible.
But not by the federal government, the distance between Ottawa & anybody else in this country sometimes makes me believe our gov exists in parallel dimensions and only occasionally bothers to check on the country.
posted by WaterAndPixels at 4:43 PM on July 11, 2022 [1 favorite]
I’m not sure right now is the moment to suggest that the federal government adds to the services it provides. They can barely deliver passports, process unemployment insurance & etc in reasonable time, there’s just some air of nonchalance about the non delivery of basic services that doesn’t make me want for them to be in charge of my internet.
And in principle I think internet access is now just as crucial as water or electricity and should either be a public service or closely regulated (price & all) to be as close to one as possible.
But not by the federal government, the distance between Ottawa & anybody else in this country sometimes makes me believe our gov exists in parallel dimensions and only occasionally bothers to check on the country.
posted by WaterAndPixels at 4:43 PM on July 11, 2022 [1 favorite]
Tangerine!! The Scotiabank ATM let me get all the way through the menu, including picking the amount I wanted to withdraw and only then did it give me an error message. I had enough cash already on me to get toilet paper, but that was about it.
The company I work for lost an entire day of work, as all of our storefronts, satellite locations, warehouses and shipping/distribution hubs use Rogers for internet, wifi and employee cell phones. We are still playing catch-up. It is terrible.
posted by janepanic at 4:50 PM on July 11, 2022 [3 favorites]
The company I work for lost an entire day of work, as all of our storefronts, satellite locations, warehouses and shipping/distribution hubs use Rogers for internet, wifi and employee cell phones. We are still playing catch-up. It is terrible.
posted by janepanic at 4:50 PM on July 11, 2022 [3 favorites]
The All Your Eggs in One Basket plan from Rogers [YouTube by brittlestar].
posted by heatherlogan at 5:00 PM on July 11, 2022 [5 favorites]
posted by heatherlogan at 5:00 PM on July 11, 2022 [5 favorites]
Were folks able to fall back to landlines for critical services?
Increasingly, even copper landlines are moving to IP on the backend as the TDM switches are slowly replaced.
posted by wierdo at 5:26 PM on July 11, 2022
Increasingly, even copper landlines are moving to IP on the backend as the TDM switches are slowly replaced.
posted by wierdo at 5:26 PM on July 11, 2022
Also the mayor of Toronto's largest city, from a TV Ontario journalist: "Can't help but note that in May, Toronto City Council's executive (led and handpicked by John Tory) decided that Toronto definitely didn't need any kind of publicly-owned alternative to the telecom oligopoly."
John Tory, you say? That is an amazing coincidence, as the former president and CEO of Rogers Media was also a Toronto lawyer named John Tory. Say, do you think these two have ever met? I bet they'd have a lot to talk about and a good laugh about the whole situation.
posted by ricochet biscuit at 5:54 PM on July 11, 2022 [20 favorites]
John Tory, you say? That is an amazing coincidence, as the former president and CEO of Rogers Media was also a Toronto lawyer named John Tory. Say, do you think these two have ever met? I bet they'd have a lot to talk about and a good laugh about the whole situation.
posted by ricochet biscuit at 5:54 PM on July 11, 2022 [20 favorites]
This is weird but I happen to keep several hundred dollars in cash at home, and at least a hundred in my wallet. Recently I started thinking - what am I doing? The cash has just been sitting there unused for months. The cash in my wallet was making a big bulge in my pants pocket, which was uncomfortable. I started to spend down the bills.
Then Friday happened.
Now I don't mind the pants bulge so much.
Anyhow, the lesson I learned is that there's no point in waiting for an emergency to withdraw cash. Emergencies by definition now seem to be "can't buy things without cash" combined with "can't get cash".
Thinking about it, I must have been insane to even consider running down my stash of cash. In February we had the Clown Convoy occupation here in Ottawa, so banks were closed downtown. In May we had the derecho, with power outages lasting up to 14 days. And now this.
I'm going to glue together a coat of Twenties, put it on, walk around, relaxed.
posted by storybored at 6:32 PM on July 11, 2022 [8 favorites]
Then Friday happened.
Now I don't mind the pants bulge so much.
Anyhow, the lesson I learned is that there's no point in waiting for an emergency to withdraw cash. Emergencies by definition now seem to be "can't buy things without cash" combined with "can't get cash".
Thinking about it, I must have been insane to even consider running down my stash of cash. In February we had the Clown Convoy occupation here in Ottawa, so banks were closed downtown. In May we had the derecho, with power outages lasting up to 14 days. And now this.
I'm going to glue together a coat of Twenties, put it on, walk around, relaxed.
posted by storybored at 6:32 PM on July 11, 2022 [8 favorites]
I'm going to glue together a coat of Twenties, put it on, walk around, relaxed.
You could call it the Ottawa Overcoat. The National Capital Region's answer to the Sudbury Dinner Jacket.
posted by mandolin conspiracy at 7:17 PM on July 11, 2022 [5 favorites]
You could call it the Ottawa Overcoat. The National Capital Region's answer to the Sudbury Dinner Jacket.
posted by mandolin conspiracy at 7:17 PM on July 11, 2022 [5 favorites]
It's not super clear to me whether the bgp thing is the root cause, or just a symptom that was visible from outside.
It's shockingly common for giant distributed systems to fail due to bad configuration updates...
posted by kaibutsu at 8:14 PM on July 11, 2022 [3 favorites]
It's shockingly common for giant distributed systems to fail due to bad configuration updates...
posted by kaibutsu at 8:14 PM on July 11, 2022 [3 favorites]
Is anyone discussing the (laughably unlikely) option of breaking up the Canadian Big 3s networks from the service provision? The problem is that the network is a natural monopoly, and the 'service' is where the money is. Other countries that are serious about managed capitalism (e.g. Sweden) always have the network operator a separate company who charge all service providers the same access fee. Presto, competitipn and lower prices.
posted by anthill at 9:01 PM on July 11, 2022 [2 favorites]
posted by anthill at 9:01 PM on July 11, 2022 [2 favorites]
My ISP piggybacks on Rogers. Of course it went down. Unlike Rogers customers, however, who will be getting an automatic refund (of whatever nominal amount that is), I have to call customer service to request mine, but I have to wait a few days so they can figure out how it will all work--at least that's what the tech rep told me on Saturday after I had to call in to restore my connectivity (because apparently the outage wiped out my network passwords from the ISP's database).
The big problem I was having on Friday, however, was a Bell one. Fine Rogers was down, but my cell phone is on Bell Mobility. And I couldn't access the Internet via it either. Eventually in the early part of the afternoon, I started getting Slack messages from work, but it wasn't until later afternoon that I could use my phone to surf the Web. I'm sure if I complain, however, Bell will just throw up its hands and say, "huh, we didn't have a problem."
I actually have a complaint to make to the CCTS about Rogers and I'm trying to figure out if it's worth making now (to pile on the company) or waiting a bit (after this whole outage mess settles down). I know I'm just one measly person (not even a Rogers customer) but if there is any teeny tiny monkey wrench I can throw that will help derail the Shaw takeover I'd be so happy.
Rogers keeps calling and texting me to let me know about bill payments that I've made or bill payments that I've missed. The problem is I'm not actually a Rogers customer. I don't have any Rogers accounts and I haven't since I've owned my cell phone. I've texted Rogers. I've spoken with their customer service multiple times, and each time I've been promised my number will be deleted from this other woman's account and then the next month rolls around and they're back to contacting me about it again. Now, it's time to sic the CCTS on them. I can say, from experience, telecom companies really don't like having the CCTS involved in disputes, and it tends to snap them into action much more quickly than just a regular complaint does.
posted by sardonyx at 9:01 PM on July 11, 2022 [1 favorite]
The big problem I was having on Friday, however, was a Bell one. Fine Rogers was down, but my cell phone is on Bell Mobility. And I couldn't access the Internet via it either. Eventually in the early part of the afternoon, I started getting Slack messages from work, but it wasn't until later afternoon that I could use my phone to surf the Web. I'm sure if I complain, however, Bell will just throw up its hands and say, "huh, we didn't have a problem."
I actually have a complaint to make to the CCTS about Rogers and I'm trying to figure out if it's worth making now (to pile on the company) or waiting a bit (after this whole outage mess settles down). I know I'm just one measly person (not even a Rogers customer) but if there is any teeny tiny monkey wrench I can throw that will help derail the Shaw takeover I'd be so happy.
Rogers keeps calling and texting me to let me know about bill payments that I've made or bill payments that I've missed. The problem is I'm not actually a Rogers customer. I don't have any Rogers accounts and I haven't since I've owned my cell phone. I've texted Rogers. I've spoken with their customer service multiple times, and each time I've been promised my number will be deleted from this other woman's account and then the next month rolls around and they're back to contacting me about it again. Now, it's time to sic the CCTS on them. I can say, from experience, telecom companies really don't like having the CCTS involved in disputes, and it tends to snap them into action much more quickly than just a regular complaint does.
posted by sardonyx at 9:01 PM on July 11, 2022 [1 favorite]
Another big fuckup here was that the payment networks didn't appear to be redundant, i.e. use more than one internet provider. THAT'S just poor planning.
How much of this was that, though? My mental model of the 2022 electronic payment system is merchant - > (the internet) - > financial company and vice versa. And my understanding of this problem was that for Rogers users, (the internet) became (yelling into an empty well) in terms of functionality.
So in that case, don't both ends of the transaction need to be redundantly connected? For a financial data centre, obviously that should be redundantly connected. Hell, get a copper landline to Korea or Finland or somewhere competent as another backup. But for the merchant? I guess a bank branch (and their ATMs) makes sense for dual connection; maybe a big single point like an IKEA. Ideally, a supermarket would be more robust just because groceries can't wait. But every corner coffee shop or food truck?
Or is my understanding of this not correct?
posted by Superilla at 10:36 PM on July 11, 2022
How much of this was that, though? My mental model of the 2022 electronic payment system is merchant - > (the internet) - > financial company and vice versa. And my understanding of this problem was that for Rogers users, (the internet) became (yelling into an empty well) in terms of functionality.
So in that case, don't both ends of the transaction need to be redundantly connected? For a financial data centre, obviously that should be redundantly connected. Hell, get a copper landline to Korea or Finland or somewhere competent as another backup. But for the merchant? I guess a bank branch (and their ATMs) makes sense for dual connection; maybe a big single point like an IKEA. Ideally, a supermarket would be more robust just because groceries can't wait. But every corner coffee shop or food truck?
Or is my understanding of this not correct?
posted by Superilla at 10:36 PM on July 11, 2022
I wasn't in Canada, and the question struck me: was even regular, over the airways TV and/or radio even working? I kind of assume that all that stuff is run on the back-end through the internet.
posted by LizBoBiz at 3:05 AM on July 12, 2022
posted by LizBoBiz at 3:05 AM on July 12, 2022
How much of this was that, though? My mental model of the 2022 electronic payment system is merchant - > (the internet) - > financial company and vice versa. And my understanding of this problem was that for Rogers users, (the internet) became (yelling into an empty well) in terms of functionality.
That isn't my area of expertise, but I have set up UK POS systems. I believe it goes card terminal -> internet -> terminal provider (commonly merchant bank A) -> interbank network, e.g. VISA -> customer's bank B; I presume Canada works similarly. Bank B responds if there's enough money (or not), puts a temporary hold on the money, so the merchant can now complete the transaction and give you your stuff. At a later point, bank A and B settle outstanding transactions in bulk (so they only have to do one big payment instead of tons of small ones), and a finalised payment ends up in merchant's account, and a debit in the customers. If the transaction is cancelled/doesn't complete, the hold just expires. Contactless payments can skip the authorisation step for small transactions in some cases I believe. (so if a card is reported stolen, contactless can continue to work for a bit, though the bank should cover the losses)
In that setup, if the merchant's internet connection is down because they use Rogers, then yeah, no card payments for them. From what I've seen though, some cards were working and others not; so that's down to the interbank network or bank B (or both) being offline. Everything after the merchant connection should be going to big datacentres, and have substantial redundant internet connections, probably managed by dedicated teams. That should not have failed just because one ISP went down (albeit a big one), but appears that it did for some banks.
Large merchants and main bank branches should also have robust connections; they send a lot of traffic to and from their WAN for security/internal business, and extended downtime is major losses. A hole in the wall ATM or small shop, not so much.
Of course, redundancy costs money, vs the theoretical risk of significant losses if things go pear shaped. It wouldn't be the first time banks picked short term savings over mitigating risk.
posted by Absolutely No You-Know-What at 3:35 AM on July 12, 2022 [2 favorites]
That isn't my area of expertise, but I have set up UK POS systems. I believe it goes card terminal -> internet -> terminal provider (commonly merchant bank A) -> interbank network, e.g. VISA -> customer's bank B; I presume Canada works similarly. Bank B responds if there's enough money (or not), puts a temporary hold on the money, so the merchant can now complete the transaction and give you your stuff. At a later point, bank A and B settle outstanding transactions in bulk (so they only have to do one big payment instead of tons of small ones), and a finalised payment ends up in merchant's account, and a debit in the customers. If the transaction is cancelled/doesn't complete, the hold just expires. Contactless payments can skip the authorisation step for small transactions in some cases I believe. (so if a card is reported stolen, contactless can continue to work for a bit, though the bank should cover the losses)
In that setup, if the merchant's internet connection is down because they use Rogers, then yeah, no card payments for them. From what I've seen though, some cards were working and others not; so that's down to the interbank network or bank B (or both) being offline. Everything after the merchant connection should be going to big datacentres, and have substantial redundant internet connections, probably managed by dedicated teams. That should not have failed just because one ISP went down (albeit a big one), but appears that it did for some banks.
Large merchants and main bank branches should also have robust connections; they send a lot of traffic to and from their WAN for security/internal business, and extended downtime is major losses. A hole in the wall ATM or small shop, not so much.
Of course, redundancy costs money, vs the theoretical risk of significant losses if things go pear shaped. It wouldn't be the first time banks picked short term savings over mitigating risk.
posted by Absolutely No You-Know-What at 3:35 AM on July 12, 2022 [2 favorites]
This morning on the radio somebody proposed forcing the carriers to allow customers on to each other's networks in the event of an outage like this.
I immediately imagined that the outcome of a setup like that would be an outage that would take out all the providers at the same time.
posted by clawsoon at 4:25 AM on July 12, 2022 [1 favorite]
I immediately imagined that the outcome of a setup like that would be an outage that would take out all the providers at the same time.
posted by clawsoon at 4:25 AM on July 12, 2022 [1 favorite]
(waterloo) I was lucky. didn't know what was happening, went to get an egg mcmuffin on the way to work (craving, shut up) and noted that the line was weirdly almost nonexistent and that people in front of me kept driving away from the order kiosk and out. I had a fiver in my wallet and that covered me but it didnt occur to me to ask and I drove to the office.
Got to the office kitchen and there's 20 people standing around going "internet's down". Engineering consultancy, we need computers for everything, and our voip phones, cell phones and network were all effed (teksavvy corporate servs). Even then I figured, well, they always come back after 15 mins, glad I am not that guy. Every now and again someone would give our backup IT resource a bit of shit (amazingly our main guy was on vac that day) but we mostly just futzed around. For whatever reason I didn't have any meetings so I spent most of the day just filing emails and blowing old files off my desktop. (I did have one internal call, and I ended up talking to that guy on saturday and explained that, yeah, all our comms were in a crater. I realised after a bit that he was irked with me like I could have done more but when I finally said "you couldn't see my smoke signals from pittsburgh, dude" he eventualy realized I was serious.
A secondary conversation around our home is that my wife insists on keeping a hard landline "for emergencies" and it occured to me that she wouldn't have been able to call me, her parents, or either of her children. I do wonder if we could have gotten through to 911 or the hospital? Happily we did not have to find out. Definitely a "some days society is a pretty thin veneer" kind of day...last time I felt like that was the hydro outage in 2003.
posted by hearthpig at 5:18 AM on July 12, 2022 [1 favorite]
Got to the office kitchen and there's 20 people standing around going "internet's down". Engineering consultancy, we need computers for everything, and our voip phones, cell phones and network were all effed (teksavvy corporate servs). Even then I figured, well, they always come back after 15 mins, glad I am not that guy. Every now and again someone would give our backup IT resource a bit of shit (amazingly our main guy was on vac that day) but we mostly just futzed around. For whatever reason I didn't have any meetings so I spent most of the day just filing emails and blowing old files off my desktop. (I did have one internal call, and I ended up talking to that guy on saturday and explained that, yeah, all our comms were in a crater. I realised after a bit that he was irked with me like I could have done more but when I finally said "you couldn't see my smoke signals from pittsburgh, dude" he eventualy realized I was serious.
A secondary conversation around our home is that my wife insists on keeping a hard landline "for emergencies" and it occured to me that she wouldn't have been able to call me, her parents, or either of her children. I do wonder if we could have gotten through to 911 or the hospital? Happily we did not have to find out. Definitely a "some days society is a pretty thin veneer" kind of day...last time I felt like that was the hydro outage in 2003.
posted by hearthpig at 5:18 AM on July 12, 2022 [1 favorite]
I have Teksavvy, too, and it didn't go down at all for me. Maybe Teksavvy resells different providers in different places?
Yes, mine's on Bell.
posted by dobbs at 5:29 AM on July 12, 2022
Yes, mine's on Bell.
posted by dobbs at 5:29 AM on July 12, 2022
This morning on the radio somebody proposed forcing the carriers to allow customers on to each other's networks in the event of an outage like this.
I immediately imagined that the outcome of a setup like that would be an outage that would take out all the providers at the same time.
We may just get to find out:
Ottawa announces it will require telecoms to provide backup for each other during outages following Rogers system failure
posted by rodlymight at 6:06 AM on July 12, 2022
I immediately imagined that the outcome of a setup like that would be an outage that would take out all the providers at the same time.
We may just get to find out:
Ottawa announces it will require telecoms to provide backup for each other during outages following Rogers system failure
posted by rodlymight at 6:06 AM on July 12, 2022
^yup absolutely nothing can go wrong with this plan yup^
posted by Kitteh at 6:15 AM on July 12, 2022 [1 favorite]
posted by Kitteh at 6:15 AM on July 12, 2022 [1 favorite]
In order for them to provide backup for each other, they would have to have BGP routes ready (and tested) in some combination of "X / Y / Z" is unavailable, so the "other 2 (or 1)" need to take up the slack.
Not going to happen. Testing their BGP routes would essentially take-down connections to the other players - so, this will be untested until an outage occurs. And then... Given the complexity of BPG, they will likely just end-up making things worse...
posted by rozcakj at 6:19 AM on July 12, 2022
Not going to happen. Testing their BGP routes would essentially take-down connections to the other players - so, this will be untested until an outage occurs. And then... Given the complexity of BPG, they will likely just end-up making things worse...
posted by rozcakj at 6:19 AM on July 12, 2022
The last thing that "grinds my gears" about this whole situation is... Who gave permission to execute this update on a Friday?
It is an extremely well-known IT "trope" that... YOU NEVER UPDATE PRODUCTION SYSTEMS ON A FRIDAY !
posted by rozcakj at 6:23 AM on July 12, 2022 [15 favorites]
It is an extremely well-known IT "trope" that... YOU NEVER UPDATE PRODUCTION SYSTEMS ON A FRIDAY !
posted by rozcakj at 6:23 AM on July 12, 2022 [15 favorites]
I believe it goes card terminal -> internet -> terminal provider (commonly merchant bank A) -> interbank network, e.g. VISA -> customer's bank B; I presume Canada works similarly.
Does it? That sounds correct.
So stores could take payments (I didn't come across any stores that didn't have working internet, but we were only hitting restaurants and gas stations) - but as you said, something like the interbank network was down. And Interac (this is what we call using a regular bank debit card) didn't work at all. But credit cards mostly worked (but not mine). And bank machines only worked if you went to your own branch? Or was that just for TD bank?
I'd really love to have this explained...
posted by kitcat at 7:10 AM on July 12, 2022
Does it? That sounds correct.
So stores could take payments (I didn't come across any stores that didn't have working internet, but we were only hitting restaurants and gas stations) - but as you said, something like the interbank network was down. And Interac (this is what we call using a regular bank debit card) didn't work at all. But credit cards mostly worked (but not mine). And bank machines only worked if you went to your own branch? Or was that just for TD bank?
I'd really love to have this explained...
posted by kitcat at 7:10 AM on July 12, 2022
I do wonder if we could have gotten through to 911 or the hospital? Happily we did not have to find out.
CBC the national last had a story last night about a woman who died on the street and no one around could call 911.
Then the report said removing a sim card from any phone would make it possible to call 911 even if cell service was down. (This is at about 5m50s on linked video)
Apparently thats the law but few know about it.
They say because then you have no network... me a non tech wonders why should you need to remove the sim card If you already have no network?
Anyway, still have my landine
Works with no power, no cell. Wouldnt help on the street, but i remember pay phones!
posted by chapps at 7:17 AM on July 12, 2022
CBC the national last had a story last night about a woman who died on the street and no one around could call 911.
Then the report said removing a sim card from any phone would make it possible to call 911 even if cell service was down. (This is at about 5m50s on linked video)
Apparently thats the law but few know about it.
They say because then you have no network... me a non tech wonders why should you need to remove the sim card If you already have no network?
Anyway, still have my landine
Works with no power, no cell. Wouldnt help on the street, but i remember pay phones!
posted by chapps at 7:17 AM on July 12, 2022
I support nationalizing. In part just because i figure only the state would make infrastructure in the north a priority.
Would be interested to know how folks fared up north / remote areas during the outage. Since they generally have lower quality internet, were they less affected with exisiting systems and fail safes?
posted by chapps at 7:22 AM on July 12, 2022 [4 favorites]
Would be interested to know how folks fared up north / remote areas during the outage. Since they generally have lower quality internet, were they less affected with exisiting systems and fail safes?
posted by chapps at 7:22 AM on July 12, 2022 [4 favorites]
This morning on the radio somebody proposed forcing the carriers to allow customers on to each other's networks in the event of an outage like this.
up until about a year ago, I spent much of my previous two decades living in a small remote community that suffered a lot of hydro outages. You could pretty much count on one whenever the wind blew past a certain velocity. Everybody ended up with their own level of backup/preparedness. From nothing at all to places like the medical clinic, the fire department and the co-op store (the main source of food etc) which all had automated generator systems (ie: they'd kick in immediately if the power went out for more than a few seconds).
I imagine (assuming rational heads prevail) the way forward with our telecom networks will, to some degree, mirror this. In other words, full and robust and pretty much immediate redundancy for essential services, with everything and everybody else figuring out their own version of necessity ... with cost likely being a factor in terms of how redundant you'll be. You want it all? You want your business or your gaming system (or whatever) to be as redundant as the fire department, expect to pay for it, maybe as an add-on to your monthly bill. And if you can't or won't afford it*, expect to be lower priority when it comes to getting reconnected to our vast and glorious interwebs.
* yes, the rich will fare better than the rest of us, same as it ever was. And more to the point, this would be consistent with how things play out in my former remote community. Beyond the essential services, the most likely people to have fully automated generator systems were the ones with the bigger, shinier homes. The rest of us -- we just made sure we had plenty of firewood and batteries and some good old school analog books to read, also a deck of playing cards.
posted by philip-random at 7:41 AM on July 12, 2022
up until about a year ago, I spent much of my previous two decades living in a small remote community that suffered a lot of hydro outages. You could pretty much count on one whenever the wind blew past a certain velocity. Everybody ended up with their own level of backup/preparedness. From nothing at all to places like the medical clinic, the fire department and the co-op store (the main source of food etc) which all had automated generator systems (ie: they'd kick in immediately if the power went out for more than a few seconds).
I imagine (assuming rational heads prevail) the way forward with our telecom networks will, to some degree, mirror this. In other words, full and robust and pretty much immediate redundancy for essential services, with everything and everybody else figuring out their own version of necessity ... with cost likely being a factor in terms of how redundant you'll be. You want it all? You want your business or your gaming system (or whatever) to be as redundant as the fire department, expect to pay for it, maybe as an add-on to your monthly bill. And if you can't or won't afford it*, expect to be lower priority when it comes to getting reconnected to our vast and glorious interwebs.
* yes, the rich will fare better than the rest of us, same as it ever was. And more to the point, this would be consistent with how things play out in my former remote community. Beyond the essential services, the most likely people to have fully automated generator systems were the ones with the bigger, shinier homes. The rest of us -- we just made sure we had plenty of firewood and batteries and some good old school analog books to read, also a deck of playing cards.
posted by philip-random at 7:41 AM on July 12, 2022
I'd really love to have this explained...
There's much more than one interbank network. Visa and Mastercard are the two major international ones for point-of-sale card transactions with debit and credit versions, but they charge fees for acting as the middleman and clearing house; more for credit than debit IIRC, and not all systems are available worldwide, especially debit. As an international traveller you used to have to make sure you had a card that would commonly work at your destination, though that's less of an issue these days. (Here, you often still see stickers on the window of the petrol station telling you which card networks they support - usually half a dozen or so)
I see Interac is the standard Canadian point-of-sale debit network, though I understand visa is more common for online and international orders - with cards often dual interac/visa, depending on how it's being used?
In the UK, while visa debit/credit is the primary transaction network for point-of-sale card terminals and online (contactless i.e. tap-to-pay rides on top of this), there's also BACS for smaller or routine account->account transfers (usually used for payroll and online banking f.ex) and CHAPS for high value immediate payments (I've only used it for a mortgage deposit; that was a painfully expensive trip to the bank!), and SWIFT/IBAN, for international bank transfers, as well as all the small players like American Express, paypal etc.
To swing back to the point, it sounds like interac was having major issues due to Rogers deciding it no longer wanted to be part of the internet for the day, which would have broken point-of-sale card terminals for interac users (i.e. most people) but would have allowed other protocols to continue, e.g. mastercard, if the merchant was using not-Rogers internet. Large bank branch ATMs would work for their own cards, but probably not 3rd party ones. Given interac have now said they're going to add an additional internet provider connection - other than Rogers - that rather implies they didn't have one before. Which is just piss poor planning if they didn't think of that until now for such a critical financial system, and some senior heads should roll for that one.
posted by Absolutely No You-Know-What at 8:11 AM on July 12, 2022 [3 favorites]
There's much more than one interbank network. Visa and Mastercard are the two major international ones for point-of-sale card transactions with debit and credit versions, but they charge fees for acting as the middleman and clearing house; more for credit than debit IIRC, and not all systems are available worldwide, especially debit. As an international traveller you used to have to make sure you had a card that would commonly work at your destination, though that's less of an issue these days. (Here, you often still see stickers on the window of the petrol station telling you which card networks they support - usually half a dozen or so)
I see Interac is the standard Canadian point-of-sale debit network, though I understand visa is more common for online and international orders - with cards often dual interac/visa, depending on how it's being used?
In the UK, while visa debit/credit is the primary transaction network for point-of-sale card terminals and online (contactless i.e. tap-to-pay rides on top of this), there's also BACS for smaller or routine account->account transfers (usually used for payroll and online banking f.ex) and CHAPS for high value immediate payments (I've only used it for a mortgage deposit; that was a painfully expensive trip to the bank!), and SWIFT/IBAN, for international bank transfers, as well as all the small players like American Express, paypal etc.
To swing back to the point, it sounds like interac was having major issues due to Rogers deciding it no longer wanted to be part of the internet for the day, which would have broken point-of-sale card terminals for interac users (i.e. most people) but would have allowed other protocols to continue, e.g. mastercard, if the merchant was using not-Rogers internet. Large bank branch ATMs would work for their own cards, but probably not 3rd party ones. Given interac have now said they're going to add an additional internet provider connection - other than Rogers - that rather implies they didn't have one before. Which is just piss poor planning if they didn't think of that until now for such a critical financial system, and some senior heads should roll for that one.
posted by Absolutely No You-Know-What at 8:11 AM on July 12, 2022 [3 favorites]
I wasn't in Canada, and the question struck me: was even regular, over the airways TV and/or radio even working? I kind of assume that all that stuff is run on the back-end through the internet.
posted by LizBoBiz at 3:05 AM on July 12
Regular TV over cable worked just fine, although some of the Rogers-owned stations were down for a bit in first thing in the morning.
posted by sardonyx at 8:11 AM on July 12, 2022
posted by LizBoBiz at 3:05 AM on July 12
Regular TV over cable worked just fine, although some of the Rogers-owned stations were down for a bit in first thing in the morning.
posted by sardonyx at 8:11 AM on July 12, 2022
I have Teksavvy, too, and it didn't go down at all for me. Maybe Teksavvy resells different providers in different places?
I think it rents whatever it can in terms of last mile providers. No idea what is the most common but I have the sense it's Rogers at least in Ontario? The Teksavvy outage map that day highlighted the entire province, at least. (I know their call center was Rogers too...)
posted by advil at 8:27 AM on July 12, 2022
I think it rents whatever it can in terms of last mile providers. No idea what is the most common but I have the sense it's Rogers at least in Ontario? The Teksavvy outage map that day highlighted the entire province, at least. (I know their call center was Rogers too...)
posted by advil at 8:27 AM on July 12, 2022
I think it rents whatever it can in terms of last mile providers. No idea what is the most common but I have the sense it's Rogers at least in Ontario?
In my experience with Teksavvy when there was a line issue it was always a Rogers technician I had to deal with (I'm in Ontario outside of the GTA). And, FWIW, the technicians were universally on the extreme defensive when they spoke to me as a customer which I gathered meant they were yelled at regularly. I can only imagine the amount of PTSD customer service must be dealing with now.
We used Rogers at home years ago and it was one debacle after another (how many times can Rogers blame poor service on squirrels, as if we had control over them? A lot) so we switched and generally haven't had an issue. With the outage, my service was back early Saturday morning which was nice but it meant I had to go into work to do what I couldn't do on Friday.
Hopefully this shakes up the industry because Canada's predilection towards monopolies & oligarchies really sucks. At the very least maybe shaking up the hard sells for bundles that Bell & Rogers always push.
posted by Ashwagandha at 9:42 AM on July 12, 2022 [1 favorite]
In my experience with Teksavvy when there was a line issue it was always a Rogers technician I had to deal with (I'm in Ontario outside of the GTA). And, FWIW, the technicians were universally on the extreme defensive when they spoke to me as a customer which I gathered meant they were yelled at regularly. I can only imagine the amount of PTSD customer service must be dealing with now.
We used Rogers at home years ago and it was one debacle after another (how many times can Rogers blame poor service on squirrels, as if we had control over them? A lot) so we switched and generally haven't had an issue. With the outage, my service was back early Saturday morning which was nice but it meant I had to go into work to do what I couldn't do on Friday.
Hopefully this shakes up the industry because Canada's predilection towards monopolies & oligarchies really sucks. At the very least maybe shaking up the hard sells for bundles that Bell & Rogers always push.
posted by Ashwagandha at 9:42 AM on July 12, 2022 [1 favorite]
I read Interac as Interlac this morning, and thought, great, I woke up in the 30th century this morning...
posted by wittgenstein at 9:47 AM on July 12, 2022
posted by wittgenstein at 9:47 AM on July 12, 2022
Why aren't there more nanny interlocks in configuration software between the admin's fingertips and a mistake of national impact like this?
posted by tigrrrlily at 10:12 AM on July 12, 2022
posted by tigrrrlily at 10:12 AM on July 12, 2022
Canada really only has one interbank network for the purposes of debit and ATMs - Interac (Wikipedia). All debit card transactions go through Interac, and all transactions at an ATM other than your own bank's are through Interac. Basically everyone has Interac-based debit cards by way of your bank card. And Interac only used Rogers. Interac is widely used, surpassed the number of cash payments two decades ago. and some people rely on it solely rather than credit cards as there's no need for them - anywhere that accepts card payments accepts debit via Interac. A reasonable number of places only accept debit since the transaction fee is less than half. There are very few cash only places. The universal dominance of Interac is also the reason why we had essentially universal chip and pin and now contactless sooner than some other places, making debit very convenient.
Then online transfers are also basically exclusively through Interac e-transfers and no one uses services such as Venmo (for Americans - we don't have it, because Interac allows sending money directly from your bank to another's by entering someone's email or cell number and it either auto-deposits if set up or the other person has to accept it, and for most major banks, without a fee).
So more Canadians than Americans may have stopped carrying cash for these reasons. It's almost never necessary to pay cash to a retailer or another person, and you can always use any ATM though for a fee. My partner tried to pick up medication Friday just before we left town, and it was impossible to pay as they don't have a credit card and the only close by ATM was a different bank. Fortunately I had some cash kicking around but otherwise we would have been in trouble. Interac not working was quite a disaster since it totally controls all means of payment other than cash and credit.
Interac is acartel former not for profit association and now corporation established by our small number of major banks. See Globe and Mail article explaining the change and reasoning.
Interac is heavily regulated by our competition bureau (a "pro-competitive collaboration") and it is supposed to run on a cost-recovery basis and provide its services on a specific basis.
The consent order between Interac, the major banks, and the competition bureau specifies various things including that banks must be given access to specific services at a specific cost, i.e. the ATM networks, and Visa and MasterCard must not be given access (it also specifies access may be restricted to financial institutions, which I believe would also include AMEX and any other credit cards).
So basically, put uncharitably, the explanation is that Canada will sacrifice competition in the name of not letting American companies win. See also airlines. More generously, in this case in the public interest - all banks can use it, and we do have more convenient, cheaper and universal payment systems than many other places and certainly than America (as I'm sure the relevant individuals would readily point out given the national pastime of favourably comparing ourselves to America).
Given the above, it is totally absurd that Interac used only Rogers and no backup.
posted by lookoutbelow at 10:24 AM on July 12, 2022 [12 favorites]
Then online transfers are also basically exclusively through Interac e-transfers and no one uses services such as Venmo (for Americans - we don't have it, because Interac allows sending money directly from your bank to another's by entering someone's email or cell number and it either auto-deposits if set up or the other person has to accept it, and for most major banks, without a fee).
So more Canadians than Americans may have stopped carrying cash for these reasons. It's almost never necessary to pay cash to a retailer or another person, and you can always use any ATM though for a fee. My partner tried to pick up medication Friday just before we left town, and it was impossible to pay as they don't have a credit card and the only close by ATM was a different bank. Fortunately I had some cash kicking around but otherwise we would have been in trouble. Interac not working was quite a disaster since it totally controls all means of payment other than cash and credit.
Interac is a
Interac is heavily regulated by our competition bureau (a "pro-competitive collaboration") and it is supposed to run on a cost-recovery basis and provide its services on a specific basis.
The consent order between Interac, the major banks, and the competition bureau specifies various things including that banks must be given access to specific services at a specific cost, i.e. the ATM networks, and Visa and MasterCard must not be given access (it also specifies access may be restricted to financial institutions, which I believe would also include AMEX and any other credit cards).
So basically, put uncharitably, the explanation is that Canada will sacrifice competition in the name of not letting American companies win. See also airlines. More generously, in this case in the public interest - all banks can use it, and we do have more convenient, cheaper and universal payment systems than many other places and certainly than America (as I'm sure the relevant individuals would readily point out given the national pastime of favourably comparing ourselves to America).
Given the above, it is totally absurd that Interac used only Rogers and no backup.
posted by lookoutbelow at 10:24 AM on July 12, 2022 [12 favorites]
Ngl, I pay for most of my everyday transactions with my debit card because it is much easier to use and cuts some slack for small businesses who get hella dinged with CC fees. Like, I am so used to tapping my card for anything that I am always startled when I go to the States and I have enter my PIN like some hooligan for every single use.
But of course this meant I was broke as a joke on Friday and I couldn't even get money out.
posted by Kitteh at 11:04 AM on July 12, 2022
But of course this meant I was broke as a joke on Friday and I couldn't even get money out.
posted by Kitteh at 11:04 AM on July 12, 2022
I think businesses are charged for debit card usage too, although not as much as credit cards.
posted by storybored at 11:37 AM on July 12, 2022
posted by storybored at 11:37 AM on July 12, 2022
If I'm reading right, Interac is 2-5 cents flat rate per transaction in person. Credit is flat rate plus percentage and has also been the target of competition investigations and class actions.
posted by lookoutbelow at 12:00 PM on July 12, 2022 [1 favorite]
posted by lookoutbelow at 12:00 PM on July 12, 2022 [1 favorite]
So...if we don't like American companies coming in to compete (I don't love the idea because they'll bring their culture with them - sorry American friends) - what about European ones?
posted by kitcat at 2:40 PM on July 12, 2022
posted by kitcat at 2:40 PM on July 12, 2022
Competition at the infrastructure level isn’t really a solution for something like telecom networks which are natural monopolies. Imagine having competition for your electricity - different companies running different sets of lines through town. Makes no sense - it’s a huge waste and makes everything more expensive. Solution is more something like what anthill describes:
> have the network operator a separate company who charge all service providers the same access fee. Presto, competitipn and lower prices.
I’d really love to see Canada do this.
posted by congen at 6:35 PM on July 12, 2022 [2 favorites]
> have the network operator a separate company who charge all service providers the same access fee. Presto, competitipn and lower prices.
I’d really love to see Canada do this.
posted by congen at 6:35 PM on July 12, 2022 [2 favorites]
>> have the network operator a separate company who charge all service providers the same access fee. Presto, competitipn and lower prices.
> I’d really love to see Canada do this.
The Interac of phones.
posted by lookoutbelow at 7:11 PM on July 12, 2022 [1 favorite]
> I’d really love to see Canada do this.
The Interac of phones.
posted by lookoutbelow at 7:11 PM on July 12, 2022 [1 favorite]
MetaFilter: Now I don't mind the pants bulge so much.
posted by ricochet biscuit at 5:57 AM on July 13, 2022 [1 favorite]
posted by ricochet biscuit at 5:57 AM on July 13, 2022 [1 favorite]
Looks like Rogers is going to fork over a paltry amount of compensation to directly-affected customers. Everyone else can go whistle, of course
posted by scruss at 3:17 PM on July 13, 2022 [1 favorite]
posted by scruss at 3:17 PM on July 13, 2022 [1 favorite]
« Older Nice threads | narrative coherence about cruelty and chaos Newer »
This thread has been archived and is closed to new comments
I couldn't even get out cash because while Tangerine is part of ScotiaBank, it just kept rejecting my debit card that day. Good thing my direct deposit for work happened before the outage!
posted by Kitteh at 1:13 PM on July 11, 2022 [5 favorites]