Zendesk fumbles software vulnerability
October 14, 2024 5:05 AM Subscribe
Tell me if you've heard this one before: a curious young bug hunter discovers a major software vulnerability, tries to report it, and is ignored and gaslit. Today's villain is Zendesk, which you've probably used if you've interacted with customer support tickets.
Well this is a lovely example for my human-factors-infosec class.
(A lovely example of "how not to respond to a vuln report.")
posted by humbug at 6:56 AM on October 14
(A lovely example of "how not to respond to a vuln report.")
posted by humbug at 6:56 AM on October 14
"Gosh, good thing this is OUT OF SCOPE, huh?"
posted by rmd1023 at 6:59 AM on October 14 [1 favorite]
posted by rmd1023 at 6:59 AM on October 14 [1 favorite]
The real villian here is outsourcing. Companies outsource customer support ticket management to Zendesk. Zendesk outsources bug report responses to HackerOne.
Outsourcing is the scourge of our times. It's supposed to be a way of obtaining expertise on the cheap. But it's really a way of avoiding responsibility. Prisons outsource guard services. Insurance companies outsource home inspections. Banks outsource mortgage application processing. Towns outsource car towing. And then when it all blows up they can point the finger at someone else who if they're smart can point the finger at yet someone else.
You can come up with thousands of other examples, and they all suck.
posted by mono blanco at 7:10 AM on October 14 [41 favorites]
Outsourcing is the scourge of our times. It's supposed to be a way of obtaining expertise on the cheap. But it's really a way of avoiding responsibility. Prisons outsource guard services. Insurance companies outsource home inspections. Banks outsource mortgage application processing. Towns outsource car towing. And then when it all blows up they can point the finger at someone else who if they're smart can point the finger at yet someone else.
You can come up with thousands of other examples, and they all suck.
posted by mono blanco at 7:10 AM on October 14 [41 favorites]
But it's really a way of avoiding responsibility.
Liability, not responsibility. I can say "I take responsibility for this harm you experienced", (but subtext: I owe you nothing in reparation).
They are avoiding LIABILITY.
posted by lalochezia at 7:22 AM on October 14 [8 favorites]
Liability, not responsibility. I can say "I take responsibility for this harm you experienced", (but subtext: I owe you nothing in reparation).
They are avoiding LIABILITY.
posted by lalochezia at 7:22 AM on October 14 [8 favorites]
I'd say it's pretty clear they're avoiding both.
posted by biogeo at 7:27 AM on October 14 [6 favorites]
posted by biogeo at 7:27 AM on October 14 [6 favorites]
Personally, I’ve always found it surprising that these massive companies, worth billions, rely on third-party tools like Zendesk instead of building their own in-house ticketing systems.
He is in no way wrong here but this is how you get a "ticketing system" made out of excel macros. That is Zendesk's competition and explains the sorry state of the thing.
posted by Vulgar Euphemism at 7:27 AM on October 14 [15 favorites]
He is in no way wrong here but this is how you get a "ticketing system" made out of excel macros. That is Zendesk's competition and explains the sorry state of the thing.
posted by Vulgar Euphemism at 7:27 AM on October 14 [15 favorites]
Personally, I’ve always found it surprising that these massive companies, worth billions, rely on third-party tools like Zendesk instead of building their own in-house ticketing systems.
Do you also expect BigCorps to write their own word processor and spreadsheet implementations?
Defect ticketing might be more complicated and harder to get right than you think. Speaking as a former user of a couple of BigCorps' homemade ticketing systems, from before ZenDesk and its like were a thing.
posted by Aardvark Cheeselog at 7:33 AM on October 14 [9 favorites]
Do you also expect BigCorps to write their own word processor and spreadsheet implementations?
Defect ticketing might be more complicated and harder to get right than you think. Speaking as a former user of a couple of BigCorps' homemade ticketing systems, from before ZenDesk and its like were a thing.
posted by Aardvark Cheeselog at 7:33 AM on October 14 [9 favorites]
Look I’m no coding genius or anything, but it seems to me one could just replace the incremental ticket ID with a randomly-generated ID and that would de-fang this attack with minimal effort.
But, you know, I don’t work for ZenDesk, I work in research, and I just read computer security articles for fun
posted by caution live frogs at 7:51 AM on October 14 [3 favorites]
But, you know, I don’t work for ZenDesk, I work in research, and I just read computer security articles for fun
posted by caution live frogs at 7:51 AM on October 14 [3 favorites]
15!? Wow.
posted by Wretch729 at 7:59 AM on October 14 [2 favorites]
posted by Wretch729 at 7:59 AM on October 14 [2 favorites]
I work on an in-house clinical trial data collection system. We have about a dozen developers, who mostly work supporting the dozens of protocols we handle.
We also support a system that was developed by another institution, and supported by a consortium. I have no idea how big their development team is, but I do know that they've incorporated some major features developed by other consortium members in their codebase. We have 3 dedicated support people for hundreds of protocols and other projects that our people use it for. My team uses it too, for stuff that we can't directly support.
Likewise, we as an institution switched from our home grown EMR to Epic, when the law changed and upgrading our internal system to be compliant would be cost prohibitive. Epic could do almost everything we needed, and the stuff they couldn't do, they developed on their dime, because they skimmed over our requirements, assuming that we were like every other hospital.
In many cases it's better to buy than to build.
posted by Spike Glee at 8:00 AM on October 14 [6 favorites]
We also support a system that was developed by another institution, and supported by a consortium. I have no idea how big their development team is, but I do know that they've incorporated some major features developed by other consortium members in their codebase. We have 3 dedicated support people for hundreds of protocols and other projects that our people use it for. My team uses it too, for stuff that we can't directly support.
Likewise, we as an institution switched from our home grown EMR to Epic, when the law changed and upgrading our internal system to be compliant would be cost prohibitive. Epic could do almost everything we needed, and the stuff they couldn't do, they developed on their dime, because they skimmed over our requirements, assuming that we were like every other hospital.
In many cases it's better to buy than to build.
posted by Spike Glee at 8:00 AM on October 14 [6 favorites]
Good job Daniel, and nicely written up.
I work with both Zendesk and a custom-built solution for helpdesks and they are both hellish in special ways. Helpdesk systems share a lot of common features while needing to be extremely customised to specific business processes. They also get tightly tracked because all helpdesks use multiple metrics to track performance tied directly to pay - a technical hiccup can cost the agents or their company a LOT of money, so you have angry public users, angry internal users and angry stakeholders.
This particular bug doesn't affect me, but now I gotta go forward this to certain people. Sigh.
posted by dorothyisunderwood at 8:11 AM on October 14 [3 favorites]
I work with both Zendesk and a custom-built solution for helpdesks and they are both hellish in special ways. Helpdesk systems share a lot of common features while needing to be extremely customised to specific business processes. They also get tightly tracked because all helpdesks use multiple metrics to track performance tied directly to pay - a technical hiccup can cost the agents or their company a LOT of money, so you have angry public users, angry internal users and angry stakeholders.
This particular bug doesn't affect me, but now I gotta go forward this to certain people. Sigh.
posted by dorothyisunderwood at 8:11 AM on October 14 [3 favorites]
The security issue is really with the Zendesk customers' handling of SSO and 2FA wrt email, as the hacker acknowledges. That's why various companies paid bounties (and also why Zendesk exempts this kind of thing from their bug bounty program). They're also doing an end run around HackerOne's explicit policies against thing kind of thing by contacting the customers directly.
The hacker is very aware of all of this and likely works these edges all the time: "I had broken HackerOne's disclosure guidelines by sharing the vulnerability with affected companies. I didn’t bother to argue :)" lmao yeah, because what you did was inarguably unethical my dude, what were you planning to argue? And it just doesn't add a lot of value: if it did, Zendesk would not explicitly disallow this kind of thing from their bug bounty program. They WANT you to find security holes, that's why the bug bounty program exists! This just isn't really one, or rather Zendesk is downstream of it.
It's like gathering up a bunch of cans from the street, trying to sue Coke for littering ("They admitted that they made the cans!"), and then smugly pointing out that the local recycling center paid out on the aluminum. Then you write a blog post: "Why doesn't Coke have Litter Prevention Teams for all of its cans that wind up in the street!!! They wouldn't even pay me for suggesting it!" It's not *useless* activity per se, but if you can do all that then there's something more useful than you can do with your time.
"Zendesk had a security hole and they didn't take me seriously" is probably a better way to build clout with less technical readers then "A bunch of mom and pop businesses don't secure email very well"
posted by Kwine at 8:32 AM on October 14 [2 favorites]
The hacker is very aware of all of this and likely works these edges all the time: "I had broken HackerOne's disclosure guidelines by sharing the vulnerability with affected companies. I didn’t bother to argue :)" lmao yeah, because what you did was inarguably unethical my dude, what were you planning to argue? And it just doesn't add a lot of value: if it did, Zendesk would not explicitly disallow this kind of thing from their bug bounty program. They WANT you to find security holes, that's why the bug bounty program exists! This just isn't really one, or rather Zendesk is downstream of it.
It's like gathering up a bunch of cans from the street, trying to sue Coke for littering ("They admitted that they made the cans!"), and then smugly pointing out that the local recycling center paid out on the aluminum. Then you write a blog post: "Why doesn't Coke have Litter Prevention Teams for all of its cans that wind up in the street!!! They wouldn't even pay me for suggesting it!" It's not *useless* activity per se, but if you can do all that then there's something more useful than you can do with your time.
"Zendesk had a security hole and they didn't take me seriously" is probably a better way to build clout with less technical readers then "A bunch of mom and pop businesses don't secure email very well"
posted by Kwine at 8:32 AM on October 14 [2 favorites]
It's like gathering up a bunch of cans from the street, trying to sue Coke for littering
Good analogy, but I actually think we should do this. Companies shouldn't get to dodge responsibility for the garbage they create by passing the buck to smaller companies or individuals for whom it is more difficult and expensive to manage that garbage. The best way to ensure that there are resources for managing garbage, either literal or metaphorical, is to make sure it's priced into the original cost of the good or service. And the best way to do that is to make the manufacturer or provider pay for it. If ZenDesk could improve security with a few simple changes (like producing random instead of incremental tokens, I mean Jesus Christ it's 2024 not 1994) they should be responsible for doing so, instead of shrugging and saying it's on their customers to secure their system properly.
posted by biogeo at 8:56 AM on October 14 [9 favorites]
simple changes like producing random instead of incremental tokens
Hoping that 'nobody will be able to guess the next call number' is security by obscurity.
Also with a helpdesk system the end users will often need to quote the log number and that is much easier to do with a simple 5 or 6 digit number than a 32 or 64 digit random code.
posted by Lanark at 9:20 AM on October 14
Hoping that 'nobody will be able to guess the next call number' is security by obscurity.
Also with a helpdesk system the end users will often need to quote the log number and that is much easier to do with a simple 5 or 6 digit number than a 32 or 64 digit random code.
posted by Lanark at 9:20 AM on October 14
lmao yeah, because what you did was inarguably unethical my dude
Was it? He tried to take it directly to Zendesk first. H1, which handled Zendesk's bug bounty program, said it was out of scope. He escalated to try to get an answer from Zendesk itself. He got the same answer.
What would have been the more ethical thing to do: just leave the vulnerability unreported and unfixed, or inform the actual users who might be affected by it?
By the sounds of it he didn't make this public until it was actually fixed; he communicated privately with the affected companies, who then turned to Zendesk, which took the issue seriously this time because paying customers were reporting it.
lmao yeah, because what you did was inarguably unethical my dude, what were you planning to argue?
Sounds like he could have argued that he had reported the issue to Zendesk first, and then reported it to Zendesk again, only taking it elsewhere when Zendesk rejected it. (Of course Zendesk could then reply that it was refusing to pay the bounty because it was technically outside of the defined scope for rewards, which is at least a better reason than "you brought it privately to the right people's attention after we told you we would not fix it.")
posted by trig at 10:17 AM on October 14 [9 favorites]
Was it? He tried to take it directly to Zendesk first. H1, which handled Zendesk's bug bounty program, said it was out of scope. He escalated to try to get an answer from Zendesk itself. He got the same answer.
What would have been the more ethical thing to do: just leave the vulnerability unreported and unfixed, or inform the actual users who might be affected by it?
By the sounds of it he didn't make this public until it was actually fixed; he communicated privately with the affected companies, who then turned to Zendesk, which took the issue seriously this time because paying customers were reporting it.
lmao yeah, because what you did was inarguably unethical my dude, what were you planning to argue?
Sounds like he could have argued that he had reported the issue to Zendesk first, and then reported it to Zendesk again, only taking it elsewhere when Zendesk rejected it. (Of course Zendesk could then reply that it was refusing to pay the bounty because it was technically outside of the defined scope for rewards, which is at least a better reason than "you brought it privately to the right people's attention after we told you we would not fix it.")
posted by trig at 10:17 AM on October 14 [9 favorites]
15 years old - plenty sharp enough to figure this out, but not enough graywrinkles yet to have experienced "in-house bug tracking system" 🙀
posted by Rat Spatula at 11:03 AM on October 14 [2 favorites]
posted by Rat Spatula at 11:03 AM on October 14 [2 favorites]
Actually, this explanation is SO well organized and written that I am starting to wonder whether a 15-year-old actually produced it... (On the other hand, if this kid can code that well, think that creatively and communicate that well, he is a real catch for a future employer!)
posted by sonofsnark at 11:14 AM on October 14
posted by sonofsnark at 11:14 AM on October 14
The security issue is really with the Zendesk customers' handling of SSO and 2FA wrt email, as the hacker acknowledges.
Yeah, giving permissions based on "can see email to any address on our domain" is bad and there are probably other ways to exploit this.
I think there is a legitimate Zendesk vulnerability here too, though. As they point out, if you know the help requester's email and can guess the sequential ID, you can see their tickets which could contain sensitive data. You could phish a certain amount of users of any big enough service into filing tickets with their passwords, driver's license images, or other information by sending a forged email saying they need to go to and open a ticket to verify their identity.
The help desk would catch on eventually but you could have already downloaded some sensitive info by the time they locked things down.
There's probably a general class of scam where you induce people to request support, or find people likely to do so, and eavesdrop on them authenticating themselves. Imagine hiding an audio recorder near an ATM in a strip club or casino and listening in on people calling their banks trying to raise their withdrawal limits.
posted by smelendez at 11:20 AM on October 14
Yeah, giving permissions based on "can see email to any address on our domain" is bad and there are probably other ways to exploit this.
I think there is a legitimate Zendesk vulnerability here too, though. As they point out, if you know the help requester's email and can guess the sequential ID, you can see their tickets which could contain sensitive data. You could phish a certain amount of users of any big enough service into filing tickets with their passwords, driver's license images, or other information by sending a forged email saying they need to go to
The help desk would catch on eventually but you could have already downloaded some sensitive info by the time they locked things down.
There's probably a general class of scam where you induce people to request support, or find people likely to do so, and eavesdrop on them authenticating themselves. Imagine hiding an audio recorder near an ATM in a strip club or casino and listening in on people calling their banks trying to raise their withdrawal limits.
posted by smelendez at 11:20 AM on October 14
Actually, this explanation is SO well organized and written that I am starting to wonder whether a 15-year-old actually produced it...
I think a bright 15-year-old could write that, but parts of it also sounded like ChatGPT's writing style, so it wouldn't surprise me if they used that to assist in writing it up.
posted by biogeo at 2:31 PM on October 14
Hoping that 'nobody will be able to guess the next call number' is security by obscurity.
Obscurity isn't sufficient for security, I agree, but certainly inasmuch as the perfect is the enemy of the good, making it harder to guess valid ticket numbers is superior to going "well we'll just rely on making it so that the easily guessable id sequence is useless to a potential attacker" and it's comparatively cheap to implement.
posted by axiom at 2:35 PM on October 14 [2 favorites]
Obscurity isn't sufficient for security, I agree, but certainly inasmuch as the perfect is the enemy of the good, making it harder to guess valid ticket numbers is superior to going "well we'll just rely on making it so that the easily guessable id sequence is useless to a potential attacker" and it's comparatively cheap to implement.
posted by axiom at 2:35 PM on October 14 [2 favorites]
parts of it also sounded like ChatGPT's writing style
Someone in the comments thought so too.
posted by We had a deal, Kyle at 3:27 PM on October 14
Someone in the comments thought so too.
posted by We had a deal, Kyle at 3:27 PM on October 14
lmao yeah, because what you did was inarguably unethical my dude
Breaking TOS isn’t unethical.
posted by Uncle at 6:14 PM on October 14
Breaking TOS isn’t unethical.
posted by Uncle at 6:14 PM on October 14
I mean, publicizing a vulnerability before you've informed the code owners and given them every chance to fix it is generally unethical, largely because it increases the chances that the vulnerability will be exploited before it's fixed because now it's public knowledge.
But that's not what happened in this case.
posted by trig at 6:40 PM on October 14
But that's not what happened in this case.
posted by trig at 6:40 PM on October 14
« Older VFX Artists Expose AI Scams | 'Shine your light on the world' Newer »
posted by Pemdas at 6:30 AM on October 14 [4 favorites]