AI-powered voice recognition software helps people with disabilities
March 1, 2025 12:55 PM Subscribe
How AI and new technologies revolutionise Eleanor Beidatsch's ability to work and study. Doctors thought Eleanor Beidatsch would die by the time she was two. Now AI is transforming her life.
AI-powered voice recognition software is helping her career as a journalist and her studies, but it's not without risks.
If a simple version of agentic AI could be applied to taking actions on screen, that would be a great boon. As someone who used Dragon Naturally Speaking 15 years ago because I couldn't use my hands for keyboard and mouse and my foot-controlled trackball wasn't accurate enough for fine movement, let me break down how you do basic actions using voice only:
Say you want to click on a link embedded in some text, or maybe click a bookmark on your browser. You say "mouse grid" into your microphone and it divides your screen into a grid of 9 squares. You say the number of a square, and that square gets divided into 9. Keep going until your cursor lands close enough to be able to contextually click that link. Or say right click to open the context menu, then do the mouse click squares 5 times again until it highlights Open Link in New Tab. There are a few shortcuts, you can speak key combinations out loud, you can create macros, but there aren't desktop programs that hook into voice commands like you get on the phone.
If agentic AI - which as far as I understand it, essentially takes screenshots and uses text recognition to control your computer and execute simple tasks - can work reasonably quickly, accurately, securely, and somewhat inexpensively, that would be huge leap in productivity and just plain ability to function in society for many disabled people. No longer would copying a file from one folder to another take a minute of your time if you could say "copy FILE 01 from FOLDER 01 to FOLDER 02" instead of "mousegrid 5, 7, doubleclick 9 [open folder] mousegrid 3 mark, 3 mark, 1 mark [select file] mousegrid 5 click, 7 click, 8 click [drag selected file to screen location]" and have it executed for you in seconds, and that's like the lowest hanging fruit. Imagine just being able to say "Computer, go to [online pharmacy], log in to my account, add prescription, attach PRESCRIPTION01.PDF from my downloads folder, submit" and watching as over the next 15 seconds your cursor jumps from location to location with a slight latency of a second or two as the screenshots are sent to the server and parsed. LLMs will have been worth everything if they can just get that functionality set up for the disabled, let alone for everyone else wanting to cosplay as Picard. I had actually assumed that the article would be talking about this kind of feature being tested, because voice recognition isn't exactly new, and the example of telling Google to turn on the TV is nowhere near the description of AI-powered software helping the disabled.
posted by phibetakafka at 9:35 PM on March 1 [7 favorites]
Say you want to click on a link embedded in some text, or maybe click a bookmark on your browser. You say "mouse grid" into your microphone and it divides your screen into a grid of 9 squares. You say the number of a square, and that square gets divided into 9. Keep going until your cursor lands close enough to be able to contextually click that link. Or say right click to open the context menu, then do the mouse click squares 5 times again until it highlights Open Link in New Tab. There are a few shortcuts, you can speak key combinations out loud, you can create macros, but there aren't desktop programs that hook into voice commands like you get on the phone.
If agentic AI - which as far as I understand it, essentially takes screenshots and uses text recognition to control your computer and execute simple tasks - can work reasonably quickly, accurately, securely, and somewhat inexpensively, that would be huge leap in productivity and just plain ability to function in society for many disabled people. No longer would copying a file from one folder to another take a minute of your time if you could say "copy FILE 01 from FOLDER 01 to FOLDER 02" instead of "mousegrid 5, 7, doubleclick 9 [open folder] mousegrid 3 mark, 3 mark, 1 mark [select file] mousegrid 5 click, 7 click, 8 click [drag selected file to screen location]" and have it executed for you in seconds, and that's like the lowest hanging fruit. Imagine just being able to say "Computer, go to [online pharmacy], log in to my account, add prescription, attach PRESCRIPTION01.PDF from my downloads folder, submit" and watching as over the next 15 seconds your cursor jumps from location to location with a slight latency of a second or two as the screenshots are sent to the server and parsed. LLMs will have been worth everything if they can just get that functionality set up for the disabled, let alone for everyone else wanting to cosplay as Picard. I had actually assumed that the article would be talking about this kind of feature being tested, because voice recognition isn't exactly new, and the example of telling Google to turn on the TV is nowhere near the description of AI-powered software helping the disabled.
posted by phibetakafka at 9:35 PM on March 1 [7 favorites]
« Older The March of Links | Personality Crisis Newer »
But voice recognition/control in general has been a game changer for disabled folks. Especially in combination with switches and the like. Though that depends on the reliability of those connections--I used voice control for my lightbulbs because of mobility issues but AppleHome has randomly decided that it can't find them, even though the app that came with the lightbulbs works with them fine (but doesn't have voice control like Home does). So it's a tenuous support. Not that this isn't the case with other kinds of assistive tech--I certainly had random battery issues or the like with my power chair, so there's always that risk.
posted by brook horse at 6:52 PM on March 1