Holly Herndon's SPAWN AI: singing with someone else's voice in real time

November 15, 2021 4:43 PM

Herndon's remarkable software allows one person to sing with the voice of another. it's uncanny and spectacular. Herndon's videos from her @SonarFest presentation/collaboration. the collaborative possibilities! the ethical questions! Musical identities now way spongier with this new tool.
posted by spoh (40 comments total)

I loved her album Proto. Hard to believe that was just 2019. Pandemic time, man.
posted by TheKaijuCommuter at 5:08 PM on November 15, 2021 [2 favorites]

does the butlerian jihad have an official song
posted by glonous keming at 5:20 PM on November 15, 2021 [21 favorites]

does the butlerian jihad have an official song

Now canonically, it's played on the bagpipes.
posted by justsomebodythatyouusedtoknow at 5:23 PM on November 15, 2021 [5 favorites]

Proto is one of the best albums I’ve ever heard
posted by moorooka at 5:59 PM on November 15, 2021 [1 favorite]

wait til the R&B people get ahold of this - and they will
posted by pyramid termite at 6:07 PM on November 15, 2021 [2 favorites]

Echoing (with my own voice) the kudos for Proto.
posted by CheeseDigestsAll at 6:08 PM on November 15, 2021 [1 favorite]

Didn't this ship sail already with Vocaloids and auto-tuning? (And apologies if my question is naive. I'm not a musician.)
posted by Insert Clever Name Here at 6:19 PM on November 15, 2021 [2 favorites]

Clever Name:
This is different; I think this video from deeper in the thread demonstrates a bit more obviously what's going on. Basically, a Magic Neural Network takes your singing voice and outputs a new singing voice that sounds like someone else. (sure, possibly a someone with better ability to hit notes, as you'd get from autotune, but also drastically changing timbre.)
posted by kaibutsu at 6:30 PM on November 15, 2021 [4 favorites]

You can look like someone else, talk like someone else, sing like someone else… Someone else can look like you, talk like you, sing like you… Why?
posted by njohnson23 at 6:48 PM on November 15, 2021 [3 favorites]

Well the question has finally been conclusively answered. Someone let the Johns know.
posted by deadaluspark at 6:48 PM on November 15, 2021 [5 favorites]

takes your singing voice and outputs a new singing voice that sounds like someone else

I'm interested in the extremes of this—presumably such a system could not take my singing voice (of absolutely no ability, range, or pitch control) to make it sound like someone with much greater range and ability, or someone very recognisable? Or could I stumble up to the karaoke booth and press one of the buttons marked 'Tina Turner' or 'Björk' or 'Chris Cornell'?
posted by Fiasco da Gama at 7:10 PM on November 15, 2021 [1 favorite]

Or Gilbert Gottfried!
posted by Abehammerb Lincoln at 7:18 PM on November 15, 2021 [9 favorites]

I cannot be trusted with the Bee-Gee-o-matic. I suggest that nobody can, and we must destroy this technology.
posted by Fiasco da Gama at 7:22 PM on November 15, 2021 [7 favorites]

I would enjoy karaoke with this so much. I want to belt out power balads in the voice of Adele!
posted by simonw at 7:35 PM on November 15, 2021 [2 favorites]

Does Adele own the sound of her voice that would be depicted through these deep neural nets?
posted by rebent at 8:09 PM on November 15, 2021 [7 favorites]

Just came here to ask rebent’s question which came to me while envisioning my future as the lead singer of a Queen tribute band.
posted by vorpal bunny at 8:16 PM on November 15, 2021 [1 favorite]

Or maybe an Elton John tribute act. Where the piano is cleverly positioned to conceal the fact that I also lack that talent.
posted by vorpal bunny at 8:19 PM on November 15, 2021 [2 favorites]

Basically, I’d like to jump around on a stage and receive adulation without needing to have talent or put in the work. I’ll take your favorites in the meantime. My psychiatrist says I need them.
posted by vorpal bunny at 8:24 PM on November 15, 2021 [18 favorites]

So, lip synching without the synch? or is it the lips? Or either one, actually...

And maybe a court order could force the examination of the neural net in question to determine its influences Really tho, this is settled case law since the sampling wars. And I'm sure the majors will work out extremely equitable compensation for the artists they've stolen borrowed from — just like they did with streaming...

I never got any farther than 3 part improvised scatting with a guitar and a harmony box. I'm such an oughtie.
posted by tspae at 9:03 PM on November 15, 2021 [1 favorite]

She's also apparently put the model of her voice on the blockchain:

Artist X produces a song using the Holly+ voice model.

Artist X uploads the song online, and submits the song as a proposal to the DAO through a public interface.

VOICE token holders vote to mint the song as an appropriate or inspiring usage of the Holly+ voice.

An NFT of the song is minted by the DAO, with 50% of sales generated going to the artist, and 40% of sales going to DAO members, and 10% reserved for me.

posted by BungaDunga at 9:06 PM on November 15, 2021 [5 favorites]

She's also apparently put the model of her voice on the blockchain:

This whole thing seemed cool and interesting. Then I read this. Now I'm just thinking "no".
posted by Dysk at 9:46 PM on November 15, 2021 [20 favorites]

I loved her album Proto. Hard to believe that was just 2019. Pandemic time, man.

PROTO [bandcamp link] was apparently made using Spawn AI:

Holly’s third full-length album 'PROTO' isn’t about A.I., but much of it was created in collaboration with her own A.I. ‘baby’, Spawn.
[…]
You can hear traces of Spawn throughout the album, developed in partnership with long time collaborator Mathew Dryhurst and ensemble developer Jules LaPlace, and even eavesdrop on the live training ceremonies conducted in Berlin, in which hundreds of people were gathered to teach Spawn how to identify and reinterpret unfamiliar sounds in group call-and-response singing sessions; a contemporary update on the religious gathering Holly was raised amongst in her upbringing in East Tennessee.

posted by chappell, ambrose at 10:14 PM on November 15, 2021 [3 favorites]

This will end up with an All Cortex Choir.
posted by NoThisIsPatrick at 11:34 PM on November 15, 2021 [7 favorites]

"What if ... a record company, but the shareholders pick the tracks to release!"

"What if ... an effect box, but licensed like Unreal Engine? No up front fee, just 5% after the first $1,000,000 50% of your earnings in perpituity, plus an up front fee!"
posted by you at 12:26 AM on November 16, 2021 [6 favorites]

nvidia announced availability of software additions to their Riva product last week that allows you to train a custom voice on as little as 30mins of input material. It takes a day or so on their A line, but I calculated it for consumer GPU and it was about 2. It is on my list of things to mess with.
posted by joelr at 2:02 AM on November 16, 2021 [1 favorite]

Herndon records the audiences at her concerts for use in training SPAWN - one of the singers does a phrase, the audience sings it back, and another and another... The most enjoyable audience participation I can remember, singing quite unselfconsciously, because we were doing it for Science.

Proto is an astonishing record, and worked beautifully live - an electronica choir.
posted by Grangousier at 3:07 AM on November 16, 2021 [2 favorites]

Hmm, on one hand, this could help vocalists preserve their voices during concert tours, and might have some self-use for voice-based disabilities. On the other hand, this undoes the plot line for The Little Mermaid…
posted by childofTethys at 3:33 AM on November 16, 2021 [1 favorite]

Dang, they could have given Ebert Siskel's voice in the last years!
posted by wenestvedt at 4:17 AM on November 16, 2021 [2 favorites]

Still missing this. To me it just sounds like a group of singers passing microphones back and forth w/ some post-processing applied to make their voices similar. Is there another clip somewhere that demonstrates the technique in a less, urm, theatrical setting? Or maybe using more disparate voices, e.g. a soprano singing as a bass?
posted by Insert Clever Name Here at 7:11 AM on November 16, 2021 [9 favorites]

fusion of Boston Dynamics and SPAWN AI

have you ever been hunted down by singing robot dogs? because this is how we get singing robot dogs
posted by elkevelvet at 7:25 AM on November 16, 2021 [5 favorites]

P.S. I like the singing...I just am not getting the 'one person singing with the voice of another.' The closest I come to 'getting this' is that there's one synthetic voice (generated by the AI) which the singers are 'using' in turn, so they all sound alike.
posted by Insert Clever Name Here at 7:33 AM on November 16, 2021

Insert Clever, you have to work your way down until you find the clip of the MAN singing in her voice in real time.
posted by scolbath at 7:39 AM on November 16, 2021 [2 favorites]

Some men can sing very clearly in a high register, so it’s hard to tell how much the software is doing, and that tweet mentions the song was chosen because it fits his range.

A simple on/off example without any other filters or backing music would do wonders to illustrate what’s happening here. She sings a few bars without the software. Other person sings a few bars without the software, then with the software.

Is this a “kinda sorta” level of mimicry, or is it uncanny? To what extent does it depend on the singer getting at least vaguely close to the imitated singer’s timbre? How well does it handle phonemes not in the original training data (if any?)
posted by jedicus at 8:01 AM on November 16, 2021 [2 favorites]

(from the artist's tweeted commentary): only works signed by the DAO will be considered official works. No confusion. An easy way to address concerns about deepfakes or whatever.

this is a hold my beer moment waiting to happen? there's a way these new processes and tools will allow artists to explore and collaborate, so that is a positive for sure.. someone posted some cool stuff on Imogen Heap a while ago (adjacent to this item, in that we're seeing artists expand and transform their works using technology) and I'm just going to focus on the positives for now.
posted by elkevelvet at 11:49 AM on November 16, 2021

I'm fired up for Ave Maria in Gordon Gano voice.
posted by Abehammerb Lincoln at 12:10 PM on November 16, 2021 [1 favorite]

O mio babbino caro
an Aria performed by David Lee Roth
posted by clavdivs at 1:15 PM on November 16, 2021 [1 favorite]

Sounds like overcompressed kids voices sped up by an engineer who has lost the 3k spectral range on his ears from too much Mesa Boogie at close range as a youth. But then again, that could describe me.

Talk about lofi.
posted by Droll Lord at 1:42 PM on November 16, 2021 [1 favorite]

Holly Herndon and her husband Mat Dryhurst seem like cool people, until you notice that they give off a vibe not unlike Scientologists, only for the blockchain/cryptocurrencies.
posted by acb at 4:08 PM on November 16, 2021 [1 favorite]

I've always wanted a new voice so I can belt.
posted by jenfullmoon at 10:11 PM on November 16, 2021

So even shitty singers like me have a new lease on karaoke.
posted by zardoz at 12:05 AM on November 17, 2021 [1 favorite]