Make Anim(ation) Real

May 17, 2024 4:36 PM

Over 15 years ago, Microsoft released Photosynth [previously], a nifty tool that could correlate dozens of photos of the same place from different angles in order to make a sort of virtual tour using photogrammetry, a technique that went on to influence Google Earth's 3D landscapes and virtual reality environments. But what if you tried the same thing with cartoons? Enter Toon3D, a novel approach to applying photogrammetry principles to hand-drawn animation. The results are imperfect due to the inherent inconsistency of drawn environments, but it's still rather impressive to see a virtual camera moving around glitched-out versions of the Krusty Krab, Bojack Horseman's living room, or the train car from Spirited Away. Interestingly, the same approach works about as well on paintings or even AI-generated video; see also the similar technique of neural radiance fields (NERFs) for creating realistic high-fidelity virtual recreations of real (and unreal) environments.
posted by Rhaomi (17 comments total)

I love how Shaggy is an integral part of the mystery machine.
posted by If only I had a penguin... at 4:54 PM on May 17 [2 favorites]

when I saw those NERF vids a few years ago, my mind simply boggled. Now?

Some ML processes now have developed correlative models 1, 2 (of MANY) that appear to 'understand" physics; or ducktype outputs that are indistinguishable from current state-of-the-art sims,..... when that happens at scale....oy.
posted by lalochezia at 5:11 PM on May 17 [1 favorite]

HULK BRAIN HURT SMASH!
posted by y2karl at 5:51 PM on May 17

The photosynth demo was amazing, but I never found a tool that would simply let me... use that? Do that? Much less dump in a bunch of pictures of a scene and then build me a model of it.

Does such a thing exist?
posted by mhoye at 6:15 PM on May 17 [4 favorites]

Doubling the ask here - are there any simple tools nowadays that let you feed in a bunch of photos at different angles, select common points, and et-voila?

(While I'm at it, does anyone know any tools to use to take something that's been photographed at an angle and de-angle it so you get a "straight-on" view?)
posted by KChasm at 6:45 PM on May 17 [3 favorites]

Photoshop has a perspective warp tool. it is for this. I don't think it works for a super complex image though. I mean if you think of a 3d object part of the problem is that the PARTS of the thing that are visible are different from different angles so you can't really expect photoshop to guess what's on the otherside of that thing, for example.
posted by If only I had a penguin... at 7:18 PM on May 17

Doubling the ask here - are there any simple tools nowadays that let you feed in a bunch of photos at different angles

COLMAP isn't exactly voila but does get you a fair bit of the way along.

NERFs are cool, but slow... last time i checked, like 6 months ago. basically you're training a neural network to interpolate between images, so every time you want to want to generate a new view, you need to run a forward pass on the network. it works, but it's not exactly realtime.

3d gaussian splatting on the other hand does get you to realtime photorealistic rendering of novel viewpoints. Splatting can take the output of COLMAP (a json file describing camera poses and 'interesting' points extracted from the image files you provide), and fit 3-dimensional gaussians (centered on the interesting points from COLMAP) with RGBA values.

there's a demo you can download from the inria site... it is possible to diy a pipeline from raw images to fully navigable gaussian splat environment, but it's fickle as fuck in my experience.
posted by logicpunk at 7:23 PM on May 17

It's funny, I used the Photosynth app on my iPhone, but the main functionality I remember was being able to take a bunch of shots all around you and have them stitched into a 360-degree immersive panoramic image. Like, you could scroll around the resulting image and see above you, behind you, on either side. It was awesome. I used it a lot to capture impressive landscapes when I travelled. But then Microsoft stopped updating the iPhone app, and it got to the point where the phone wouldn't even let me open the app anymore. So now all my cool panoramas are locked into a dead app, inaccessible. After all these years, I still haven't deleted Photosynth from my phone, hoping against hope that someday there might be a way to retrieve the panos. It's in a sad little "needs to be updated" folder right next to Flappy Bird.
posted by flod at 7:23 PM on May 17 [3 favorites]

I love how Shaggy is an integral part of the mystery machine

Scooby Dooby Doo
Where are you
David Cronenburg's merged you with the scenery
posted by credulous at 7:42 PM on May 17 [4 favorites]

re de-angling a photo -

If you happen to be on a Mac, I'm pretty sure Graphic Converter has some good tools for doing that.
posted by kristi at 8:09 PM on May 17

Crashed Firefox on my phone, twice.
posted by Ursula Hitler at 8:30 PM on May 17 [1 favorite]

For Gaussian Splatting of real-world scenes, I’m fond of using the free iOS app Scaniverse! It’s not perfect but it’s free and pretty neat
posted by DoctorFedora at 10:06 PM on May 17 [1 favorite]

Wait, I thought we were hating on CGI this month?
posted by fairmettle at 11:32 PM on May 17

Photosynth… hoping against hope that someday there might be a way to retrieve the panos

I don’t know how helpful this is if the main problem is extracting the data from the app, but there appears to be source on GitHub and a web-based viewer on Internet Archive.
posted by rh at 5:56 AM on May 18 [1 favorite]

(Rhaomi, thank you for your series of excellent and informative AI posts. We're in the middle of a hugely creative period in tech development in this area and it's great to be able to follow along on MeFi thanks to your research.)
posted by Nelson at 7:11 AM on May 18 [2 favorites]

> NERFs are cool, but slow...

logicpunk, you need to check out NVIDIA's Instant Neural Graphics Primitives: https://github.com/NVlabs/instant-ngp. They sped up NERF training by a factor of 10,000. They train in a couple seconds now. The most time consuming part is the sparse reconstruction that COLMAP does.
posted by riotnrrd at 9:42 AM on May 18 [1 favorite]

I wonder what you'd end up with by running this on all the 2D storyboards for a 3D scene that hasn't yet gone through layout/animation/etc. Would it give you a sense of the implied space the storyboards are describing, or would it just be a mess?
posted by nobody at 6:16 PM on May 18