shaky cam + time lapse + inferred geometry + smoothed path =
August 11, 2014 8:46 AM Subscribe
very useful. Would like to see the algorithm ran on some of those crazy down-hill-in-the-city bike videos. (Or really almost all go-pro footage, since they're a wobbly train wreck of vertigo to watch.. )
(and some snark for the biker in the video riding on the sidewalk, jerk. )
posted by k5.user at 9:02 AM on August 11, 2014 [2 favorites]
(and some snark for the biker in the video riding on the sidewalk, jerk. )
posted by k5.user at 9:02 AM on August 11, 2014 [2 favorites]
So very cool.
I used to be addicted to watching bike videos, back when I had free time at work---it was the next best thing to riding when shimminig instruments. We've come such a long way from the shaky, lo-res lipstick cams of the 90s. It's amazing to see stuff like this.
posted by bonehead at 9:07 AM on August 11, 2014
I used to be addicted to watching bike videos, back when I had free time at work---it was the next best thing to riding when shimminig instruments. We've come such a long way from the shaky, lo-res lipstick cams of the 90s. It's amazing to see stuff like this.
posted by bonehead at 9:07 AM on August 11, 2014
The continual adjustments in the terrain remind me of walking through a video game world, watching objects pop into view and increase resolution as they get nearer.
posted by Phatty Lumpkin at 9:07 AM on August 11, 2014 [5 favorites]
posted by Phatty Lumpkin at 9:07 AM on August 11, 2014 [5 favorites]
It's that magical time of year when this year's web pages for accepted SIGGraph papers blossom.
posted by ardgedee at 9:08 AM on August 11, 2014 [11 favorites]
posted by ardgedee at 9:08 AM on August 11, 2014 [11 favorites]
> The continual adjustments in the terrain remind me of walking through a video game world, watching objects pop into view and increase resolution as they get nearer.
Also amusing to see humans momentarily snap from one pose to another or leap between action modes, kind of like NPCs in a video game world.
posted by ardgedee at 9:12 AM on August 11, 2014
Also amusing to see humans momentarily snap from one pose to another or leap between action modes, kind of like NPCs in a video game world.
posted by ardgedee at 9:12 AM on August 11, 2014
What's amazing is that these actually feel more natural and give more of a sense of presence than the raw input video. In the raw input video the head movements take away your sense of control and isn't how you would remember a ride down that street anyway. I really hope this becomes a Thing in the future.
posted by thewumpusisdead at 9:20 AM on August 11, 2014 [4 favorites]
posted by thewumpusisdead at 9:20 AM on August 11, 2014 [4 favorites]
Our wetware do an amazing amount of video processing. I think the reason this is so compelling it that it meshes with our memories of what we see, as opposed to the experience. This technique doesn't reproduce being there in the flesh, it's like a daydream, a recollection of a pleasant afternoon's ride.
posted by bonehead at 9:24 AM on August 11, 2014 [10 favorites]
posted by bonehead at 9:24 AM on August 11, 2014 [10 favorites]
Or rather, experiencing someone else's memory.
posted by bonehead at 9:28 AM on August 11, 2014 [1 favorite]
posted by bonehead at 9:28 AM on August 11, 2014 [1 favorite]
There is a certain tension between the slick, dreamlike finished product and how I surely would have reacted had I been walking down one of those streets when a dickhead cyclist with a fucking camera strapped to his head was riding along the sidewalk weaving between pedestrians.
posted by ricochet biscuit at 9:43 AM on August 11, 2014 [5 favorites]
posted by ricochet biscuit at 9:43 AM on August 11, 2014 [5 favorites]
But seriously, this is cool. Scorcese will go nuts!
posted by mazola at 9:43 AM on August 11, 2014 [2 favorites]
posted by mazola at 9:43 AM on August 11, 2014 [2 favorites]
I will be very interested to see if this makes it to the consumer desktop. I image the rendering time could be prohibitively long. On the other hand, if this could get a little smoother, who needs a Steadicam.
posted by doctor_negative at 9:59 AM on August 11, 2014
posted by doctor_negative at 9:59 AM on August 11, 2014
I wonder if this leverages any of the computer vision tech that went into Photosynth?
posted by The Legit Republic of Blanketsburg at 10:08 AM on August 11, 2014
posted by The Legit Republic of Blanketsburg at 10:08 AM on August 11, 2014
Hyperlapse videos are far less frightening than prolapse videos.
posted by fairmettle at 10:32 AM on August 11, 2014 [2 favorites]
posted by fairmettle at 10:32 AM on August 11, 2014 [2 favorites]
You know I think it does, TLRoB. Based on the description, I think they are literally creating a 3d environment of polygons that they then apply the appropriate texture from a single frame.
I think I've seen similar technology with the new 3D google maps, where they attempt (quite poorly sometimes) to paint the sides of (what appear to be) algorithmically-created 3d structures with images of what those "buildings" look like from streetview.
And this really bothers me but it also intrigues me. On the one hand, I don't like the idea of a video being turned into a 3d polygon, then manipulated, and then turned back into video. That seems too forced, too complicated, too outside the ken of photographers who now would have to work through yet another middle man to practice their art.
On the other hand, it really sounds like a jump forward in AI - the ability for a computer to look at an image or video and be able to "comprehend" it not just as different colored pixels, but as a 3d environment filled with objects.
posted by rebent at 10:34 AM on August 11, 2014 [1 favorite]
I think I've seen similar technology with the new 3D google maps, where they attempt (quite poorly sometimes) to paint the sides of (what appear to be) algorithmically-created 3d structures with images of what those "buildings" look like from streetview.
And this really bothers me but it also intrigues me. On the one hand, I don't like the idea of a video being turned into a 3d polygon, then manipulated, and then turned back into video. That seems too forced, too complicated, too outside the ken of photographers who now would have to work through yet another middle man to practice their art.
On the other hand, it really sounds like a jump forward in AI - the ability for a computer to look at an image or video and be able to "comprehend" it not just as different colored pixels, but as a 3d environment filled with objects.
posted by rebent at 10:34 AM on August 11, 2014 [1 favorite]
riding along the sidewalk weaving between pedestrians.
The paper mentions both biking and walking test videos. Of course, once they've gone through this process, you may not be able to tell the difference...
posted by effbot at 10:39 AM on August 11, 2014
The paper mentions both biking and walking test videos. Of course, once they've gone through this process, you may not be able to tell the difference...
posted by effbot at 10:39 AM on August 11, 2014
The paper mentions both biking and walking test videos. Of course, once they've gone through this process, you may not be able to tell the difference...
I base my supposition on the camera operator keeping pace with other cyclists visible and passing every pedestrian in the section I watched. Could be Usain Bolt with a fucking camera tied to his head, I suppose.
posted by ricochet biscuit at 10:46 AM on August 11, 2014
I base my supposition on the camera operator keeping pace with other cyclists visible and passing every pedestrian in the section I watched. Could be Usain Bolt with a fucking camera tied to his head, I suppose.
posted by ricochet biscuit at 10:46 AM on August 11, 2014
Hyperlapse videos are far less frightening than prolapse videos.
Goprolapse?
posted by pmcp at 10:49 AM on August 11, 2014 [4 favorites]
Goprolapse?
posted by pmcp at 10:49 AM on August 11, 2014 [4 favorites]
The height, speed and view angle are unmistakably a bike. The source vids can't be anything else.
posted by bonehead at 10:50 AM on August 11, 2014
posted by bonehead at 10:50 AM on August 11, 2014
Scorcese will go nuts!
I was thinking Frankenheimer.
riding along the sidewalk weaving between pedestrians
Cycling culture varies widely around the world still. There are places where this would be totally unacceptable and places where it would be normal.
posted by dhartung at 10:51 AM on August 11, 2014 [1 favorite]
I was thinking Frankenheimer.
riding along the sidewalk weaving between pedestrians
Cycling culture varies widely around the world still. There are places where this would be totally unacceptable and places where it would be normal.
posted by dhartung at 10:51 AM on August 11, 2014 [1 favorite]
He does look like he's on MUPs for most of it though---note the lane markings, typical of multi-use paths.
posted by bonehead at 10:52 AM on August 11, 2014 [2 favorites]
posted by bonehead at 10:52 AM on August 11, 2014 [2 favorites]
It's from Richard Szeliski's group, so it's fair to say that it is related to Photosynth. I skimmed through the paper and they seem to have tried using Photosynth methods in structure from motion and path smoothing but ended up using something different at both counts, so none of it is directly lifted from Photosynth.
My mountain bike videos need this very badly. At 1x they are extremely boring, at 4x only very boring and at 8x nausea-inducing. I have hours of material, none of which I have even considered showing anyone. Mounting the camera to the bike frame produces watchable 8x videos when driving on smooth roads, but on single-track paths 8x is still much too shaky.
posted by ikalliom at 10:59 AM on August 11, 2014
My mountain bike videos need this very badly. At 1x they are extremely boring, at 4x only very boring and at 8x nausea-inducing. I have hours of material, none of which I have even considered showing anyone. Mounting the camera to the bike frame produces watchable 8x videos when driving on smooth roads, but on single-track paths 8x is still much too shaky.
posted by ikalliom at 10:59 AM on August 11, 2014
Cycling culture varies widely around the world still. There are places where this would be totally unacceptable and places where it would be normal.
No doubt. The trouble is when the different places are the same place.
posted by ricochet biscuit at 11:02 AM on August 11, 2014 [4 favorites]
No doubt. The trouble is when the different places are the same place.
posted by ricochet biscuit at 11:02 AM on August 11, 2014 [4 favorites]
I'm not prone to motion sickness, but after about 30 seconds of watching the hyperlapse I really wanted to turn it off. The perspective reminded me of first person shooters, which I don't have a lot of experience with either. It feels like if my field of view is panning my neck should be turning.
posted by benito.strauss at 11:28 AM on August 11, 2014 [1 favorite]
posted by benito.strauss at 11:28 AM on August 11, 2014 [1 favorite]
A transformed video footage throws away a lot of transient but potentially interesting details — people, noticeably.
It might be useful for some applications, but its end user (sports film director, etc.) would definitely want to understand what it discards.
posted by Mr. Six at 11:43 AM on August 11, 2014
It might be useful for some applications, but its end user (sports film director, etc.) would definitely want to understand what it discards.
posted by Mr. Six at 11:43 AM on August 11, 2014
this has the same feel as the timelapse of pyongyang that went around recently.
posted by bruceo at 11:50 AM on August 11, 2014
posted by bruceo at 11:50 AM on August 11, 2014
Auto-tune for GoPro.
posted by Anticipation Of A New Lover's Arrival, The at 11:55 AM on August 11, 2014 [2 favorites]
posted by Anticipation Of A New Lover's Arrival, The at 11:55 AM on August 11, 2014 [2 favorites]
This is definitely building on Photosynth-like procedures. In fact, based on their making-of video, the creation is way more complex than I would have expected, really an impressive case of (very successful) overkill. Why do clever frame selection and frame-matching when you could just reconstruct the entire 3D world, replot the camera through virtual space, and then restitch and white-balance the photos into that virtual world?
posted by chortly at 12:36 PM on August 11, 2014 [2 favorites]
posted by chortly at 12:36 PM on August 11, 2014 [2 favorites]
(I'm glad MetaFilter has yet another threat for that scourge of cities everywhere, the bicycle. Known for its pollution and the tens of thousands of deaths they cause every year, I'm glad somebody has the strength to stand up to this menace. )
That said, this looks awesome and I hope there is a Mac version in the works. (Shut up, I can dream.)
posted by entropicamericana at 1:03 PM on August 11, 2014 [1 favorite]
That said, this looks awesome and I hope there is a Mac version in the works. (Shut up, I can dream.)
posted by entropicamericana at 1:03 PM on August 11, 2014 [1 favorite]
Granted, if you're shooting a walk through a park or climb up a mountain, real-time is going to be agonizing to watch, but bombing down South American favelas doesn't need any speeding up, it's dizzying already. So I wonder why they keep emphasizing how important speeding up the playback is. Is it to help mask the artifacting that the reconstruction produces?
posted by ardgedee at 1:37 PM on August 11, 2014
posted by ardgedee at 1:37 PM on August 11, 2014
> Auto-tune for GoPro.
Yeah, although I wonder if some form of this won't get applied to cinema and video effects libraries. Imagine shooting on location; now you can get away with much smaller, lighter equipment for certain takes because you don't need a mounted or ballasted rig to get a Steadicam-style moving shot.
posted by ardgedee at 1:40 PM on August 11, 2014
Yeah, although I wonder if some form of this won't get applied to cinema and video effects libraries. Imagine shooting on location; now you can get away with much smaller, lighter equipment for certain takes because you don't need a mounted or ballasted rig to get a Steadicam-style moving shot.
posted by ardgedee at 1:40 PM on August 11, 2014
"We are working hard on making our Hyperlapse algorithm available as a Windows app"
They may not explicitly mention it, but I guarantee it will be on iOS as well. I'll place the icon right next to Photosynth, too.
posted by phong3d at 1:43 PM on August 11, 2014
They may not explicitly mention it, but I guarantee it will be on iOS as well. I'll place the icon right next to Photosynth, too.
posted by phong3d at 1:43 PM on August 11, 2014
fairmettle: You are a bad, bad person.
posted by Chitownfats at 2:31 PM on August 11, 2014
posted by Chitownfats at 2:31 PM on August 11, 2014
According to the paper the current implementation is painfully slow on desktop PCs. Several of the processing stages are listed to take 1 hour of processing time for the 13 minute Bike3 video. However, "Source selection" takes the cake with 1min/frame. That means 16.5 days of computing time for the 23700 frames of Bike3 if I understood Table 2 correctly. Or like the authors put it, "our current implementation is slow". I would be surprised to see this on iOS anytime soon.
posted by ikalliom at 2:52 PM on August 11, 2014 [2 favorites]
posted by ikalliom at 2:52 PM on August 11, 2014 [2 favorites]
Or you could take a camera equipped Chicken on your next bike ride.
posted by boilermonster at 11:28 PM on August 11, 2014 [1 favorite]
posted by boilermonster at 11:28 PM on August 11, 2014 [1 favorite]
ExtremeTech article on Hyperlapse has a quote from Kopf saying the processing time is now down to a couple of hours for a 10 minute video on a desktop PC, which seems already quite usable. Hopefully the parameters of the algorithm do not require too much per-video adjustment for good results.
posted by ikalliom at 1:43 PM on August 12, 2014
posted by ikalliom at 1:43 PM on August 12, 2014
Yeah, I didn't mean to be dismissive of this with my glib "Autotune for GoPro" comment; this is genuinely quite cool. So cool, in fact, that I can see it becoming quickly ubiquitous once it becomes generally available and the kinks are smoothed out.
I could see it being a feature for YouTube uploads, and the applications for phone cams and gopros and such are so natural that I can't see it not catching like wildfire. May take a few more generations of refinement before we see it in more professional cinematic settings, but once it's ready I think it'll quickly catch on there too, as sort of a reverse bullet-time.
Like bullet-time though, or like auto-tune, I think we're in for a phase of tiresome overuse before it calms down and becomes just another effect among many.
posted by Anticipation Of A New Lover's Arrival, The at 6:58 AM on August 14, 2014 [1 favorite]
I could see it being a feature for YouTube uploads, and the applications for phone cams and gopros and such are so natural that I can't see it not catching like wildfire. May take a few more generations of refinement before we see it in more professional cinematic settings, but once it's ready I think it'll quickly catch on there too, as sort of a reverse bullet-time.
Like bullet-time though, or like auto-tune, I think we're in for a phase of tiresome overuse before it calms down and becomes just another effect among many.
posted by Anticipation Of A New Lover's Arrival, The at 6:58 AM on August 14, 2014 [1 favorite]
« Older Reasoning with your muscles. | Today We Honor Ralph Wiggum Newer »
This thread has been archived and is closed to new comments
posted by Tell Me No Lies at 8:52 AM on August 11, 2014 [1 favorite]