A little blob of Infinity

January 26, 2022 7:03 AM

The Biography of the Pixel - The Elementary Particle of Pictures "I have billions of pixels in my cellphone, and you probably do too. But what is a pixel? Why do so many people think that pixels are little abutting squares? Now that we’re aswim in an ocean of zettapixels (21 zeros), it’s time to understand what they are. The underlying idea – a repackaging of infinity – is subtle and beautiful. Far from being squares or dots that ‘sort of’ approximate a smooth visual scene, pixels are the profound and exact concept at the heart of all the images that surround us – the elementary particles of modern pictures." An essay by Alvy Ray Smith, one of the early founders of Pixar.

"Thus, a digital picture is like a bed of nails, each nail a pixel. To restore the original picture to visibility, we spread each pixel with the little hillock spreader, and add up the results. That hillock – that little ‘blob of infinity’ – supplies the missing infinities between the pixels. That’s the repackaging of infinity. It’s a very neat trick. It’s not at all obvious that this scheme should work, but the mathematics of the sampling theorem proves it so. And it demonstrates once again the remarkable power of mathematics to take us to unintuitive and extremely useful places."

posted by storybored (31 comments total)

He’s really talking about compression, isn’t he?
posted by ChurchHatesTucker at 7:41 AM on January 26, 2022

Why do people think of pixels as abutting squares? Maybe because that’s what they traditionally have been. I guess they’ve been redefined since at least Microsoft Cleartype, but for those of us who grew up drawing pictures on graph paper and converting them to binary, a pixel is always going to be a literal box with a definitive value.
posted by rikschell at 8:00 AM on January 26, 2022 [7 favorites]

Fun article. (I admire the author's restraint in not pointing out that Fourier died while falling down the stairs.) I'm ashamed that I don't know anything about Kotelnikov.

It does seem worth pointing out that most detectors are also rectangular arrays of rectangles. Not always. But, it doesn't seem like too much of a stretch to look at a CCD and think of pixels as squares. The distinction is pretty subtle, even to someone who builds things called "pixels." But, interesting to point out and talk about.
posted by eotvos at 8:12 AM on January 26, 2022

So I guess he’s talking about converting real-world and imaginary pictures and curves into display points, which is one way to conceive of it, though most of that work was originally done by the human mind, not mathematically, in terms of hand-drawing pixel art, fonts, etc.

Reminds me of this, and the change from a halftone world to a pixelated one.
posted by rikschell at 8:13 AM on January 26, 2022 [2 favorites]

(previously, also btw ;)
posted by kliuless at 8:19 AM on January 26, 2022

I think the "little squares" interpretation is closer to the truth than the "point samples" interpretation: Smith writes "Picture-taking is a straightforward 2D sampling of the real world" but that's not true, a camera is a physical device and a sensor element can't sample a 1d light ray but has to integrate light-- a double integral in fact, over both the area of the camera lens and the area of the pixel itself, neither of which can ever be zero. (If you want to be precise there are two more integrated dimensions: time and wavelength of light). Once you enter the physical world everything has area, everything is integrals.
posted by Pyry at 8:19 AM on January 26, 2022 [3 favorites]

It's cool, sure, but this person is looking through the wrong end of the telescope. Pixels don't occupy zero space and this theory isn't what makes them operate, it merely explains why we see a screen as a contiguous field. Pixels really are, physically, points of light, square or some other shape, that occupy a non- zero region of space.

Or what everyone else said.
posted by klanawa at 8:22 AM on January 26, 2022 [1 favorite]

I found this Twitter account that shows how pixel art changes (and improves) when seen on a CRT display. There's an entire subreddit devoted to CRT gaming too.

I once heard that the "music sounds better on vinyl" trope was really just having nostalgia for the imperfect artifacts introduce by the limited technology of your youth. That tracks for me.

XKCD, of course, has a comic about how digital copies aren't such perfect copies after all.
posted by AlSweigart at 8:49 AM on January 26, 2022 [3 favorites]

Pointillism seems relevant

As important to Pointillism as any artist was the French chemist, Michel Eugène Chevreul – and his book, Principles of Harmony and Contrast of Colours. Employed by a Parisian tapestry works that wished to improve the strength of its colours, he discovered that the issue wasn't the dyes being used but the way different hues were being combined.

In short, the visual impact of a tapestry was actually a matter of optics, not chemistry. It depended on the juxtaposition of complementary colours (which enhanced each other's intensity) – blue and orange, for example. Seurat and the Pointillists drew heavily on Chevreul’s discoveries, applying to paints what the chemist had found in threads.
posted by Brian B. at 8:54 AM on January 26, 2022 [1 favorite]

In a very real sense the images you actually see were indeed once a 2D matrix of points. Points that were derived from the result of reading the voltage levels of a bunch of square pixels representing the intensity of light after that light was chromatically downsampled through a Bayer filter, which is itself a form of mathematical magic, but points nonetheless.

Thanks to Fourier (and some optical theory to account for diffraction, lens distortion, chromatic aberration, etc), we can know exactly how precise a representation of reality that plane of points actually is.
posted by wierdo at 9:02 AM on January 26, 2022 [1 favorite]

He has a point in one sense about pixels not being squares, but it doesn't hold fully:

If you zoom in on a photo (or other image), the Fourier transform should be able to smooth the borders between pixels so that there isn't an obvious pixel border. This is true for some image formats, and not for others.

However, they absolutely are squares insofar as the distribution of pixels is itself square. If your data describes values in a grid format, your pixels are square. Insisting that pixels are 0D when each "display element" is a square associated with a pixel is a quibbling technicality.
posted by explosion at 9:33 AM on January 26, 2022

I've always been fascinated by the fact that all pixelated displays have a finite number of images. It's a big number, sure, but still not infinite. Now if you could only separate the signal from the noise.
posted by pashdown at 9:57 AM on January 26, 2022 [3 favorites]

Used to be that pixels were fuzzy overlapping circular blobs, like Gaussians, where the electron beam hit the phosphored glass.

This is the two-dimensional version of how people often describe digital audio as stairstepped, which it fundamentally isn't, that's just one bad choice of reconstruction filter. A good reconstruction is smooth: its the same as the analog input, except without the frequencies too high for the digitization to carry.

The old CRT Gaussian was a better pixel than a sharp square is.
posted by away for regrooving at 10:02 AM on January 26, 2022 [6 favorites]

From the article:

Many people call these spots pixels – a very common error. Pixels are digital, separated, spiky things, and are invisible. The little glowing spots are analogue, overlapped, smooth things, and are visible. I suggest we call each a ‘display element’ to distinguish it from a ‘picture element’ (which is what the word ‘pixel’ abbreviates).

He blew right past "dixel" like it wasn't even there. Disappointing.
posted by Spathe Cadet at 10:16 AM on January 26, 2022 [8 favorites]

My understanding of Fourier transformation and using it to understand and process frequency-domain information is really getting in the way of understanding the basis of TFA. The author seems to imply that my photos are stored as frequency data, rather than spacial data (I understand that compression does weird stuff, but it's not pure FFT). To increase my comprehension I'm imagining the author's starting point as being a model/description of a picture rather than an actual picture (e.g. "imagine a red circle on a black field"). Also there are gems like this:

To see them, you ask for a picture file to be displayed. Typically, you ‘click on it’. Because of the astounding speed of today’s computers, this seems to happen instantaneously.

posted by achrise at 10:20 AM on January 26, 2022

He blew right past "dixel" like it wasn't even there. Disappointing.

I was hoping for "dispel" but maybe he didn't want to remove the magic.
posted by otherchaz at 10:41 AM on January 26, 2022 [2 favorites]

If your data describes values in a grid format, your pixels are square.

This is not true. Photoshop even has a built-in setting for Pixel Aspect Ratio.
posted by oulipian at 10:58 AM on January 26, 2022

He blew right past "dixel" like it wasn't even there. Disappointing.

I immediately thought of "dixel" too. He didn't address it, which suggests to me it did occur to him but he hoped it would pass unnoticed.

This is definitely interesting, but I think he's overstating the "dirty trick" being pulled by modern digital displays. A grid of pixels (as he describes them) naturally is best represented by a rectangular grid of display elements, since they have to be addressed in a logical (literally and figuratively) way.

As oulipian points out, there are certainly non-square pixels, and it's conceivable even to have some kind of exotic octagonal addressing grid with diagonal as well as horizontal and vertical lines defining a picture element's boundaries, or even something more analog than that. But the reason we are all able to use these things is because LCDs are a cheap and super effective way of representing pixels. (If you've worked in video, you know non-square pixels are super annoying to work with.)

That said it's good to keep in mind the mathematical nature of what you're seeing on any screen, since it's unnecessarily limiting to think of any computer's output as simply a grid of colored squares. Learning the history behind it is definitely illuminating and will certainly help me explain it going forward!
posted by BlackLeotardFront at 11:35 AM on January 26, 2022

One of my favorite paper titles ever is Smith's 1995 "A Pixel Is Not A Little Square, A Pixel Is Not A Little Square, A Pixel Is Not A Little Square! (And a Voxel is Not a Little Cube)". This is not a new discussion for him.
posted by grouse at 12:47 PM on January 26, 2022 [1 favorite]

I'm reminded that Nicholas Negroponte argued at the dawn of digital convergence that there was no good reason for a 1:1 pixel aspect ratio and that technology could, eventually, deal perfectly well with anisotropic displays. That really bugged me at the time but it turns out—many years later—that he was right but I'm still glad everybody ignored him.
posted by sjswitzer at 2:36 PM on January 26, 2022

Non-square pixels crop up in Quantum-Dot OLED and Apple's OLED displays. For example.
posted by k3ninho at 2:55 PM on January 26, 2022

This is not true. Photoshop even has a built-in setting for Pixel Aspect Ratio.

That was true with CRTs, at least. I worked with images destined for CRTs when I was designing DVDs and found it easier just to work with square pixels and a slightly larger image area (in the Y axis) and then resize to anamorphically compress the image. When displayed it would 'magically' be corrected.

(Flat panels changed all of that, of course.)
posted by Insert Clever Name Here at 3:30 PM on January 26, 2022

(Frickin CGA. 320×200 was close enough to square that lazy people would treat it that way, but 6:5 is painfully not-actually-square.)
posted by away for regrooving at 5:01 PM on January 26, 2022

Insisting that pixels are 0D when each "display element" is a square associated with a pixel is a quibbling technicality.

If a pixel in an image demanded that it must only be rendered on a single display element, it would be a total dixel.
posted by UN at 10:11 PM on January 26, 2022

The author seems to imply that my photos are stored as frequency data, rather than spacial data

If your photos are JPEGs, they are stored as frequency data. Most of the compression comes from throwing away the frequency components the algorithm thinks you can't see/will notice the least.
posted by wierdo at 10:52 PM on January 26, 2022 [2 favorites]

He’s really talking about compression, isn’t he?
Yes, I think so. He seems to get to the point about 10 minutes read in:

Many people call these spots pixels – a very common error. Pixels are digital, separated, spiky things, and are invisible. The little glowing spots are analogue, overlapped, smooth things, and are visible. I suggest we call each a ‘display element’ to distinguish it from a ‘picture element’ (which is what the word ‘pixel’ abbreviates). Display elements and pixels are fundamentally different kinds of things. Display elements vary from manufacturer to manufacturer, from display to display, and over time as display technologies evolve. But pixels are universal, the same everywhere – even on Mars – and across the decades.

I'm not sure I really agree with the part in italics. For example, he is at odds with the Wikipedia definition which says:

In digital imaging, a pixel, pel, or picture element is the smallest addressable element in a raster image, or the smallest addressable element in an all points addressable display device; so it is the smallest controllable element of a picture represented on the screen.

.
posted by rongorongo at 10:56 PM on January 26, 2022

This subject is near and dear to my heart, having first learned it in computer graphics class and eventually using it in my thesis. So no, it's not really about compression as people understand it but the realities of converting between analog and digital. And one of the misconceptions that people can form about that because of how much of the deep tech in that process is hidden, so it's not easy to see what sort of hidden assumptions are being made and what they mean -- in the limit, with enough pixels, it doesn't matter what shape they have and that's the real magic, so why worry about square versus gaussian? That's for boffins.

This is the two-dimensional version of how people often describe digital audio as stairstepped, which it fundamentally isn't, that's just one bad choice of reconstruction filter

Yep.

Fun fact, there are many real-world situations where we deal with non-square or non-cube pixels: medical imaging, for example, the pixel spacing in 2 dimensions is often different from the pixel spacing in the third dimension between "image slices" and medical tech has to handle that. Or remote sensing, where an airplane flying over the land takes a stripe of pixels (with a big spectrum of colors per pixel) every X milliseconds, and there's easily a big difference between the pixel spacing on the sensor stripe, and the pixel spacing along the flight path. Or even worse: NEXRAD digital weather radar! The voxels are in a spherical coordinate system, have very non-square shapes, and have three different resolutions for azimuth, altitude, and range.
posted by traveler_ at 11:56 AM on January 27, 2022 [2 favorites]

I read this article maybe a year ago and I thought it was fantastic.

Listen y'all can quibble all you want about rectangular arrays of sensors, about grids of samples, etc—but a pixel is a sample and a sample fundamentally is a point.

(Digital cameras typically use a Bayer filter in order to detect color, which makes their sensory array rectangular, sure, but not with equal types of pixel elements. A regular grid of samples doesn't mean the samples themselves are just squares or any kind of quadrilateral. Integrating infalling light over a small area still gives you a point sample--or rather, an estimate of the infinitesimal illuminance field.)

The fact that we have built much of our imaging infrastructure around arrays of colored squares gives rise to the understanding of pixels as having extension and shape. It could be otherwise.

As a clinical neurophysiologist this kind of article just like instantly tied up a lot of thoughts about sampling and the Nyqvist limit and all that stuff.
posted by adoarns at 1:05 PM on January 27, 2022 [4 favorites]

If a picture is worth a thousand words, it's because that picture is a million pixels.
posted by UN at 5:51 PM on January 27, 2022 [1 favorite]

I, I feel like I know less now, having read this all.

Magnets, how do they work?
posted by Don.Kinsayder at 5:55 PM on January 28, 2022

I admittedly couldn't quite follow the explanation of pixels and waves/Furnier, even though I write a lot of image processing software.

In either case, on the topic of square/rectangular pixels, this is my simplified explanation of how they work (and why a pixel is not a rectangle):

A lot of software handles pixels as a list of [R,G,B] values. A pixel that's completely red is [1,0,0]. Blue is [0,0,1]. Gray is [0.5,0.5,0.5].

Without any image compression or other fancy algorithms, a photo of a vibrant cloudless blue sky is something like [0,0,1][0,0,1][0,0,1][0,0,1][0,0,1] and so on.

What a pixel does not do is tell the computer where to draw a rectangle — or that it should draw a rectangle at all. In that way, a pixel is agnostic as to how it should be displayed. It's a number without a description of the shape it should have.

I write software for industrial knitting machines. When I write a program to knit images, I take the raw pixel data and turn each pixel into a stitch of yarn. These stitches are anything but rectangular. They're fuzzy, squiggly things and each stitch is unlike the next.

When I take a photo and throw it into my knitting software directly, those pixels may may never experienced life as a rectangle.
posted by UN at 6:09 AM on January 29, 2022 [2 favorites]