"It’s like people who try to clone their dogs"

January 3, 2022 10:44 AM

"For decades, only black-and-white photographs of [Klimt's painting] Philosophy existed. Now, thanks to artificial intelligence, we can see the work in full color. But does the re-creation really look like the original? Does it even look like a Klimt?"
posted by jessamyn (23 comments total)

As a self-confessed AI doubter, I would like to know if they tried these same techniques to recolor a number of known, full color Klimt paintings from B&W renditions? This seems to be the only obvious test of the "accuracy" of the process.
posted by njohnson23 at 10:59 AM on January 3, 2022 [23 favorites]

It's always interesting to me when people are really invested in holding up a black and white photo as the best version of an artwork and complaining about colorization as inaccurate to the original when the black and white is also inaccurate to the original. I get the conflict, and that the colorized version is never going to be the same thing, but when we're trying to find the closest thing to what the original artist had made, I'd probably go for showing multiple versions so people can make their own comparisons.
posted by bile and syntax at 11:03 AM on January 3, 2022 [3 favorites]

The article doesn't seem to mention if there are any known contemporaneous descriptions of the painting. That would seem a useful clue.
posted by jedicus at 11:04 AM on January 3, 2022 [8 favorites]

I have it on good authority that the world was black and white then.
posted by TedW at 11:13 AM on January 3, 2022 [15 favorites]

Some more details from Google here. It includes some examples of it "restoring" Klimt works that survive and comparing them to the real thing. The model was trained to be good at doing that, but that doesn't necessarily mean that a model that's good at colorizing a bunch of known Klimt paintings can colorize these lost ones convincingly.
posted by BungaDunga at 11:15 AM on January 3, 2022 [7 favorites]

So based on that Google writeup, it sounds like they took two ML models, one trained on Klimt paintings and one trained on a million unrelated photographs (DeOldify), and combined them to use the colors inferred by the first one to instruct the second one to use Klimt's style rather than the non-style of the DeOldify dataset. After that they intervened manually based on historical descriptions of the painting and other research.
posted by BungaDunga at 11:24 AM on January 3, 2022 [4 favorites]

Interesting. I never heard of Klimt until one day, I happened upon some tins of coffee (from Austria) with his artwork. Bought for the tin, and discovered the coffee was actually fabulous.

The Klimpt Tin
posted by Goofyy at 11:26 AM on January 3, 2022 [3 favorites]

I have it on good authority that the world was black and white then.

Oh, God, one of my oldest friends always used to say he'd pulled that on his own kids and then I told it here to get that thrown in my face. I've bitten my tongue about it with him ever since.

On topic: Oh, jeez, what a crap shoot and who's to tell when AI's start in with tropes of the colorization of the black and white world in every multi-dimensional direction? Imagine the future with the Matrix's boot stamping into our Octopoid sensory cluster forever.
posted by y2karl at 11:29 AM on January 3, 2022 [2 favorites]

The article doesn't seem to mention if there are any known contemporaneous descriptions of the painting.

Saw this at the recent Klimt exhibition here in Rome, and from how it was described there, yes, they worked period comments about and descriptions of the work into the premise they then set the AI. All in all… still felt pretty gimmicky.

But the rest of the work shown (and the way it was all organized and contextualized) was pretty awesome. Klimt’s work will likely always seem decorative, but there really is a whole lot more going on when you get time & space to sit with it.
posted by progosk at 11:29 AM on January 3, 2022 [1 favorite]

On a sidenote, one of my favorite PBS series is the BBC's Vienna Blood, set in the Vienna of the Klimt era. It is really really good.
posted by y2karl at 11:39 AM on January 3, 2022

Oh and: there's an art historian who has made a pretty detailled case disputing that these panels were, as is generally presumed, lost in a fire at Schloss Immendorf; here's a short Austrian TV segment about this...
posted by progosk at 11:42 AM on January 3, 2022 [5 favorites]

Its a bit misleading for the article to compare the recolored "Philosophy" to "The Kiss" or "Adele Bloch Bauer I," both of which appear use a huge amount of actual gold for huge swathes of the painting. (Incidentally, this means that no photo representation of those paintings will be accurate to the real thing.) It might be better to compare it to something like Danae, where the use of gold is more limited, or his later landscape paintings, which used a lot of green.

Also, did anybody ever try to make a copy? Are there no student studies of the work? That might get you in the ballpark, colorwise. Hmm. It just occurred to me to wonder if different black and white films might be differently sensitive to colors. In that case, a restorer might be able to use that fact to reconstruct the actual colors, if they had enough photographs, and knew what film was used.
posted by surlyben at 12:29 PM on January 3, 2022 [2 favorites]

Also, I think it's pretty cool that they are trying to restore lost work. I'd only have a problem with it if they were trying to destroy the sources in order to do it, but that would be crazy.
posted by surlyben at 12:31 PM on January 3, 2022

This far in to a Klimt thread and no Back to School reference?

Trendy Man: Mr. Melon, your wife was just showing us her Klimt.
Thornton Melon: You too, huh? She's shown it to everybody.
Trendy Man: Well, she's very proud of it.
Thornton Melon: I'm proud of mine too. I don't go waving it around at parties, though.
Trendy Man: It's an exceptional painting.
Thornton Melon: Oh, the painting.
posted by Sphinx at 12:35 PM on January 3, 2022 [3 favorites]

This reminds me of the use of AI to de-blur and/or enlarge photographs—they accomplish images with a strong reality-effect. For instance, taking an image of a license plate represented as a 8×4 grid of pixels and guess a number that could have produced it. Doesn't mean you actually know what the car's license plate number was.
It's GAN-generated simulacra, all the way down.
posted by signal at 1:29 PM on January 3, 2022 [1 favorite]

Just as an aside, when I saw Klimt and Philosophy in the same sentence, I thought of Klimt's portrait of Ludwig Wittgenstein's sister.
posted by StickyCarpet at 2:12 PM on January 3, 2022 [3 favorites]

signal: taking an image of a license plate represented as a 8×4 grid of pixels and guess a number that could have produced it

Having done some courtroom digital evidence enhancement, to be valid in court the digital processing must be "uniform global", a pixel-by-pixel convolution with the exact same operation. There can't be any if/then conditionals, because like GANs, the algorithm could be conditionalizing to a desired outcome.
posted by StickyCarpet at 2:23 PM on January 3, 2022 [1 favorite]

All in all… still felt pretty gimmicky.

Yeah I've been curious about this both because of the breathless way it's beeen described, the generally good outcome they got (but is it accurate?) and the fact that it takes a shitton of work--thanks BungaDunga for that link--which may or may not lead to better outcomes. And wow, progosk , I didn't know that story at all, though the idea of lost-not-lost art is always interesting to me.
posted by jessamyn at 2:24 PM on January 3, 2022 [1 favorite]

I find it interesting that this article slides around describing this as a version, a recreation, and a colorization. Those are all very different things, and it seems odd to say these are all of these.
posted by lesbiassparrow at 3:08 PM on January 3, 2022 [2 favorites]

Thanks for this post!
To be very specific and detailed, what I think the AI version is missing, and maybe can't be programmed to reproduce, is the characteristic and idiosyncratic manner in which Klimt mixed flat and spatial methods of painting/representation. Most of his mature painting was almost all flat, but in each piece, he would let spatial elements pop out. In the AI version, everything is spatial, nothing is flat. If you look at his early work, there will be more space and less flat, but this is not that.
Look at the female nudes in the B/W image: they are very white, with little modulation of light and no nuances of skin tone. I think that was how they were. It isn't that nuance was lost in the B/W image, it was that Klimt tried to see how far he could go with just line and plane.
With the green thing in the right of the painting, I think it was much more similar to other Klimt decorative flat parts than the weird forest monster we see in the AI. I think he was going for a mosaic feeling, like in Ravenna, where there are many weird combinations of abstraction, realism and symbolism with a big dose of materiality, like this. The context here is not irrelevant: it was planned to be an integrated artwork, part of a public space. Not a framed piece over a sofa in a private living room. In the opinion of many early twentieth century artists, that context called for a more graphic and surprising composition. The details of the pimples in a lady's thigh were fun to notice if you were in a rococo salon and the painting was right next to you, but less interesting when viewed from a distance of 10 meters in a public space, because in such a context, the details will just blur and make the work look like a paint salad. Even rococo paintings in the public realm are far less "realistic" than the AI here suggests, when you look at them close up.
So Klimt's private portraits might have more realistic detail, albeit still with flat elements, but for his integrated works, or any other integrated works of the early twentieth century, the AI version seems unlikely, both for practical and ideological reasons.
posted by mumimor at 3:33 PM on January 3, 2022 [10 favorites]

This might be good PR for Google's AI division, which it certainly needs after scaring everybody with its robocop-dogs, but by no stretch of the imagination is it a 'restoration'. It's an interpretation, a cover version. The fact that it looks 'real' doesn't make it real.
posted by signal at 3:47 PM on January 3, 2022 [1 favorite]

the idea of lost-not-lost art is always interesting to me.

The exhibit in Rome also reserved a space for the other, more well-known, lost-not-lost Klimt, the one that hadn't really ever left the Ricci Oddi at all...
posted by progosk at 3:51 PM on January 3, 2022 [2 favorites]

> The article doesn't seem to mention if there are any known contemporaneous descriptions of the painting. That would seem a useful clue.

From the article:

That was part of the impetus to recolor the works with AI. Emil Wallner, a researcher at Google, built the algorithm. He used 100,000 art historical references and programmed it to have a bias toward Klimt’s style. For his part, Smola combed through articles and texts where writers responded to the faculty paintings, seeking an objective sense of color in the subjective writing of criticism.

Throughout the process, Smola had to make bold choices, such as instructing the AI to color “Jurisprudence’s” background red, a decision made after discovering that one of Klimt’s biggest critics noted that “Jurisprudence” featured the colors of the German flag.

That said, I love watching AI develop, both in the ways it's faikling and the ways it's succeeding. This is a mixed bag, but I agree with the comments upthread: did they try to take known Klimts and turn b&w into color? And yeah, we may have b&w but they are also not a perfect representation of Klimt's intent either. (I thought using descriptions of the piece was crucials, so I was glad they mentioned they did that. Still, very speculative.)

Anyway, earn cash money now let me show you how.
posted by not_on_display at 6:53 PM on January 3, 2022