Painterly depiction of material properties
August 27, 2024 4:34 AM Subscribe
"Furthermore, painters are not constrained by reality, meaning that they could paint materials without exactly following the laws of nature, while still evoking the perception of materials." [SLPDF]
(This post brought to you as a research "offcut" associated with a story I'm working on that involves mental processing of art images.)
Abstract:
Painters are masters of depiction and have learned to evoke a clear perception of materials and material attributes in a natural, three-dimensional setting, with complex lighting conditions. Furthermore, painters are not constrained by reality, meaning that they could paint materials without exactly following the laws of nature, while still evoking the perception of materials. Paintings have to our knowledge not been studied on a big scale from a material perception perspective. In this article, we studied the perception of painted materials and their attributes by using human annotations to find instances of 15 materials, such as wood, stone, fabric, etc. Participants made perceptual judgments about 30 unique segments of these materials for 10 material attributes, such as glossiness, roughness, hardness, etc. We found that participants were able to perform this task well while being highly consistent. Participants, however, did not consistently agree with each other, and the measure of consistency depended on the material attribute being perceived. Additionally, we found that material perception appears to function independently of the medium of depiction—the results of our principal component analysis agreed well with findings in former studies for photographs and computer renderings.
Citation:
van Zuijlen, Mitchell JP, Sylvia C. Pont, and Maarten WA Wijntjes. "Painterly depiction of material properties." Journal of vision 20, no. 7 (2020): 7-7.
(This post brought to you as a research "offcut" associated with a story I'm working on that involves mental processing of art images.)
Abstract:
Painters are masters of depiction and have learned to evoke a clear perception of materials and material attributes in a natural, three-dimensional setting, with complex lighting conditions. Furthermore, painters are not constrained by reality, meaning that they could paint materials without exactly following the laws of nature, while still evoking the perception of materials. Paintings have to our knowledge not been studied on a big scale from a material perception perspective. In this article, we studied the perception of painted materials and their attributes by using human annotations to find instances of 15 materials, such as wood, stone, fabric, etc. Participants made perceptual judgments about 30 unique segments of these materials for 10 material attributes, such as glossiness, roughness, hardness, etc. We found that participants were able to perform this task well while being highly consistent. Participants, however, did not consistently agree with each other, and the measure of consistency depended on the material attribute being perceived. Additionally, we found that material perception appears to function independently of the medium of depiction—the results of our principal component analysis agreed well with findings in former studies for photographs and computer renderings.
Citation:
van Zuijlen, Mitchell JP, Sylvia C. Pont, and Maarten WA Wijntjes. "Painterly depiction of material properties." Journal of vision 20, no. 7 (2020): 7-7.
Thanks, HearHere, you're too kind. If it appears in the world, I'll link somewheres (permitted) here.
posted by cupcakeninja at 5:00 AM on August 27 [4 favorites]
posted by cupcakeninja at 5:00 AM on August 27 [4 favorites]
Thanks for posting this, it looks very interesting.
posted by mumimor at 6:00 AM on August 27 [2 favorites]
posted by mumimor at 6:00 AM on August 27 [2 favorites]
Interesting, thanks for the link cupcakeninja!
Material approximation is a regular exercise done by 3D and concept artists, specifically for a stylized, hand-painted look.
Seeing studies like the one above mentioned in this post, I've wondered if they've been affected by the advent of "physically-based rendering" (PBR), where what something "should" look like is given an exact numerical value, describing albedo, roughness/gloss, metalness, etc.
So now artists have an actual numerical reference point to work with, adjusting the material qualities either toward or further away from those data points, depending on their interpretation of what "feels" right.
posted by ishmael at 9:55 AM on August 27 [1 favorite]
Material approximation is a regular exercise done by 3D and concept artists, specifically for a stylized, hand-painted look.
Seeing studies like the one above mentioned in this post, I've wondered if they've been affected by the advent of "physically-based rendering" (PBR), where what something "should" look like is given an exact numerical value, describing albedo, roughness/gloss, metalness, etc.
So now artists have an actual numerical reference point to work with, adjusting the material qualities either toward or further away from those data points, depending on their interpretation of what "feels" right.
posted by ishmael at 9:55 AM on August 27 [1 favorite]
I've wondered if they've been affected by the advent of "physically-based rendering" (PBR), where what something "should" look like is given an exact numerical value, describing albedo, roughness/gloss, metalness, etc.
I’ve spent about 30% of my professional life in realtime tech art the past several years (Unreal, which uses PBR, though much of what I’m about to say applies to at least a few other engines), and yes this affects us and makes things difficult when going for a toon shader or painterly style. For probably most developers these days the majority of materials work is done using a high-level node-based programming language structured so as to reflect the unique needs of the GPU and its hundreds of vector math pipelines (minimal branching, tight controls on number of textures sampled to reduce state flushes when virtual texturing fails, etc.). Unreal does this exceptionally well but other engines have taken it up. Actually going into the HLSL source is relatively uncommon and only done by specialist graphics programmers…or Tech Designers who are bad at knowing their limits *cough*. This isn’t because tech artists aren’t adept at simple or intermediate shadercode, but rather the overall structural complexity of modern realtime renderer architecture is pretty near the ceiling for biological intelligence as we know it. Adding post process tonal maps or special treatments for specific objects in post (eg fake rimlights on characters) is easy, but the odds of screwing something up by modifying how lighting works more fundamentally are damned high.
The shift to deferred rendering in the late aughts/early teens means that we lost any semblance of a single lighting vector for a given fragment. Lights are composited into a standalone buffer after depth pre-pass and separate from base/diffuse (raw color). Without a definitive light vector during base, getting proper localized tri-color banding (or any form of illustrative look) is barely feasible. Most games going for any kind of toon stylization fake it with just tracking the dominant light for a given object and passing its properties as a pair of float4 parameters (RGB|Intensity, XYZ|Radius) to a unique shader instance, or those plus a separate set for the sun (as above + directional vector param). If you want an excellent deepdive into overcoming these challenges, I cannot recommend this breakdown on Guilty Gear Xrd rendering highly enough (talk starts at 1:30 after the game trailer/showcase).
Closer to home, our brilliant lead graphics programmer on Bioshock Infinite wrote a hell of a technical report on splicing deferred lighting - but not full deferred rendering - into Unreal 3.5’s forward-renderer (4.0 in 2014 marked the transition to fully deferred + PBR), and making this work with our “physically-influenced” semi-painterly hybrid visual style. Hopefully this will lend some teeth to my claim of “ceiling of biological intelligence” above, for anyone who raised an eyebrow at that. The key insight of exploiting the four color theorem for compositing baked shadows with fully dynamic scene geometry still leaves me astounded a decade later, and I’ve utilized the underlying insight in multiple ways on multiple projects since, often for projects with heavy illustrative stylization.
I don’t have time to read parent post’s paper tonight, but I’m looking forward to sitting down with it this weekend.
posted by Ryvar at 11:57 PM on August 27 [2 favorites]
I’ve spent about 30% of my professional life in realtime tech art the past several years (Unreal, which uses PBR, though much of what I’m about to say applies to at least a few other engines), and yes this affects us and makes things difficult when going for a toon shader or painterly style. For probably most developers these days the majority of materials work is done using a high-level node-based programming language structured so as to reflect the unique needs of the GPU and its hundreds of vector math pipelines (minimal branching, tight controls on number of textures sampled to reduce state flushes when virtual texturing fails, etc.). Unreal does this exceptionally well but other engines have taken it up. Actually going into the HLSL source is relatively uncommon and only done by specialist graphics programmers…or Tech Designers who are bad at knowing their limits *cough*. This isn’t because tech artists aren’t adept at simple or intermediate shadercode, but rather the overall structural complexity of modern realtime renderer architecture is pretty near the ceiling for biological intelligence as we know it. Adding post process tonal maps or special treatments for specific objects in post (eg fake rimlights on characters) is easy, but the odds of screwing something up by modifying how lighting works more fundamentally are damned high.
The shift to deferred rendering in the late aughts/early teens means that we lost any semblance of a single lighting vector for a given fragment. Lights are composited into a standalone buffer after depth pre-pass and separate from base/diffuse (raw color). Without a definitive light vector during base, getting proper localized tri-color banding (or any form of illustrative look) is barely feasible. Most games going for any kind of toon stylization fake it with just tracking the dominant light for a given object and passing its properties as a pair of float4 parameters (RGB|Intensity, XYZ|Radius) to a unique shader instance, or those plus a separate set for the sun (as above + directional vector param). If you want an excellent deepdive into overcoming these challenges, I cannot recommend this breakdown on Guilty Gear Xrd rendering highly enough (talk starts at 1:30 after the game trailer/showcase).
Closer to home, our brilliant lead graphics programmer on Bioshock Infinite wrote a hell of a technical report on splicing deferred lighting - but not full deferred rendering - into Unreal 3.5’s forward-renderer (4.0 in 2014 marked the transition to fully deferred + PBR), and making this work with our “physically-influenced” semi-painterly hybrid visual style. Hopefully this will lend some teeth to my claim of “ceiling of biological intelligence” above, for anyone who raised an eyebrow at that. The key insight of exploiting the four color theorem for compositing baked shadows with fully dynamic scene geometry still leaves me astounded a decade later, and I’ve utilized the underlying insight in multiple ways on multiple projects since, often for projects with heavy illustrative stylization.
I don’t have time to read parent post’s paper tonight, but I’m looking forward to sitting down with it this weekend.
posted by Ryvar at 11:57 PM on August 27 [2 favorites]
Flagged as fantastic, Ryvar. Thank you.
posted by cupcakeninja at 4:05 AM on August 28 [2 favorites]
posted by cupcakeninja at 4:05 AM on August 28 [2 favorites]
If it appears in the world, I'll link somewheres (permitted) here
i'll keep checking mefi projects :-)
posted by HearHere at 3:26 PM on September 12
i'll keep checking mefi projects :-)
posted by HearHere at 3:26 PM on September 12
« Older sunshine follows thunder | Canadian PM Justin Trudeau's government announces... Newer »
This thread has been archived and is closed to new comments
looking forward to this!
posted by HearHere at 4:43 AM on August 27 [2 favorites]