When confronted with Lesley Dill’s White Poem Figure (The Soul Has Moments of Escape), students suggested they might use a denoising algorithm to extract data from the artwork.
A denoising algorithm compares pixels to their nearby neighbors. If there’s a pixel that’s really different from all of its neighbors, and it’s not on a boundary or an edge, then the algorithm would make the pixel more similar to its neighbors. Denoising Dill’s work would reduce scratches, smudges, and stray pixels in order to make color areas more uniform. Because a lot of the pixels are affected by the artist’s scratches, the algorithm might help the letters appear darker or clearer—more legible to us—and make the white spaces on the edges appear more bright.
Students suggested they might use an edge detection algorithm on Khalid al-Saa’i’s Winter in Ann Arbor.
An edge detection algorithm finds the largest difference between neighboring pixel values, which helps the algorithm mark edges in an image. Thus, using edge detection would identify the boundaries between the edges of letters and the background, which would allow students to detect the really faint brushwork in the background and make legible letters that our eyes struggle to see.
Finally, optical character recognition might help identify features of Jasper Johns’s Figure 3 from ‘Black Numeral Series’ .
There are so many applications for which detecting handwritten characters are useful. For example, machine learning is used to very accurately sort mail at the Post Office or to make transactions at a bank. If you’ve ever used mobile deposit and taken a picture of your check, the app has to think about the image, using an algorithm trained on handwritten characters to check whether the numbers on the check match the number you claimed to be depositing.
Because accurately identifying handwritten characters was such a priority in early machine learning efforts, many algorithms were trained on the MNIST database, which contains over 60,000 handwritten characters to feed an algorithm. Today, the MNIST database is a benchmark standard used to test and train machine learning algorithms on their handwriting detection abilities.
If students use an algorithm that has been trained on the MNIST database, or on digit recognition, the algorithm may be able to immediately recognize the “3” in Johns’s piece, whereas humans may look at other features of the image first, given that the 3 is occluded and at such a large scale.