How AI learned to paint like Rembrandt

Robert Erdmann, a senior scientist working for the Rijksmuseum, cannot help but smile when I ask him to explain — in as much detail as possible — how exactly he used artificial intelligence to recreate long-lost portions of Rembrandt van Rijn‘s most famous painting, The Night Watch (1642). “Most people just want the elevator pitch,” he tells me over Zoom.

The Night Watch is a mammoth of a painting, and it used to be even bigger. In 1715, it came into the possession of the bureaucrats in charge of Amsterdam’s Town Hall. In order to fit it on their wall, they sliced off all four outer edges of Rembrandt’s priceless masterpiece, inadvertently creating the compromised version we know today.

The missing pieces of “The Night Watch”were never recovered, but we know what they looked like thanks to Gerrit Lundens, a contemporary of Rembrandt who copied the painting when it was complete. These missing sections depict the top of the arch, a balustrade at the bottom, and two soldiers of Frans Banninck Cocq’s militia company that stood at the far left.

Though the absence of these elements does not make “The Night Watch“ any less impressive, their presence greatly alters the painting’s look and feel. The balustrade emphasizes the company’s movement forward. Together, the four missing pieces shift the principal figures — Cocq and Willem van Ruytenburch — to the right, creating a more compelling composition.

As part of Operation Night Watch, a multimillion-dollar restoration mission, the Rijksmuseum set out to recreate these missing pieces of the painting to show visitors The Night Watch as Rembrandt had originally constructed it. One easy way to do this would be to upload the smaller Lundens copy into Photoshop, blow it up by a factor of five, print it out, and call it a day.

Easy, but far from adequate. As Erdmann puts it: “There’s nothing wrong with using an artist like that. However, the final product would still contain traces of that artist’s own style.” For Erdmann, the only viable solution was to create a series of neural networks — software that mimics the human brain through the use of artificial neurons — to transform the Lundens copy into an “original” Rembrandt.

Humans, unlike computers, aren’t able to make perfect copies. Faithful though Lundens’ painting is — especially in its visual detail, for example, the number of buttons on a coat, plumes on a feather, or engravings on a halberd — it still contains a myriad of miniscule differences that prevented Erdmann from simply copy-pasting it onto the original.

Perspective was the first and arguably most important item on Erdmann’s list. “The geometric correspondence is pretty good at the bottom of the copy,” he says. “At the top, that correspondence starts to fall apart; the composition looks stretched out, supposedly because Lundens was unable to reach the top of the painting to get its precise measurements.”

After creating a neural network that could identify corresponding elements in both versions of The Night Watch — from faces and hands to clothing and weapons — Erdmann made a second neural network that could stretch, rotate, foreshorten, compress, and decompress the Lundens copy so that its measurements matched the Rembrandt original as closely as possible.

According to Erdman, this step was “a guide to where we should place the figures on the left, because they need to be consistent with the extrapolation from the original Night Watch.” Aside from aligning the two paintings, Erdmann’s adjustments also transformed the facial structure of figures like Cocq, bringing them closer to Rembrandt’s expert rendering.

Just as a painter must tone their canvas before they can work on composition and color, so too did Erdmann have to get the dimensions right before he could move on to the third and final stage of his coding process. Erdmann’s next part of the neural network involved — to paraphrase his elevator pitch — sending the artificial intelligence algorithm to art school.

“Not unlike how you might translate a text from Dutch to English, we wanted to see if we could transform Lundens’ painterly style and palette into Rembrandt’s,” he explains, comparing the learning curve to a quiz. To educate it, the AI was given random tiles from the Lundens copy and asked to render the tiles in the style of Rembrandt.

As with any pedagogical situation, Erdmann evaluated the AI’s efforts with a corresponding grade. The closer its output matched the contents of the original Night Watch, the higher the grade it received. When grading, Erdmann considered things like color, texture, and representation (i.e., how well does this frowning face resemble a frowning face, or this sword an actual sword?).

“Once you’ve defined what makes a good copy, you can train the network on thousands and thousands of these tiles,” Erdmann goes on. There are 265 gigabytes of memory of thousands of attempts stored, which demonstrates improvement in quality over a very short time. Within less than a day, the error margin between the AI and the real Rembrandt grew so small it became insignificant; the training was complete.

Along the way, the AI had developed a thorough understanding of what made Rembrandt Rembrandt. When translating Lundens’ copy, it used a less saturated color palette and thicker, sketchier brushstrokes. It even adopted the painter’s signature use of chiaroscuro — a technique involving sharp contrasts between light and shadow.

Then it was time for the final exam. Using the knowledge gained from copying Rembrandt, Erdmann ordered the AI to transform the four outer edges of the Lundens copy — removed from the original Night Watch — into Rembrandt’s signature style. The result, an unprecedented collaboration between man and machine, is now on display in the Eregalerij of the Rijksmuseum.

The missing pieces, resuscitated by AI, were printed onto canvas and varnished so that they had a similar gloss to the rest of the painting. The pieces were then attached to metal plates, which were placed in front of the original Night Watch at a distance of less than one centimeter, thus creating an optical illusion for visitors without actually touching Rembrandt’s work.

While conservation science is evolving rapidly, the achievements of people like Erdmann are still eclipsed by the artistic genius of the painters whose work they try to preserve, which is a shame because Erdmann’s software can be just as inventive as Rembrandt’s brushwork. At the very least, Erdmann’s problem-solving skills would have made the master proud.