Shulou(Shulou.com)06/02 Report--

Selected from Nature

Author: David Adam

Participation: Hu Xiyue, Lu

Machine learning is helping experts find out which painter painted which work.

In March, a group of art thieves broke into a church in northwestern Italy and thought they were stealing a 17th-century work by the Flemish master Peter Brougell Jr. In fact, however, the local police already got the news and replaced the 3 million euro work "Crucifixion" with a cheap copy.

To be fair, many of Bruegel's works do look almost interchangeable, with a similar painting of the same name hanging in the Philadelphia Museum of Art in Pennsylvania. Moreover, the two paintings are most likely copied from another painting by his groundbreaking father, Peter Pieter Bruegel the Elder Sr., who also seriously affected his other son, Jan Brueghel the Elder Sr. In that prolific generation of artists, artists copied each other and even their own works, so that it was a nightmare to find the ownership of the paintings.

Elizabeth Honig, an art historian at the University of California, Berkeley, focuses on these complex questions to reveal what works were created by a painter in northern Renaissance art and who he influenced. Now, she begins to ask for the help of the computer.

Honig has a database of more than 1500 digital copies of Bruegel paintings, most of which are owned by Jan. In 2016, she launched an unusual collaboration with artificial intelligence researchers in France and the United States, deploying a state-of-the-art computer vision system to track these works one by one and analyze the similarities between them. Other art historians also see the opportunity to use machine learning to provide empirical support for theories and viewpoints that were previously limited to the viewer's subjective perspective.

Honig says computers can "get more details more easily". Take the windmill as an example: her Breughel database contains hundreds of paintings containing this element. The algorithm obtains the same structural image from multiple paintings, even in the case of flipping. It can also pinpoint copies of lions, dogs and other characters. Many Renaissance artists' studios are joint creative spaces, so computer technology can also help Honig understand how different artists work together (whether they belong to the same family or not). Honig said, "Peter Paul Rubens drew some characters, then old Jan Bruegel drew horses, dogs and lions, and finally combined them." "

Based on historical records and close observations of the works, many art historians speculate that many of the paintings of Peter Bruegel Jr. And Jan Bruegel Sr. were done in this way. And computers can help prove it. "it solves a lot of problems about the creation process," Honig said. "

Computer scientists also have their own considerations in this project. For them, Honig's collection is a perfect dataset to extend the algorithm. Mathieu Aubry, an expert on computer vision and deep learning at É coledesPonts ParisTech, a French national school of roads and bridges, said that using algorithms to process paintings challenged the program's ability to match patterns. The difficulty depends on the difference in media and color. "without training, computer vision cannot recognize that the sketch is the same as the house in the oil painting," he explained. "the lines of the sketch are clear, while the edges of the oil painting are blurred, which may confuse the algorithm.

It takes a lot of time to label the same objects or teach computers to find specific similarities such as shapes. Therefore, Aubry and his colleagues use unsupervised deep learning techniques to make the algorithm look for similarities after inputting images. He believes that the results can also be used in practical applications related to AI vision, such as self-driving cars.

In March, his team published a study (https://arxiv.org/abs/1903.02678) on arXiv, which was accepted by CVPR 2019. Aubry says that while unsupervised deep learning usually requires a lot of arithmetic, it is immune from human preconceptions. Therefore, it can avoid prejudice, such as focusing only on the main features of the image.

Computer distinguishing trend

Rutgers University in New Jersey (Piscataway Campus) is using similar techniques to depict how the styles of different artists are defined and developed over time. Marian Mazzone, an art historian and a member of the Rutgers Laboratory of Art and artificial Intelligence, said: "We have some unprovable theories, and computer science may help me get empirical answers to these questions. She collaborated with lab director Ahmed Elgammal to conduct a digital analysis of 77000 works of art from the Renaissance to pop art over the five centuries (see https://arxiv.org/abs/1801.07729 for related research). To the team's surprise, the use of unsupervised learning by computers can put these works of art in chronological order. The project confirms the theory of Heinrich Wolffin, a famous art historian of the 20th century. He believes that the transformation of artistic style can be analyzed and classified according to five dual characteristics. One of the works is whether it is "line drawing" (outline-dominated, such as Sandro Botticelli's) or "drawing" (relying more on strokes that depict light and shadow, such as Tintoretto's paintings). Elgammal, director of the Rutgers Art and artificial Intelligence Laboratory, believes that for the first time, AI has made art history a predictive science that compares theory with observation.

Elsewhere, AI is being used to solve a long-standing problem with the material heritage of art history: damage. For example, Verus Art from Arius Technology, a Vancouver start-up in Canada, is deploying a 3D scanning printing system that can accurately reproduce works of art, even texture strokes and pigment tones. The system was originally designed to study the damage to Leonardo da Vinci's Mona Lisa. For education, outreach and archiving, "backup" paintings may have another use: to thwart thieves who are more discerning than thieves who can't see copies.

Paper: Discovering Visual Patterns in Art Collections with Spatially-consistent Feature Learning

Paper link: https://arxiv.org/abs/1903.02678

Abstract: the purpose of this study is to find almost repetitive pattern patterns from a large number of works of art. Due to the differences of art media (oil painting, pink painting, sketch, etc.) and the inherent deviation in the process of reproduction, this goal is more difficult than standard case mining. The key technology is to fine-tune the standard depth features in a specific art collection to adapt to this task through the use of self-supervised learning. specifically, the spatial consistency between adjacent feature matches is used as a supervised fine-tuning signal. The adjusted features can be matched more accurately (independent of style differences) and can be used in conjunction with standard pattern discovery methods based on geometric validation to identify duplicate patterns in the dataset. This method is evaluated on several different data sets and shows very good qualitative findings. In terms of quantitative evaluation, the researchers marked 273 approximately repetitive details in the data set of 1587 works of art from Jan Bruegel and his studio. In addition to works of art, the researchers also demonstrated improvements in the location of Oxford5K photo datasets and historical photos on LTLL (Large Time Lags Location) datasets.

Figure 1: an example of repetitive visual patterns automatically discovered by the algorithm.

Figure 2: feature learning strategy of this method.

Figure 5: an example of the detection of training features on the Bruegel data set.

Reference link:

Https://www.nature.com/articles/d41586-019-01794-3

Https://www.toutiao.com/a6708541485484081676/

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.