MIT and Microsoft develops algorithm to spot links between artworks

Developed by a team at MIT and Microsoft, a new AI algorithm can identify subtle visual similarities between artworks created in vastly different media, cultures, and time periods. 

MosAIc algorithm was designed as a tool to help curators streamline the process of comparing artworks in large collections. Researchers based their model on the Rijksmuseum in Amsterdam and the Metropolitan Museum of Art in New York. 

Mark Hamilton, Lead Researcher for MosAIc and PhD student, remarked “every time I use the algorithm, I find surprises.” Unlike other image comparison systems, MosAIc looks beyond colour and style comparisons to examine shared meanings and themes. 

Restricting an image retrieval system to particular subsets of images can yield new insights into relationships in the visual world,” explained Hamilton. “We aim to encourage a new level of engagement with creative artifacts.” 

Hamilton was fascinated when the algorithm found thematic links between a Delft violin and an 18th-century banyan, which is a Dutch garment inspired by Japanese kimonos. The algorithm picked up that both are strongly coloured with blue and white, “but also hints at a more fundamental connection between these two works,” Hamilton said. 

More specifically, these works are evidence of the shared influence of Dutch-Chinese porcelain trade in the 16th to 18th centuries. This trade pipeline ignited Europe’s love of blue and white and cobalt-blue glazed porcelain from Jingdezhen.” 

The algorithm utilises machine learning and an image-retrieval system. To find visual connections between different cultures, researchers applied a new image-search data structure called a ‘conditional KNN tree’. This technology groups together similar images in a large tree-like structure.  

Although some have argued that MosAIc could rival the role of curators, Hamilton assured “this software could help curate an exhibition, but it’s not aiming to replace curators.” 

In recent years, the art world has increasingly turned to technology to propel research and provide new, exciting experiences. “Going forward, we hope this work inspires others to think about how tools from information retrieval can help other fields like the arts, humanities, social science, and medicine,” declared Hamilton.  

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s