Here’s How AI Helped Recover Lost Paintings by Gustav Klimt

In May 1945, the retreating Nazi forces burnt down an Austrian castle that contained three of the most daring and complex works by Gustav Klimt. The triptych, popularly known as the Faculty Paintings, was a scandalous commission made by the University of Vienna. The work was supposed to represent the three disciplines of jurisprudence, medicine, and philosophy. However, the paintings strived too far from the desired ideal of the University. Instead of praising humanity’s achievements, Klimt highlighted its flaws, which resulted in the University’s refusal to accept the commission.

In 1894, the famous Austrian painter Gustav Klimt received a commission from the University of Vienna to design three decorative panels for their main building. Klimt painted three compositions from 1899 to 1907, each one serving as an allegory of a discipline taught at the University. Philosophy was represented through a group of nude figures showing life and decay, and a disembodied collective mind in a mist-like mass. Medicine showed the natural life cycle, with life and death co-existing regardless of the involvement of science and medicine. Jurisprudence was equally unoptimistic, showing a man mid-execution, strangled by a sea monster, with apathetic allegorical figures of Truth, Law, and Justice in the background.

Klimt was already an established and respected artist in Vienna, revered by local art collectors. However, the public reaction to his works was not as complimentary. Upon revealing Klimt’s work, the university officials accused Klimt of painting pornography and endangering public morals. His allegorical thinking was also hardy welcome. The commissioners were outraged by Klimt’s decision to illustrate the limitations of each discipline rather than human achievement. For Klimt, human knowledge had enough flaws and needed humbling in the face of larger forces of life and time.

However, the most debatable aspect of the Faculty Paintings’ existence is their supposed demise. According to widely accepted belief, the triptych was destroyed in a fire that happened on May 8th, 1945. At that time, the paintings, confiscated from their Jewish owners’ families, were moved by the Nazis to the Immendorf castle in Lower Austria. The Third Reich authorities recognized the importance and the quality of Klimt’s art, interpreting it in line with their ideology of the supremacy of German culture. This interpretation saved many works of Klimt from destruction but, unfortunately, directly led to the loss of his most important masterpieces. On the last day of the war, the retreating German tank division set fire to the castle.

Some art experts believe that the Faculty Paintings might have survived and that they were evacuated by some of the retreating officers. However, most notice that given the scale of the scale of the paintings, their evacuation would have required too much time and effort to remain unnoticed. All remaining evidence is limited to several black-and-white photographs, a series of preparatory sketches by Klimt, and a collection of journalistic publications about the paintings. The information given by these sources is scarce and insufficient, and cannot possibly give us a complete understanding of what the paintings looked like.

In 2021, Google collaborated with the Leopold Museum in Vienna to imagine the way the Faculty Paintings looked before May 1945 and re-colorize them using AI. Google expert Emil Wallner spent six months working on an algorithm that could reconstruct Klimt’s way of thinking, consulting with a Klimt expert Franz Smola. First, Wallner fed 100,000 different artwork images to the algorithm so it could understand the concept of an art object, painting, and visual culture. During the next step, the machine had to learn more about Klimt specifically,  processing 80 works by the artist. The third and final step was to incorporate the non-visual data related to the Faculty Paintings—visual descriptions from the press, Klimt’s notes and letter excerpts, and evidence from those who have seen the works.

The controversial reputation of the series helped the research process. The public outrage that happened when the works were first revealed was followed by dozens of newspaper publications, both praising and ridiculing the work. From one of these critical texts, Wallner and Smola learned that Klimt painted the sky not with expected tones of blue but with a greenish hue. Many aesthetic choices of Gustav Klimt seemed counterintuitive and developed from his unique approach to art.

The case of digital restoration of the lost Gustav Klimt paintings is far from the first or the only occasion of art experts implementing AI in their research. Although such an approach receives much-deserved criticism, it has also contributed to discussions about the authenticity of art and the nature of artistic intent. One of the most famous AI projects was launched by the team of Rijksmuseum in Amsterdam. While researching the world-famous painting Nightwatch by Rembrandt van Rijn, the restoration team learned that the original painting was significantly larger. Its owners cut off large chunks from its sides to fit it into a new location. The missing parts were never found or identified. The conservation team currently uses AI to try and reconstruct the lost elements according to lines and shapes preserved on the original canvas.

In 2022, Dutch art experts digitally restored a lost painting of wrestlers by Vincent van Gogh. He first painted it in 1886, however, he later used the same canvas to create another work, Still-life with meadow flowers and roses. Based on the lines visible under the X-ray, researchers were able to see the composition of the first painting. Then, using an AI algorithm trained on other Van Gogh paintings, they reconstructed the tones and brushwork of the original piece. Although the result is undoubtedly impressive, we cannot truly rely on it, since it is more of a suggestion than a certain answer.

Despite the initial claims that it will revolutionize art history, Google’s Klimt project gathered much more criticism than praise. Many experts and viewers believe that the restored versions were too primitive to be accurate, given Klimt’s thorough and complex work with color. Despite its familiarity with Klimt’s oeuvre, the algorithm chose a surprisingly flat color scheme, lacking nuance. Moreover, some specific hues had no analogs in the artist’s earlier paintings. We cannot be sure, of course, that Klimt avoided radically new steps in his special series. However, this would be an unforgivable assumption for art historical research. Despite the technological advancements, Klimt experts had to intervene to fix the color palette. The earlier version of AI-colorized works was painted with neon colors which were non-existent in the artist’s time.

Critics also have noticed the dramatic difference between the amount of artworks used to train the algorithm. With 100,000 works by other artists, the machine received only 80 works by Klimt. Moreover, the overwhelming majority of art experts believe that the application of AI to the discipline of art history is confusing if not harmful, at least given the current state of technology. While it can make educated guesses, its way of thinking is radically different from that of humans. The machine is incapable of artistic intent and tone-deaf to many emotional aspects of human existence, which makes it unsuitable for research related to art.