One of the greatest challenges of art restoration is that once an art conservator has decided to use a particular set of colours or techniques, then the work can’t easily be undone. It is not usually possible to experiment with different theories about how an original piece might have looked and there is an ever-present risk of accidental damage during the restoration.
Applied mathematicians, art conservators and art historians from the University of Cambridge have helped overcome this problem by developing pioneering techniques which allow a piece to be digitally restored. This is especially important in the case of illuminated manuscripts which can’t be physically altered, for example by applying new ink or repairing holes. These manuscripts are deemed to be simply too fragile for physical restoration meaning that digital restoration offers the only hope of viewing them like they might have looked originally.
In-painting
Many old manuscripts and paintings are damaged so badly that they have been torn or have actual holes in them. The aim of the art restorer is to fill in the holes in a way that is faithful to the original, a technique which is known as ‘in- painting’. Researchers at the Cambridge Image Analysis Group which is part of the Department of Applied Mathematics and Theoretical Physics (DAMPT) have developed ways of digitally in- painting artwork using mathematical techniques pioneered by mathematicians around the world in the last twenty years or so.
Breaking images down
Every image can be broken down into pixels, which are tiny blocks of solid colour – this is easy to see if you zoom in too far on a digital photograph. If you have a black and white photograph then each pixel has a number between 0 and 255 associated with it, indicating where it is on the spectrum between black and white. If the small block of colour has a value of 0 then it will be black, whereas 255 means white – a number of 200 would be a light grey.
In a colour photograph each pixel has three numbers associated with it, indicating the constituent amount of red, green and blue (r,g,b). For example a pixel with a value of (255,0,0) would be a solid block of red. Breaking images down into pixels means that every image is essentially a collection of numbers, meaning that mathematics is the perfect tool for analysing images.
Closing the gaps
When art conservators are presented with a hole in a painting they look at what is around the hole and then try to mimic what they see, using their judgment to fill in the gaps. The mathematicians aim to replicate this process digitally, without ever having to touch the image itself. First of all they determine where the boundary of the hole is. Sometimes the boundary will be drawn on manually, however the latest research means that sometimes this can be done automatically.
Once the boundary of the hole is known there are various ways that mathematicians in-paint the hole. One of the most common approaches is a two-step process, first applying a combination of diffusion and transport to govern how the colours on the boundary propagate into the hole. On its own this wouldn’t recreate the ‘texture’ of the painting, for example the texture of an animal’s fur. That is why, after adding colour into the hole, mathematicians then perform an intelligent ‘cut and paste’ process which fills in any textures guided by the already in-painted colour.
In-painting in reverse
Usually the process of in-painting involves filling in gaps as described above, however in a recent project it was used to do completely the opposite.
The Fitzwilliam Museum in Cambridge is in possession of a precious 16th Century manuscript called the Primer of Claude of France which depicts a naked Adam and Eve wearing coverings. Recent infrared scans of the manuscript revealed the fact that the coverings hadn’t always been there, they had been painted on by a subsequent owner. The mathematicians were able to use their digital in-painting techniques to remove these additions from the painting, giving a view of what the manuscript would have looked like originally.
They did this by treating the additions that had been added later like holes that need to be in- painted – propagating the colour from around the edges. The results of the digital restoration were so successful that these were displayed alongside the original manuscript at an exhibition which celebrated 200 years of the Fitzwilliam Museum. http://www.fitzmuseum.cam.ac.uk/colour.
Other applications
In-painting has a number of other useful applications beyond the world of art restoration, for example in the analysis of fingerprints. If an incomplete fingerprint is gathered from a crime scene then in-painting techniques can fill in the gaps and help match the fingerprint to a crime database. It also has important uses in medical imaging where it can be used to improve the quality of scans which usually suffer from degradation due to noise and gaps in the acquired measurements. In the modern world images are everywhere, meaning that one mathematical technique can make a big impact in areas as diverse as medicine, crime detection and art restoration.
Technical Supplement
One of the most foundational mathematical equations in image processing is the Heat Equation. It is a so-called partial differential equation and was invented by a 19th Century French mathematician called Joseph Fourier who first used it to describe how the concentration of heat changes in a solid object over time. In image in-painting the Heat Equation can be used to propagate the colour from around the boundary of the hole into the hole.
Fortunately the Heat Equation can be used to describe lots of different situations, as it can model how the concentrations of different quantities change with respect to time. This is why it is sometimes called the Diffusion Equation, as it can be used in a variety of situations where there is some kind of diffusion going on.
In the case of in-painting it can be used to describe how concentrations of colour change. At the beginning of the process there are various concentrations of colour around the boundary of the hole. The Diffusion Equation describes what will happen as the colours diffuse into the hole, just as heat would diffuse into a room from a heat source. Eventually the colours will have diffused into the hole and we will have restored the main colour constituencies in the hole as suggested by the colour distribution on the boundary.
In reality the process is more complicated. For example, a visually pleasing in-painting result will very rarely be based on diffusion only as this process usually smears the edges between different objects in the image, and diffusion alone will not be able to restore non-local structures such as texture. Mathematical researchers have now proposed more advanced mathematical techniques for in-painting, still using the heat equation as their inspiration. One aspect of this is to replace the linear diffusion of the Heat Equation with a non-linear diffusion. This weights the amount of diffusion with the size of the colour gradient, and combines it with transport mechanisms, diffusing more when the colour gradient is small and less when it is large, thereby preventing the smearing of edges.
For reproducing repetitive structures in the hole, such as textures, intelligent copy-and-paste algorithms have been proposed that, guided by example patches in the intact part of the image, fill in missing textures by copying in texture- patches from the outside of the hole (from the intact part of the image or from a whole image library of similar artwork).
Links
View the manuscripts: www.fitzmuseum.cam.ac.uk/illuminated
Experts
Dr Carola-Bibiane Schönlieb, Head of the Cambridge Image Analysis (CIA) group at the Department of Applied Mathematics and Theoretical Physics (DAMTP), University of Cambridge.
Marie D’Autume, ENS Cachan, Paris.
Dr Stella Panayotova, Keeper of Manuscripts, Fitzwilliam Museum, Cambridge.
Dr Paola Ricciardi, Fitzwilliam Museum’s Research Scientist, Cambridge.
Dr Spike Bucklow, Senior Research Scientist, Hamilton-Kerr Institute, University of Cambridge.
Note
The IMA would like to thank Dr Carola-Bibiane Schönlieb, Marie D’Autume, Dr Stella Panayotova, Dr Paola Ricciardi and Dr Spike Bucklow, for their help in the preparation of this document.
Filling in the Gaps The Hidden Role of Mathematics in Art Restoration (pdf)
