Unsupervised Digital Movie Restoration with Spatial Models of Colors

Since its invention in the nineteenth century, cinema has become one of the most important media of popular culture, becoming in fact, next to the books, our historical memory.

Unfortunately, color films are subject to the aging of dyes contained in the emulsion and of the support itself. The decay is a natural process that usually introduces a color dominant, loss of contrast and/or color desaturation. Color negative and positive film, color print material, interpositives, and color motion-picture release print are all affected in the same way. The decaying process is irreversible, and the most effective method to slow it is to store photographic and cinematographic material at controlled temperature and humidity.

It has been estimated that about 80% of the movies produced between 1910 and 1920 have been lost. If not restored, a large part of the film cultural heritage will be lost relatively quickly. Hence the strong ethical obligation to preserve and restore those moving images that represent our social memory.

Thanks to the relevant technical progresses of the last years, the digital restoration of color has widely spreaded. Digital techniques allow an easier removal of color casts, equalization of the frame histogram, expansion of the dynamic range and adjustment of desaturated colors. Moreover, it can address artifacts not treatable by traditional photochemical techniques. Finally, digital techniques allow easily multiple attempts, however, constant supervision by qualified personnel is still required. Moreover, a digital pipeline can address effectively the scaling of the restored content to the different formats and gamuts, necessary for the multiple different visualization devices currently available in cinemas.

Current state of the art in digital restoration is mainly based on color grading techniques: "primary" grading affects the levels of RGB, gamma, shadows and highlights in the entire frame, while "secondary" techniques apply changes only on a selected range of colors or windows in the frame.

We have proposed an alternative approach to the standard digital color film restoration, based on the idea of recovering the appearance of color rather than the original color signal, by means of the application of Spatial Color Algorithms (SCAs), a family of algorithms inspired by the capabilities of the Human Vision System (HVS) of autonomously adjust to the variation of color and lightness in the scene.

A digital restoration pipeline based on the application of SCAs can be divided in these steps:

  • Shot detection
  • Key Frames extractions
  • SCAs parameters tuning on keyframes
  • Unsupervised filtering of each shots
  • Final remounting and armonization stage

SCAs are widely used for general-purpose enhancement of digital images. However, in digital movie restoration, there are cases in which the editing of artistic choices of the director may have introduced some characteristis in the frames (like, e.g., fade-in and fade-out, a color dominant, or a reduced dynamic range) that should not be removed by the application of the SCA. Some options are needed to prevent the algorithm to change specific artistic choices of the director:

  • Keep Original Gray (KOG) option centers the chromatic channels around the middle gray, with this option the original mean values (independently in R, G and B channels) are preserved. The result is a histogram more similar in shape with the original one. This function is fundamental to keep fade-in and fade-out sequences.
  • The color dominant present in a sequence is not always due to the deterioration of the film, but maybe it is introduced for artistic purpose. With the Keep Original color Cast (KOC) option the color dominant is kept and only contrast and dynamic are modified.
  • Sometimes directors do not use the whole available dynamic range of the film in order to obtain specific visual effects and low or high key pictures. In these cases, the use of KODR (Keep Original Dynamic Range) respects the original intention of the director. This function can also be used to manage frames that are excessively corrupted. 

Demo reel of digital movie restoration project 

Some examples of digital movie restoration using ACE for film