Deep multispectral painting reproduction via multi-layer, custom-ink printing

Liang Shi, Vahid Babaei, Changil Kim, Michael Foshey, Yuanming Hu, Pitchaya Sitthi-Amorn, Szymon Rusinkiewicz, Wojciech Matusik

Research output: Contribution to journalArticle

Abstract

We propose a workflow for spectral reproduction of paintings, which captures a painting's spectral color, invariant to illumination, and reproduces it using multi-material 3D printing. We take advantage of the current 3D printers' capabilities of combining highly concentrated inks with a large number of layers, to expand the spectral gamut of a set of inks. We use a data-driven method to both predict the spectrum of a printed ink stack and optimize for the stack layout that best matches a target spectrum. This bidirectional mapping is modeled using a pair of neural networks, which loss function helps find the best possible ink layout resulting in the balance between spectral reproduction and colorimetric accuracy under a multitude of illuminants. In addition, we introduce a novel spectral vector error diffusion algorithm based on combining color contoning and halftoning, which simultaneously solves the layout discretization and color quantization problems, accurately and efficiently. Our workflow outperforms the state-of-the-art models for spectral prediction and layout optimization. We demonstrate reproduction of a number of real paintings and historically important pigments using our prototype implementation that uses 10 custom inks with varying spectra and a resin-based 3D printer.

Original languageEnglish (US)
Article number271
JournalACM Transactions on Graphics
Volume37
Issue number6
DOIs
StatePublished - Nov 2018

All Science Journal Classification (ASJC) codes

  • Computer Graphics and Computer-Aided Design

Keywords

  • 3d printing
  • Multi-spectral imaging
  • Spectral reproduction

Fingerprint Dive into the research topics of 'Deep multispectral painting reproduction via multi-layer, custom-ink printing'. Together they form a unique fingerprint.

  • Cite this

    Shi, L., Babaei, V., Kim, C., Foshey, M., Hu, Y., Sitthi-Amorn, P., Rusinkiewicz, S., & Matusik, W. (2018). Deep multispectral painting reproduction via multi-layer, custom-ink printing. ACM Transactions on Graphics, 37(6), [271]. https://doi.org/10.1145/3272127.3275057