Segmentation-Based Parametric Painting

Manuel Ladron de Guevara1, Matthew Fisher2, Aaron Hertzmann2,
1Carnegie Mellon University, 2Adobe Research
Paper arXiv Code Supplemental
testing

Our method turns images into paintings of different styles. Abstract expressionist (left), painterly (right). We achieve a higher control than existing methods through the use of segmentation and dynamic attention maps.

Abstract

We introduce a novel image-to-painting method that facilitates the creation of large-scale, high-fidelity paintings with human-like quality and stylistic variation. To process large images and gain control over the painting process, we introduce a segmentation-based painting process and a dynamic attention map approach inspired by human painting strategies, allowing optimization of brush strokes to proceed in batches over different image regions, thereby capturing both large-scale structure and fine details, while also allowing stylistic control over detail. Our optimized batch processing and patch-based loss framework enable efficient handling of large canvases, ensuring our painted outputs are both aesthetically compelling and functionally superior as compared to previous methods, as confirmed by rigorous evaluations.

How does it work?

Our goal is to efficiently optimize a collection of stroke parameters that define a large-scale painting from a given image, capturing micro and macro structures with high fidelity, and providing control for stylistic variation. Our system takes as input an image \(X\), and style parameters, encapsulated by three style presets: "realistic," "painterly," and "abstraction." Then, our objective is to optimize a parameter vector \(\bar{Y}\) of strokes, which, when rendered on a canvas \(C\), will approximate the given image \(X\) as a stylized painting. Our scalable solution accommodates varying sizes and aspect ratios through batch-patch optimization, cropping the input and canvas into a set of \(128 \times 128\) patches, which are reassembled after the optimization process to match the original image scale.

Inspired by human painting techniques, and to provide more control over the painting process, we first use a segmentation network that decomposes the painting into semantic areas, and then employ dynamic attention maps to determine where to paint.

Our innovative painting method uses a layered approach to create art. Each layer corresponds to different parts of the image, allowing for unique styles and details in each area.

We start with broad strokes and gradually add finer details, mimicking the process of a traditional artist. This step-by-step approach helps us craft a painting that combines both large-scale structure and intricate elements.

The essence of our technique lies in focusing on specific areas of the painting in each step. We use dynamic attention maps to guide this process, which either uniformly covers the whole image or focuses on areas needing more work. This strategy ensures that every part of the painting gets the right amount of detail, making our paintings strikingly realistic and artistically expressive.

BibTeX

@misc{deguevara2023segmentationbased,
      title={Segmentation-Based Parametric Painting}, 
      author={Manuel Ladron de Guevara and Matthew Fisher and Aaron Hertzmann},
      year={2023},
      eprint={2311.14271},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      }