Learning to Infer Graphics Programs from Hand-Drawn Images

© 2018 Curran Associates Inc.All rights reserved. We introduce a model that learns to convert simple hand drawings into graphics programs written in a subset of LAT E X. The model combines techniques from deep learning and program synthesis. We learn a convolutional neural network that proposes plau...

Full description

Bibliographic Details
Main Authors: Ellis, Kevin (Author), Ritchie, Daniel (Author), Solar-Lezama, Armando (Author), Tenenbaum, Joshua B. (Author)
Other Authors: Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory (Contributor)
Format: Article
Language:English
Published: 2021-11-08T21:02:24Z.
Subjects:
Online Access:Get fulltext
Description
Summary:© 2018 Curran Associates Inc.All rights reserved. We introduce a model that learns to convert simple hand drawings into graphics programs written in a subset of LAT E X. The model combines techniques from deep learning and program synthesis. We learn a convolutional neural network that proposes plausible drawing primitives that explain an image. These drawing primitives are a specification (spec) of what the graphics program needs to draw. We learn a model that uses program synthesis techniques to recover a graphics program from that spec. These programs have constructs like variable bindings, iterative loops, or simple kinds of conditionals. With a graphics program in hand, we can correct errors made by the deep network and extrapolate drawings.