Data-Driven Scene Understanding from 3D Models

Input Image
Predicted Orientations
Predicted Depths
Predicted Objects


Scott Satkin
Jason Lin
Martial Hebert


In this paper, we propose a data-driven approach to leverage repositories of 3D models for scene understanding. Our ability to relate what we see in an image to a large collection of 3D models allows us to transfer information from these models, creating a rich understanding of the scene. We develop a framework for auto-calibrating a camera, rendering 3D models from the viewpoint an image was taken, and computing a similarity measure between each 3D model and an input image. We demonstrate this data-driven approach in the context of geometry estimation and show the ability to find the identities and poses of object in a scene. Additionally, we present a new dataset with annotated scene geometry. This data allows us to measure the performance of our algorithm in 3D, rather than in the image plane.


Data-Driven Scene Understanding from 3D Models,
S. Satkin, J. Lin and M. Hebert,
Proceedings of the 23rd British Machine Vision Conference (BMVC),
September 2012.
[PDF] [BibTeX]
    author={Scott Satkin and Jason Lin and Martial Hebert},
    title={Data-Driven Scene Understanding from 3{D} Models},
    booktitle={Proceedings of the 23rd
               British Machine Vision Conference},

Scott Satkin, September 2012.

CMU 3D-Annotated Scene Database

Creative Commons License This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License.

ASCII files - txt.tar.gz (158MB)
Raw SketchUp files - skp.tar.gz (580MB)
Surface normal & object label renderings - renderings.tar.gz (125MB)
Dataset description- README.txt
Please contact the authors with any additional requests for data.


This material is based upon work partially supported by the Office of Naval Research under MURI Grant N000141010934.

Copyright notice