Occupancy Networks: Learning 3D Reconstruction in Function Space, Mescheder, Oechsle, Niemeyer, Nowozin, Geiger; 2018 - Summary
author: ishanashah
score: 10 / 10

Problem: Need a good representation of 3D data for deep learning.

Unlike images, there’s no canonical representation for 3D objects.



Reason about occupancy not only at fixed discrete 3D locations, but at any possible 3D point.

Learn a nonlinear binary classifier that can determine whether any given 3D point is inside or outside the object.

The function is called an occupancy network, because it predicts the probability that a point is occupied.

Using an occupancy network results in less space complexity, because you only need to store the network instead of every 3D point.

The surface of points where the probability of occupancy is 0.5 is the surface of the object.

For 3D reconstruction, the occupancy network takes a pair of the input image and 3D location as inputs.

Training and Inference

Implemented using a fully-connected neural network with 5 ResNet blocks.

Occupancy network is trained on a uniform random sample of points with known labels.

Use Multiresolution IsoSurface Extraction (MISE) to extract surface from occupancy network.



Evaluate on ShapeNet dataset against voxel-based, point-based, and mesh-based approaches.

Representation Power:


Condition Occupancy Networks on Noisy Point Clouds:


Condition Occupancy Networks on Low-Resolution Voxels (3D Super-resolution):


Found that uniform random sampling performs significantly better than sampling the same number of points inside and outside the mesh (equal sampling) or sampling half the points uniformly and half the points near the surface of the mesh (surface sampling).