The first ECCV 2020 Workshop on
Learning 3D Representations for Shape and Appearance

Abstract: We propose a generative model of 2D and 3D natural textures with diversity, visual fidelity and at high computational efficiency. This is enabled by a family of methods that extend ideas from classic stochastic procedural texturing (Perlin noise) to learned, deep, non-linearities. The key idea is a hard-coded, tunable and differentiable step that feeds multiple transformed random 2D or 3D fields into an MLP that can be sampled over infinite domains. Our model encodes all exemplars from a diverse set of textures without a need to be re-trained for each exemplar. Applications include texture interpolation, and learning 3D textures from 2D exemplars.

Bio: Philipp Henzler is a third year PhD student at University College London under the supervision of Tobias Ritschel and Niloy J. Mitra. His research focuses on explaining our world from 2D visual observations. More specifically, weakly supervised 3D texture synthesis and 3D reconstruction from 2D images.

Abstract: Inspired by classical Binary Space Partitioning (BSP) data structures from computer graphics, we introduce a network architecture that learns to represent a 3D shape via convex decomposition. The network is trained to reconstruct a shape using a set of convexes obtained from a BSP-tree built on a set of planes. The convexes inferred by BSP-Net can be easily extracted to form a polygon mesh, without any need for iso-surfacing. The generated meshes are compact (i.e., low-poly) and well suited to represent sharp geometry; they are guaranteed to be watertight and can be easily parameterized. We also show that the reconstruction quality by BSP-Net is competitive with state-of-the-art methods while using much fewer primitives.

Bio: Zhiqin Chen is a first-year Ph.D. student at Simon Fraser University, under the supervision of Prof. Hao (Richard) Zhang. His research interest is computer graphics with specialty in geometric modeling and machine learning.

Abstract: In this talk, I cover the results of three recent papers which leverage the smooth inductive biases of ReLU networks to reconstruct surfaces from point clouds: “Deep Geometric Prior for Surface Reconstruction” constructs a manifold atlas consisting of parametric patches encoded as a fully connected ReLU network. This work empirically demonstrates that the inductive bias of ReLU networks leads to good solutions on surface reconstruction outperforming other state of the art methods. “Gradient Dyanics of Shallow Univariate Networks” formally analyzes this inductive bias in the case of curves, and shows that under some initializations, shallow ReLU networks behave as kernel machines, minimize curvature, and are equivalent to cubic splines. Inspired by these theoretical results, “Neural Splines: Fitting 3D Surfaces with infinitely wide neural networks” uses the kernels arising from shallow ReLU networks to represent an implicit function. This simple linear model outperforms state of the art traditional methods and neural network based methods, and suggests that the success of neural implicit methods may be attributed to the kernel behavior of neural networks.

Bio: Francis Williams is a fourth year PhD student at New York University advised by Denis Zorin and Joan Bruna. Francis’s work lies at the intersection of machine learning, computer vision, and computer graphics. Recently, Francis has been focusing on understanding and leveraging the inductive biases of neural networks to solve geometric problems in a principled manner. In addition to research, Francis actively maintains several open source libraries including Point Cloud Utils, NumpyEigen and FML.

Abstract: In this work we address the challenging problem of multiview 3D surface reconstruction. We introduce a neural network architecture that simultaneously learns the unknown geometry, camera parameters, and a neural renderer that approximates the light reflected from the surface towards the camera. The geometry is represented as a zero level-set of a neural network, while the neural renderer, derived from the rendering equation, is capable of (implicitly) modeling a wide set of lighting conditions and materials. We trained our network on real world 2D images of objects with different material properties, lighting conditions, and noisy camera initializations from the DTU MVS dataset. We found our model to produce state of the art 3D surface reconstructions with high fidelity, resolution and detail."

Bio: Ben is final year PhD student working with Ren Ng in the EECS department at UC Berkeley. He works on problems in computer vision and graphics, and previously interned in Marc Levoy's group in Google Research as well as with Rodrigo Ortiz-Cayon and Abhishek Kar at Fyusion. Ben did his undergrad at Stanford University and worked at Pixar Research in the summer of 2014.

Abstract: Unsupervised learning with generative models has the potential of discovering rich representations of 3D scenes. While geometric deep learning has explored 3D structure-aware representations of scene geometry, these models typically require explicit 3D supervision. Emerging neural scene representations can be trained only with posed 2D images, but existing methods ignore the 3D structure of scenes. We propose Scene Representation Networks (SRNs), a continuous, 3D-structure-aware scene representation that encodes both geometry and appearance. SRNs represent scenes as continuous functions that map world coordinates to feature representations of local scene properties. By formulating the image formation as differentiable ray-marching, SRNs can be trained end-to-end from only 2D images and their camera poses, without access to depth / shape. This formulation naturally generalizes across scenes, learning powerful geometry and appearance priors in the process. This enables novel view synthesis, few-shot reconstruction, joint shape and appearance interpolation, and unsupervised discovery of a non-rigid face model.

Bio: Vincent has just finished his PhD at Stanford University and is now a Postdoc at MIT's CSAIL with Josh Tenenbaum, Bill Freeman, and Fredo Durand. His research interest lies in neural scene representations - the way neural networks learn to represent information on our world. His goal is to allow independent agents to reason about our world given visual observations, such as inferring a complete model of a scene with information on geometry, material, lighting etc. from only few observations, a task that is simple for humans, but currently impossible for AI.

Abstract: We introduce a method for learning to generate the surface of 3D shapes. Our approach represents a 3D shape as a collection of parametric surface elements and, in contrast to methods generating voxel grids or point clouds, naturally infers a surface representation of the shape. Beyond its novelty, our new shape generation framework, AtlasNet, comes with significant advantages, such as improved precision and generalization capabilities, and the possibility to generate a shape of arbitrary resolution without memory issues. We demonstrate these benefits and compare to strong baselines on the ShapeNet benchmark for two applications: (i) auto-encoding shapes, and (ii) single-view reconstruction from a still image. We also provide results showing its potential for other applications, such as morphing, parametrization, super-resolution, matching, and co-segmentation.

Bio: Thibault Groueix worked in the Imagine group of Ecole des Ponts ParisTech under the supervision of Mathieu Aubry. He worked in close collaboration with Adobe research, co-supervised by Mathew Fisher, Bryan Russel abd Vova Kim. He will join Naver Labs as a research scientist in September 2020. He introduced a novel method to parameterize 3D data (AtlasNet). Using this new parameterization for 3D shape synthesis, he developed analysis-by-synthesis methods to learn 3D shape correspondences (3D-CODED), 3D parts discovery (AtlasNetv2), and clustering (DTI-Clustering).

Abstract: Computer graphics, 3D computer vision and robotics communities have produced multiple approaches to rep- resenting 3D geometry for rendering and reconstruction. These provide trade-offs across fidelity, efficiency and com- pression capabilities. In this work, we introduce DeepSDF, a learned continuous Signed Distance Function (SDF) rep- resentation of a class of shapes that enables high qual- ity shape representation, interpolation and completion from partial and noisy 3D input data. DeepSDF, like its clas- sical counterpart, represents a shape’s surface by a con- tinuous volumetric field: the magnitude of a point in the field represents the distance to the surface boundary and the sign indicates whether the region is inside (-) or outside (+) of the shape, hence our representation implicitly encodes a shape’s boundary as the zero-level-set of the learned func- tion while explicitly representing the classification of space as being part of the shapes’ interior or not. While classical SDF’s both in analytical or discretized voxel form typically represent the surface of a single shape, DeepSDF can repre- sent an entire class of shapes. Furthermore, we show state- of-the-art performance for learned 3D shape representation and completion while reducing the model size by an order of magnitude compared with previous work.

Bio: I'm a PhD student at the University of Washington CSE, where I work with Steve Seitz. I'm broadly interested in computer vision and graphics. My current research focus is on 3D reconstruction and realistic rendering using physics and neural representations. I am fortunate to work with awesome collaborators including Richard Newcombe and Qi Shan, labmates at GRAIL, and mentors and friends I met during my internship at Apple, Oculus, and Adobe. Prior to PhD, I received B.S. from Caltech, working at Pietro Perona's vision lab.