EllipssianNet: Image-guided Sampling of 2D Gaussians for Gaussian Splatting

In this paper, we present a neural sampling method, EllipssianNet, which predicts 2D Gaussians from an input RGB image. Trained with a Voronoi diagram-based synthetic dataset, EllipssianNet outputs a center map and a covariance map, which are combined with the colors sampled from the input image to generate 2D Gaussians. The Gaussians are anisotropic and aligned with local complexities of the input RGB image. The 2D Gaussians are converted into 3D ones that are then optimized and rasterized in the 3D Gaussian Splatting framework.EllipssianNet is tested in two applications. In Gaussian-based image representation, initialization with EllipssianNet enables faster convergence and higher rendering quality. EllipssianNet is also seamlessly integrated into a real-time SLAM system, producing high-quality reconstructions under online constraints.