Computer Vision

hu-po

InstantSplat, DUSt3R

Stream SummaryWell, that was a wild ride through the cosmos of computer vision, wasn’t it? We’ve journeyed from the clunky, hand-crafted contraptions of yesteryear’s structure from motion pipelines to the sleek, learned elegance of InstantSplat and its trusty sidekick, Duster. It’s like swapping a horse-drawn carriage for a spaceship, and who wouldn’t want to travel at warp speed?InstantSplat is the new kid on the block, making waves by fusing the power of neural fields with the interpretability and composability of explicit 3D representations. It’s like having your cake and eating it too, without the need for obscure, hard-coded rules and a mountain of training images.Duster, the unsung hero of this saga, takes center stage by sidestepping the tedious camera calibration dance. It simply observes a scene from multiple angles and, with the wisdom of a seasoned space explorer, deduces the camera positions and a rough 3D sketch. This initial map then guides InstantSplat, allowing it to focus its efforts on refining the details and producing a stunning 3D reconstruction.It’s a match made in the heavens of computer vision, folks. This dynamic duo has left traditional methods in the dust, achieving state-of-the-art results on a plethora of tasks, from depth estimation to pose estimation, all while being faster and more user-friendly.So, whether you’re a seasoned computer vision veteran or a curious newcomer, InstantSplat and Duster are here to make your 3D dreams a reality. So buckle up, grab your yerba mate (or coffee, if you prefer), and prepare to explore the vast universe of 3D possibilities with these revolutionary tools!Paper AbstractsInstantSplat: Unbounded Sparse-view Pose-free Gaussian Splatting in 40 Seconds (http://arxiv.org/pdf/2403.20309v1)While novel view synthesis (NVS) has made substantial progress in 3D computervision, it typically requires an initial estimation of camera intrinsics andextrinsics from dense viewpoints. This pre-processing is usually conducted viaa Structure-from-Motion (SfM) pipeline, a procedure that can be slow andunreliable, particularly in sparse-view scenarios with insufficient matchedfeatures for accurate reconstruction. In this work, we integrate the strengthsof point-based representations (e.g., 3D Gaussian Splatting, 3D-GS) withend-to-end dense stereo models (DUSt3R) to tackle the complex yet unresolvedissues in NVS under unconstrained settings, which encompasses pose-free andsparse view challenges. Our framework, InstantSplat, unifies dense stereopriors with 3D-GS to build 3D Gaussians of large-scale scenes from sparseview &pose-free images in less than 1 minute. Specifically, InstantSplat comprises aCoarse Geometric Initialization (CGI) module that swiftly establishes apreliminary scene structure and camera parameters across all training views,utilizing globally-aligned 3D point maps derived from a pre-trained densestereo pipeline. This is followed by the Fast 3D-Gaussian Optimization (F-3DGO)module, which jointly optimizes the 3D Gaussian attributes and the initializedposes with pose regularization. Experiments conducted on the large-scaleoutdoor Tanks & Temples datasets demonstrate that InstantSplat significantlyimproves SSIM (by 32%) while concurrently reducing Absolute Trajectory Error(ATE) by 80%. These establish InstantSplat as a viable solution for scenariosinvolving posefree and sparse-view conditions. Project page:instantsplat.github.io.DUSt3R: Geometric 3D Vision Made Easy (http://arxiv.org/pdf/2312.14132v1)Multi-view stereo reconstruction (MVS) in the wild requires to first estimatethe camera parameters e.g. intrinsic and extrinsic parameters. These areusually tedious and cumbersome to obtain, yet they are mandatory to triangulatecorresponding pixels in 3D space, which is the core of all best performing MVSalgorithms. In this work, we take an opposite stance and introduce DUSt3R, aradically novel paradigm for Dense and Unconstrained Stereo 3D Reconstructionof arbitrary image collections, i.e. operating without prior information aboutcamera calibration nor viewpoint poses. We cast the pairwise reconstructionproblem as a regression of pointmaps, relaxing the hard constraints of usualprojective camera models. We show that this formulation smoothly unifies themonocular and binocular reconstruction cases. In the case where more than twoimages are provided, we further propose a simple yet effective global alignmentstrategy that expresses all pairwise pointmaps in a common reference frame. Webase our network architecture on standard Transformer encoders and decoders,allowing us to leverage powerful pretrained models. Our formulation directlyprovides a 3D model of the scene as well as depth information, butinterestingly, we can seamlessly recover from it, pixel matches, relative andabsolute camera. Exhaustive experiments on all these tasks showcase that theproposed DUSt3R can unify various 3D vision tasks and set new SoTAs onmonocular/multi-view depth estimation as well as relative pose estimation. Insummary, DUSt3R makes many geometric 3D vision tasks easy.This content was generated by gemini-1.5-pro-latest on 2024–04–10. Find more Hu-Po content on:youtube.com/@hu-potwitter.com/hupobubootwitch.tv/hu_podiscord.gg/pPAFwndTJdsubstack.com/@huporeddit.com/r/hupos/linkedin.com/in/hugoponte/tiktok.com/@hu.po.aipatreon.com/user?u=89667142hu-po.medium.comresearchhub.com/user/986081/overviewhu-po.github.io/