GET3D AI technology page Top Builders
Explore the top contributors showcasing the highest number of GET3D AI technology page app submissions within our community.
GET3D by Nvidia
Introduction to GET3D: A Generative Model of High Quality 3D Textured Shapes
GET3D is a groundbreaking generative model designed by Nvidia. This technology synthesizes high-quality 3D textured shapes learned directly from images. It encompasses complex topologies and rich geometries, which can be directly consumed by 3D rendering engines.
Basic Theoretical Information
The GET3D model comprises the following key components:
- Generates 3D SDF & Texture Field: Two latent codes are utilized to create a 3D SDF and a texture field.
- Utilizes DMTet: The model uses DMTet to extract a 3D surface mesh from the SDF and retrieve the texture field's colors.
- Rasterization-Based Differentiable Renderer: It helps obtain RGB images and silhouettes using adversarial losses.
- End-to-End Trainable Model: With two 2D discriminators, the model classifies whether the inputs are real or fake.
Use Cases and Tips
GET3D is designed to generate a variety of textured 3D shapes, making it highly suitable for various industries.
- Virtual World Modeling: In sectors like gaming and VR, it generates textured meshes like cars, chairs, buildings, etc.
- Unsupervised Material Generation: Integrated with DIBR++, GET3D can produce view-dependent lighting effects.
- Text-Guided Shape Generation: Utilizing textual prompts, the model generates meaningful shapes.
- Diverse Shapes Generation: Leverage GET3D for creating diverse shapes with complex topology and high-quality textures.
- Disentanglement: GET3D efficiently disentangles geometry and texture, allowing unique control over shape creation.
- Latent Code Interpolation: For smooth transitions between different shapes, use random walks in the latent space.
How-To Set Up Guide with Code Samples
Setting up GET3D requires careful preparation. This guide provides a comprehensive overview of the necessary steps to prepare datasets, train the GET3D model, conduct inference, and evaluate the model's performance.
GET3D is trained using a synthetic dataset. Rendering scripts for ShapeNet are provided, and instructions for downloading the ShapeNet dataset and rendering it can be found in the accompanying README file.
Training the Model
1. Cloning the Repository and Acquiring Necessary Files
The necessary code and files can be obtained from GitLab by executing the following commands:
cd YOUR_CODE_PATH git clone firstname.lastname@example.org:nv-tlabs/GET3D.git cd GET3D; mkdir cache; cd cache wget https://api.ngc.nvidia.com/v2/models/nvidia/research/stylegan3/versions/1/files/metrics/inception-2015-12-05.pkl
2. Model Training
Training the model involves the following steps:
- Setting the working directory and environment variables:
cd YOUR_CODE_PATH export PYTHONPATH=$PWD:$PYTHONPATH export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
- Training on the unified generator for cars, motorbikes, or chairs (improved generator details can be found in the Appendix):
python train_3d.py --outdir=PATH_TO_LOG --data=PATH_TO_RENDER_IMG --camera_path PATH_TO_RENDER_CAMERA --gpus=8 --batch=32 --gamma=40 --data_camera_mode shapenet_car --dmtet_scale 1.0 --use_shapenet_split 1 --one_3d_generator 1 --fp32 0 python train_3d.py --outdir=PATH_TO_LOG --data=PATH_TO_RENDER_IMG --camera_path PATH_TO_RENDER_CAMERA --gpus=8 --batch=32 --gamma=80 --data_camera_mode shapenet_motorbike --dmtet_scale 1.0 --use_shapenet_split 1 --one_3d_generator 1 --fp32 0 python train_3d.py --outdir=PATH_TO_LOG --data=PATH_TO_RENDER_IMG --camera_path PATH_TO_RENDER_CAMERA --gpus=8 --batch=32 --gamma=400 --data_camera_mode shapenet_chair --dmtet_scale 0.8 --use_shapenet_split 1 --one_3d_generator 1 --fp32 0
Three different commands can be used for training on different objects, and variations exist for training on separate generators, as described in the main figure of the paper.
- Debugging the Model (Optional):
To debug the model, the number of GPUs can be reduced to 1 and the batch size to 4 using the flags
1. Visualization on a Pretrained Model
A pretrained model can be downloaded from the provided link for visualization. Inference can operate on a single GPU with 16 GB of memory:
python train_3d.py --outdir=save_inference_results/shapenet_car --gpus=1 --batch=4 --gamma=40 --data_camera_mode shapenet_car --dmtet_scale 1.0 --use_shapenet_split 1 --one_3d_generator 1 --fp32 0 --inference_vis 1 --resume_pretrain MODEL_PATH python train_3d.py --outdir=save_inference_results/shapenet_chair --gpus=1 --batch=4 --gamma=40 --data_camera_mode shapenet_chair --dmtet_scale 0.8 --use_shapenet_split 1 --one_3d_generator 1 --fp32 0 --inference_vis 1 --resume_pretrain MODEL_PATH python train_3d.py --outdir=save_inference_results/shapenet_motorbike --gpus=1 --batch=4 --gamma=40 --data_camera_mode shapenet_motorbike --dmtet_scale 1.0 --use_shapenet_split 1 --one_3d_generator 1 --fp32 0 --inference_vis 1 --resume_pretrain MODEL_PATH
Additional options can be added to the inference command to generate mesh with textures (
--inference_to_generate_textured_mesh 1) or to generate results with latent code interpolation (
1. Computing FID
To evaluate the model with the FID metric, add the option
--inference_compute_fid 1 to the inference command.
2. Computing COV & MMD Scores for LFD & CD
First, generate 3D objects for evaluation by adding the option
--inference_generate_geo 1 to the inference command. Then, follow the instructions in the README to compute the metrics.
Links for Useful Materials
- Paper: GET3D Paper
- Project Page: Project Page
- Nvidia Source Code License: License
- Pretrained Model on Shapenet: Download
- Code Release: Github Repository
The GET3D model by Nvidia opens up a new realm of possibilities in 3D shape generation. By understanding its theoretical underpinnings, real-world applications, and carefully following the setup guide, users can leverage this advanced tool for cutting-edge Point-E and 3D AI tasks. Visit the linked resources for more in-depth information and assistance.
GET3D AI technology page Hackathon projects
Discover innovative solutions crafted with GET3D AI technology page, developed by our community members during our engaging hackathons.
Transforming Text into Tangible 3D Objects
Our team harnessed the power of OpenAI's shap-e and gpt4all technologies to transform mere text into tangible 3D objects, all within a tight timeframe. But what sets our project apart is our commitment to sustainability and resourcefulness. We utilized recycled plastic filament as our raw material and self-assembled 3D printers for production. This project is not just about technological innovation. It's about envisioning a future where personalized consumer goods, from furniture to fashion items, can be produced on demand using sustainable materials. Join us as we delve deeper into this exciting journey of combining AI, 3D printing, and sustainability to revolutionize the manufacturing landscape.
PhenoVis 3D Learning
PhenoVis is a revolutionary educational platform that transcends traditional learning boundaries, inviting students to embark on a captivating journey of scientific exploration. Through cutting-edge 3D simulations, PhenoVis bridges the gap between theory and reality, offering learners an immersive experience of complex scientific phenomena. Imagine delving into the intricacies of physics, chemistry, and biology within interactive virtual environments. With real-time visualizations and interactive elements, students grasp concepts with unparalleled depth and engagement. PhenoVis empowers curiosity-driven learning, fostering a profound connection to scientific principles and encouraging a lifelong pursuit of knowledge. Join us in reshaping education through the transformative power of PhenoVis.
QuantumVisions, driven by the fusion of 3D AI's capabilities and the language of mathematics, embarks on a journey to craft captivating visual artistry that seamlessly connects the realms of scientific exploration and boundless human imagination. By harnessing the intricate power of mathematical concepts, this team aims to breathe life into abstract ideas, transforming them into mesmerizing 3D artworks that serve as intricate bridges between the worlds of analytical thought and artistic expression. Through QuantumVisions, the boundaries of science and creativity blur, inviting viewers to explore the interconnectedness of two seemingly distinct domains.