diff --git a/.cursor/rules/project-rules.mdc b/.cursor/rules/project-rules.mdc index da34d0b..562e702 100644 --- a/.cursor/rules/project-rules.mdc +++ b/.cursor/rules/project-rules.mdc @@ -7,6 +7,7 @@ globs: - 使用中文回复 - 先查看Reference文件夹下的代码参考, 参考虚拟环境的创建和管理和使用,并按照 Readme.md 来查Readme和install.sh和run.sh以及Reference路径下参考代码来协助用户完成项目 +- 完善Readme.md - Reference文件夹仅作为参考而不参与项目运行 - 将该项目改成Windows平台应用, 代码必须考虑可在windows运行, 使用micromamba.exe来拿新建和管理虚拟环境 - 对于不同的环境分别创建activate的bat文件,虚拟环境分别有:gaussian_splatting_hair, matte_anything, openpose, pixie-env对应的文件有:activate_gaussian_splatting_hair.bat, activate_matte_anything.bat, activate_openpose.bat, activate_pixie-env.bat diff --git a/README.md b/README.md index 7d6a117..e2e2553 100644 --- a/README.md +++ b/README.md @@ -1,26 +1,28 @@ -# Gaussian Haircut: Human Hair Reconstruction with Strand-Aligned 3D Gaussians +# Gaussian Haircut:使用股线对齐 3D 高斯模型进行人体头发重建 -[**Paper**](https://arxiv.org/abs/2409.14778) | [**Project Page**](https://eth-ait.github.io/GaussianHaircut/) +[**论文**](https://arxiv.org/abs/2409.14778) | [**项目页面**](https://eth-ait.github.io/GaussianHaircut/) -This repository contains an official implementation of Gaussian Haircut, a strand-based hair reconstruction approach for monocular videos. +本仓库包含了 Gaussian Haircut 的官方实现,这是一种基于股线的人体头发重建方法,用于单目视频。 -## Getting started +## 入门指南 -1. **Install CUDA 11.8** +### Linux 平台 - Follow the instructions on https://developer.nvidia.com/cuda-11-8-0-download-archive. +1. **安装 CUDA 11.8** - Make sure that - - PATH includes /bin - - LD_LIBRARY_PATH includes /lib64 + 按照 https://developer.nvidia.com/cuda-11-8-0-download-archive 上的说明进行操作。 - The environment was tested only with this CUDA version. + 确保: + - PATH 包含 /bin + - LD_LIBRARY_PATH 包含 /lib64 -2. **Install Blender 3.6** in order to create strand visualizations + 该环境仅在此 CUDA 版本下进行了测试。 - Follow instructions on https://www.blender.org/download/lts/3-6. +2. **安装 Blender 3.6** 以创建股线可视化 -3. **Close the repo and run the install script** + 按照 https://www.blender.org/download/lts/3-6 上的说明进行操作。 + +3. **克隆仓库并运行安装脚本** ```bash git clone git@github.com:eth-ait/GaussianHaircut.git @@ -29,33 +31,71 @@ This repository contains an official implementation of Gaussian Haircut, a stran ./install.sh ``` -## Reconstruction +### Windows 平台 -1. **Record a monocular video** +1. **安装 CUDA 11.8** + - 从 https://developer.nvidia.com/cuda-11-8-0-download-archive 下载并安装 + - 默认安装路径:C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8 - Use examples on the project page as references and introduce as little motion blur as possible. +2. **安装 Blender 3.6** + - 从 https://www.blender.org/download/lts/3-6 下载并安装 + - 默认安装路径:C:\Program Files\Blender Foundation\Blender 3.6 -2. **Setup a directory for the reconstructed scene** +3. **安装 Visual Studio 2019 Build Tools** + - 从 https://visualstudio.microsoft.com/vs/older-downloads/ 下载并安装 + - 选择"C++构建工具"工作负载 + - 默认安装路径:C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools - Put the video file in it and rename it to raw.mp4 +4. **安装 COLMAP** + - 从 https://github.com/colmap/colmap/releases 下载并安装 + - 确保将COLMAP添加到系统PATH环境变量中 -3. **Run the script** +5. **安装 micromamba** + - 从 https://mamba.readthedocs.io/en/latest/installation.html 下载并安装 +6. **克隆仓库并运行安装脚本** + + ```cmd + git clone git@github.com:eth-ait/GaussianHaircut.git + cd GaussianHaircut + install.bat + ``` + +## 重建 + +1. **录制单目视频** + + 参考项目页面上的示例,并尽可能减少运动模糊。 + +2. **设置重建场景的目录** + + 将视频文件放入其中,并将其重命名为 raw.mp4 + +3. **运行脚本** + +Linux: ```bash export PROJECT_DIR="[/path/to/]GaussianHaircut" export BLENDER_DIR="[/path/to/blender/folder/]blender" DATA_PATH="[path/to/scene/folder]" ./run.sh ``` - The script performs data pre-processing, reconstruction, and final visualization using Blender. Use Tensorboard to see intermediate visualizations. +Windows: + ```cmd + set PROJECT_DIR=[path\to\]GaussianHaircut + set DATA_PATH=[path\to\scene\folder] + run.bat + ``` -## License + 该脚本执行数据预处理、重建以及使用 Blender 进行最终可视化。使用 Tensorboard 查看中间可视化结果。 -This code is based on the 3D Gaussian Splatting project. For terms and conditions, please refer to LICENSE_3DGS. The rest of the code is distributed under CC BY-NC-SA 4.0. +## 许可证 -If this code is helpful in your project, cite the papers below. +此代码基于 3D Gaussian Splatting 项目。有关条款和条件,请参阅 LICENSE_3DGS。其余代码根据 CC BY-NC-SA 4.0 分发。 -## Citation +如果此代码对您的项目有帮助,请引用以下论文。 + +## 引用 ``` @inproceedings{zakharov2024gh, @@ -66,16 +106,11 @@ If this code is helpful in your project, cite the papers below. } ``` -## Links +## 链接 - [3D Gaussian Splatting](https://github.com/graphdeco-inria/gaussian-splatting) - -- [Neural Haircut](https://github.com/SamsungLabs/NeuralHaircut): FLAME fitting pipeline, strand prior and hairstyle diffusion prior - -- [HAAR](https://github.com/Vanessik/HAAR): hair upsampling - -- [Matte-Anything](https://github.com/hustvl/Matte-Anything): hair and body segmentation - -- [PIXIE](https://github.com/yfeng95/PIXIE): initialization for FLAME fitting - -- [Face-Alignment](https://github.com/1adrianb/face-alignment), [OpenPose](https://github.com/CMU-Perceptual-Computing-Lab/openpose): keypoints detection for FLAME fitting +- [Neural Haircut](https://github.com/SamsungLabs/NeuralHaircut): FLAME 拟合管线、股线先验和发型扩散先验 +- [HAAR](https://github.com/Vanessik/HAAR): 头发上采样 +- [Matte-Anything](https://github.com/hustvl/Matte-Anything): 头发和身体分割 +- [PIXIE](https://github.com/yfeng95/PIXIE): FLAME 拟合的初始化 +- [Face-Alignment](https://github.com/1adrianb/face-alignment), [OpenPose](https://github.com/CMU-Perceptual-Computing-Lab/openpose): 用于 FLAME 拟合的关键点检测 diff --git a/Reference/activate.bat b/Reference/activate.bat new file mode 100644 index 0000000..1d7160f --- /dev/null +++ b/Reference/activate.bat @@ -0,0 +1,2 @@ +@CALL "%~dp0micromamba.exe" shell init --shell cmd.exe --prefix "%~dp0\" +start cmd /k "%~dp0condabin\micromamba.bat" activate StableDiffusion_001 diff --git a/Reference/install.bat b/Reference/install.bat new file mode 100644 index 0000000..b59b304 --- /dev/null +++ b/Reference/install.bat @@ -0,0 +1,32 @@ +CALL "%~dp0micromamba.exe" create -n StableDiffusion_001 python==3.10.14 git==2.41.0 git-lfs==3.2.0 -c pytorch -c conda-forge -r "%~dp0\" -y +@CALL "%~dp0micromamba.exe" shell init --shell cmd.exe --prefix "%~dp0\" +@CALL condabin\micromamba.bat activate StableDiffusion_001 +@CALL set GDOWN_CACHE=cache\gdown +@CALL set TORCH_HOME=cache\torch +@CALL set HF_HOME=cache\huggingface +@CALL set PYTHONDONTWRITEBYTECODE=1 +@CALL set CUDA_HOME=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.1 +@CALL pip install --force-reinstall torch==2.2.0+cu121 torchvision==0.17.0+cu121 torchaudio==2.2.0+cu121 --index-url https://download.pytorch.org/whl/cu121 --no-cache-dir +@CALL git clone https://github.com/facebookresearch/xformers.git +@CALL cd xformers +@CALL git submodule update --init --recursive +@CALL pip install . +@CALL cd .. +@CALL git clone https://github.com/NVlabs/nvdiffrast .\nvdiffrast +@CALL cd nvdiffrast +@CALL pip install . +@CALL cd .. +@CALL git clone -b v1.9.4 https://github.com/AUTOMATIC1111/stable-diffusion-webui/ .\stable-diffusion-webui\ +@CALL mkdir .\stable-diffusion-webui\cache\gdown\ +@CALL mkdir .\stable-diffusion-webui\cache\torch\ +@CALL mkdir .\stable-diffusion-webui\cache\huggingface\ +@CALL mkdir .\stable-diffusion-webui\repositories\ +@CALL git clone https://github.com/CompVis/stable-diffusion.git .\stable-diffusion-webui\repositories\stable-diffusion +@CALL git clone https://github.com/CompVis/taming-transformers.git .\stable-diffusion-webui\repositories\taming-transformers +@CALL git clone https://github.com/crowsonkb/k-diffusion.git .\stable-diffusion-webui\repositories\k-diffusion +@CALL git clone https://github.com/sczhou/CodeFormer.git .\stable-diffusion-webui\repositories\CodeFormer +@CALL git clone http://github.com/salesforce/BLIP.git .\stable-diffusion-webui\repositories\BLIP +@CALL cd .\stable-diffusion-webui +@CALL pip install -r .\stable-diffusion-webui\requirements.txt +@CALL python -B launch.py --xformers --opt-sdp-attention --autolaunch --theme dark --listen --port 7860 --enable-insecure-extension-access --skip-torch-cuda-test +@CALL PAUSE diff --git a/Reference/micromamba.exe b/Reference/micromamba.exe new file mode 100644 index 0000000..bd9e56a --- /dev/null +++ b/Reference/micromamba.exe @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:77f9efb2c50a480086b93320829bb12cfd78ebaa9a5285a917b143e9b848378c +size 8381952 diff --git a/Reference/requirements.txt b/Reference/requirements.txt new file mode 100644 index 0000000..b338ba6 --- /dev/null +++ b/Reference/requirements.txt @@ -0,0 +1,42 @@ +pip==23.3.1 +xformers==0.0.21 +torch==2.0.1+cu118 +torchvision +torchaudio +torchtext +PySoundFile +torchdiffeq==0.2.3 +torchsde==0.2.5 +GitPython==3.1.32 +Pillow==9.5.0 +accelerate==0.21.0 +basicsr==1.4.2 +blendmodes==2022 +clean-fid==0.1.35 +einops==0.4.1 +fastapi==0.94.0 +gfpgan==1.3.8 +gradio==3.41.2 +httpcore==0.15 +inflection==0.5.1 +jsonmerge==1.8.0 +kornia==0.6.7 +lark==1.1.2 +numpy==1.23.5 +omegaconf==2.2.3 +open-clip-torch==2.20.0 +piexif==1.1.3 +psutil==5.9.5 +pytorch_lightning==1.9.4 +realesrgan==0.3.0 +resize-right==0.0.2 +safetensors==0.3.1 +scikit-image==0.21.0 +timm==0.9.2 +tomesd==0.1.3 +transformers==4.30.2 +httpx==0.24.1 +groundingdino +segment_anything +supervision +pyfunctional \ No newline at end of file diff --git a/Reference/run.bat b/Reference/run.bat new file mode 100644 index 0000000..244ce61 --- /dev/null +++ b/Reference/run.bat @@ -0,0 +1,11 @@ +@CALL "%~dp0micromamba.exe" shell init --shell cmd.exe --prefix "%~dp0\" +@CALL condabin\micromamba.bat activate StableDiffusion_001 +@CALL set GDOWN_CACHE=cache\gdown +@CALL set TORCH_HOME=cache\torch +@CALL set HF_HOME=cache\huggingface +@CALL set PYTHONDONTWRITEBYTECODE=1 +@CALL set CUDA_HOME=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.1 +@CALL set COMMANDLINE_ARGS=--listen --port 7860 --autolaunch --theme dark --xformers --opt-sdp-attention --api --skip-install --skip-torch-cuda-test --skip-version-check --enable-insecure-extension-access +@CALL cd stable-diffusion-webui +@CALL python -B webui.py %COMMANDLINE_ARGS% +@CALL PAUSE diff --git a/activate_gaussian_splatting_hair.bat b/activate_gaussian_splatting_hair.bat index 5f28270..19adeaf 100644 --- a/activate_gaussian_splatting_hair.bat +++ b/activate_gaussian_splatting_hair.bat @@ -1 +1,13 @@ - \ No newline at end of file +@echo off +SET CUDA_HOME="C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8" +SET PATH=%CUDA_HOME%\bin;%PATH% +SET PROJECT_DIR=%~dp0 +SET PYTHONDONTWRITEBYTECODE=1 +SET TORCH_HOME=%PROJECT_DIR%\cache\torch +SET HF_HOME=%PROJECT_DIR%\cache\huggingface +SET GDOWN_CACHE=%PROJECT_DIR%\cache\gdown +SET BLENDER_DIR="C:\Program Files\Blender Foundation\Blender 3.6" +SET VS2019_DIR="C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools" +SET VS2019_VCVARS="%VS2019_DIR%\VC\Auxiliary\Build\vcvars64.bat" + +CALL micromamba activate gaussian_splatting_hair diff --git a/activate_matte_anything.bat b/activate_matte_anything.bat index 5f28270..a552372 100644 --- a/activate_matte_anything.bat +++ b/activate_matte_anything.bat @@ -1 +1,13 @@ - \ No newline at end of file +@echo off +SET CUDA_HOME="C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8" +SET PATH=%CUDA_HOME%\bin;%PATH% +SET PROJECT_DIR=%~dp0 +SET PYTHONDONTWRITEBYTECODE=1 +SET TORCH_HOME=%PROJECT_DIR%\cache\torch +SET HF_HOME=%PROJECT_DIR%\cache\huggingface +SET GDOWN_CACHE=%PROJECT_DIR%\cache\gdown +SET BLENDER_DIR="C:\Program Files\Blender Foundation\Blender 3.6" +SET VS2019_DIR="C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools" +SET VS2019_VCVARS="%VS2019_DIR%\VC\Auxiliary\Build\vcvars64.bat" + +CALL micromamba activate matte_anything diff --git a/activate_openpose.bat b/activate_openpose.bat index 5f28270..6337ef2 100644 --- a/activate_openpose.bat +++ b/activate_openpose.bat @@ -1 +1,13 @@ - \ No newline at end of file +@echo off +SET CUDA_HOME="C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8" +SET PATH=%CUDA_HOME%\bin;%PATH% +SET PROJECT_DIR=%~dp0 +SET PYTHONDONTWRITEBYTECODE=1 +SET TORCH_HOME=%PROJECT_DIR%\cache\torch +SET HF_HOME=%PROJECT_DIR%\cache\huggingface +SET GDOWN_CACHE=%PROJECT_DIR%\cache\gdown +SET BLENDER_DIR="C:\Program Files\Blender Foundation\Blender 3.6" +SET VS2019_DIR="C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools" +SET VS2019_VCVARS="%VS2019_DIR%\VC\Auxiliary\Build\vcvars64.bat" + +CALL micromamba activate openpose diff --git a/activate_pixie-env.bat b/activate_pixie-env.bat index 5f28270..8a8ec33 100644 --- a/activate_pixie-env.bat +++ b/activate_pixie-env.bat @@ -1 +1,13 @@ - \ No newline at end of file +@echo off +SET CUDA_HOME="C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8" +SET PATH=%CUDA_HOME%\bin;%PATH% +SET PROJECT_DIR=%~dp0 +SET PYTHONDONTWRITEBYTECODE=1 +SET TORCH_HOME=%PROJECT_DIR%\cache\torch +SET HF_HOME=%PROJECT_DIR%\cache\huggingface +SET GDOWN_CACHE=%PROJECT_DIR%\cache\gdown +SET BLENDER_DIR="C:\Program Files\Blender Foundation\Blender 3.6" +SET VS2019_DIR="C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools" +SET VS2019_VCVARS="%VS2019_DIR%\VC\Auxiliary\Build\vcvars64.bat" + +CALL micromamba activate pixie-env diff --git a/install.bat b/install.bat index 5f28270..ca7eb38 100644 --- a/install.bat +++ b/install.bat @@ -1 +1,145 @@ - \ No newline at end of file +@echo off +setlocal enabledelayedexpansion + +REM 设置环境变量 +SET CUDA_HOME=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8 +SET PATH=%CUDA_HOME%\bin;%PATH% +SET PROJECT_DIR=%~dp0 +SET PYTHONDONTWRITEBYTECODE=1 +SET GDOWN_CACHE=cache\gdown +SET TORCH_HOME=cache\torch +SET HF_HOME=cache\huggingface +SET BLENDER_DIR=C:\Program Files\Blender Foundation\Blender 3.6 +SET VS2019_DIR=C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools +SET VS2019_VCVARS=%VS2019_DIR%\VC\Auxiliary\Build\vcvars64.bat + +REM 检查必要的环境和依赖 +IF NOT EXIST "%CUDA_HOME%" ( + echo ERROR: CUDA 11.8 not found at %CUDA_HOME% + exit /b 1 +) +IF NOT EXIST "%BLENDER_DIR%" ( + echo ERROR: Blender 3.6 not found at %BLENDER_DIR% + echo Please install Blender 3.6 from https://www.blender.org/download/lts/3-6/ + exit /b 1 +) + +REM 检查Visual Studio +IF NOT EXIST "%VS2019_VCVARS%" ( + echo ERROR: Visual Studio 2019 not found + echo Please install Visual Studio 2019 Build Tools with C++ development tools + exit /b 1 +) + +REM 检查COLMAP +where colmap >nul 2>nul +IF %ERRORLEVEL% NEQ 0 ( + echo ERROR: COLMAP not found in PATH + echo Please install COLMAP and add it to PATH + exit /b 1 +) + +REM 创建缓存目录 +mkdir cache\gdown +mkdir cache\torch +mkdir cache\huggingface + +REM 创建ext目录 +mkdir ext +cd ext + +REM 克隆外部库 +git clone https://github.com/CMU-Perceptual-Computing-Lab/openpose --depth 1 +cd openpose +git submodule update --init --recursive --remote +cd .. + +git clone https://github.com/hustvl/Matte-Anything +cd Matte-Anything +git clone https://github.com/IDEA-Research/GroundingDINO.git +cd .. + +git clone https://github.com/egorzakharov/NeuralHaircut.git --recursive +git clone https://github.com/facebookresearch/pytorch3d +cd pytorch3d +git checkout 2f11ddc5ee7d6bd56f2fb6744a16776fab6536f7 +cd .. + +git clone https://github.com/camenduru/simple-knn +cd diff_gaussian_rasterization_hair\third_party +git clone https://github.com/g-truc/glm +cd glm +git checkout 5c46b9c07008ae65cb81ab79cd677ecc1934b903 +cd ..\..\.. + +git clone --recursive https://github.com/NVIDIAGameWorks/kaolin +cd kaolin +git checkout v0.15.0 +cd .. + +git clone https://github.com/SSL92/hyperIQA + +REM 创建环境 +CALL micromamba create -n gaussian_splatting_hair python=3.8 pytorch=2.0.0 torchvision pytorch-cuda=11.8 cmake ninja -c pytorch -c nvidia -c conda-forge -y +CALL micromamba create -n matte_anything pytorch=2.0.0 pytorch-cuda=11.8 torchvision tensorboard timm=0.5.4 opencv=4.5.3 mkl=2024.0 setuptools=58.2.0 easydict wget scikit-image gradio=3.46.1 fairscale -c pytorch -c nvidia -c conda-forge -y +CALL micromamba create -n openpose python=3.8 cmake=3.20 -c conda-forge -y +CALL micromamba create -n pixie-env python=3.8 pytorch=2.0.0 torchvision pytorch-cuda=11.8 fvcore pytorch3d==0.7.5 kornia matplotlib -c pytorch -c nvidia -c fvcore -c conda-forge -c pytorch3d -y + +REM 安装 gaussian_splatting_hair 环境 +CALL activate_gaussian_splatting_hair.bat +pip install -r requirements.txt +cd %PROJECT_DIR%\ext\pytorch3d +pip install -e . +cd %PROJECT_DIR%\ext\NeuralHaircut\npbgpp +pip install -e . +cd %PROJECT_DIR%\ext\simple-knn +pip install -e . +cd %PROJECT_DIR%\ext\diff_gaussian_rasterization_hair +pip install -e . +cd %PROJECT_DIR%\ext\kaolin +pip install -e . +cd %PROJECT_DIR% + +REM 下载 Neural Haircut 文件 +cd %PROJECT_DIR%\ext\NeuralHaircut +gdown --folder https://drive.google.com/drive/folders/1TCdJ0CKR3Q6LviovndOkJaKm8S1T9F_8 +cd pretrained_models\diffusion_prior +gdown 1_9EOUXHayKiGH5nkrayncln3d6m1uV7f +cd ..\..\PIXIE +gdown 1mPcGu62YPc4MdkT8FFiOCP629xsENHZf +tar -xf pixie_data.tar.gz +del pixie_data.tar.gz + +REM 安装 Matte-Anything 环境 +CALL activate_matte_anything.bat +cd %PROJECT_DIR%\ext\Matte-Anything +pip install git+https://github.com/facebookresearch/segment-anything.git +pip install 'git+https://github.com/facebookresearch/detectron2.git' +pip install -e GroundingDINO +pip install supervision==0.22.0 +mkdir pretrained +cd pretrained +curl -LO https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth +curl -LO https://github.com/IDEA-Research/GroundingDINO/releases/download/v0.1.0-alpha/groundingdino_swint_ogc.pth +cd .. +gdown 1d97oKuITCeWgai2Tf3iNilt6rMSSYzkW + +REM 安装 OpenPose +CALL activate_openpose.bat +cd %PROJECT_DIR%\ext\openpose +gdown 1Yn03cKKfVOq4qXmgBMQD20UMRRRkd_tV +tar -xf models.tar.gz +del models.tar.gz +mkdir build +cd build +CALL "%VS2019_VCVARS%" +cmake .. -DBUILD_PYTHON=true -DUSE_CUDNN=off -G "Visual Studio 16 2019" -A x64 +cmake --build . --config Release + +REM 安装 PIXIE 环境 +CALL activate_pixie-env.bat +cd %PROJECT_DIR%\ext\PIXIE +pip install pyyaml==5.4.1 +pip install git+https://github.com/1adrianb/face-alignment.git@54623537fd9618ca7c15688fd85aba706ad92b59 + +echo Installation completed! diff --git a/requirements.txt b/requirements.txt new file mode 100644 index 0000000..f0d29b7 --- /dev/null +++ b/requirements.txt @@ -0,0 +1,35 @@ +numpy>=1.21.0 +scipy>=1.7.0 +pillow>=8.3.1 +tqdm>=4.62.2 +matplotlib>=3.4.2 +opencv-python>=4.5.3 +scikit-image>=0.18.2 +imageio>=2.9.0 +lpips>=0.1.4 +gdown>=4.7.1 +face-alignment>=1.3.5 +pytorch3d==0.7.5 +ninja>=1.10.2 +trimesh>=3.9.35 +pyhocon>=0.3.59 +wandb>=0.12.0 +tensorboard>=2.6.0 +open3d>=0.13.0 +kaolin>=0.15.0 +pytorch-lightning>=1.6.0 +plyfile>=0.8.1 +icecream>=2.1.3 +einops>=0.6.0 +accelerate>=0.18.0 +jsonmerge>=1.9.0 +easydict>=1.9 +tensorboardx>=2.6 +fvcore>=0.1.5 +toml>=0.10.2 +pysdf>=0.1.9 +clean-fid>=0.1.35 +clip>=0.2.0 +torchdiffeq>=0.2.3 +torchsde>=0.2.5 +resize-right>=0.0.2 \ No newline at end of file diff --git a/run.bat b/run.bat index 5f28270..92b2675 100644 --- a/run.bat +++ b/run.bat @@ -1 +1,197 @@ - \ No newline at end of file +@echo off +setlocal enabledelayedexpansion + +REM 设置环境变量 +SET GPU=0 +SET CAMERA=PINHOLE +SET EXP_NAME_1=stage1 +SET EXP_NAME_2=stage2 +SET EXP_NAME_3=stage3 +SET BLENDER_DIR="C:\Program Files\Blender Foundation\Blender 3.6" +SET CUDA_HOME="C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8" + +REM 确保以下环境变量可用: +REM PROJECT_DIR 和 DATA_PATH + +REM 检查必要的环境变量 +IF "%PROJECT_DIR%"=="" ( + echo ERROR: PROJECT_DIR environment variable is not set + exit /b 1 +) +IF "%DATA_PATH%"=="" ( + echo ERROR: DATA_PATH environment variable is not set + exit /b 1 +) +IF NOT EXIST "%BLENDER_DIR%" ( + echo ERROR: BLENDER_DIR path does not exist: %BLENDER_DIR% + exit /b 1 +) + +REM ################## +REM # 预处理阶段 # +REM ################## + +REM 将原始图像整理成3D Gaussian Splatting格式 +CALL activate_gaussian_splatting_hair.bat +cd %PROJECT_DIR%\src\preprocessing +set CUDA_VISIBLE_DEVICES=%GPU% +python preprocess_raw_images.py --data_path %DATA_PATH% + +REM 运行COLMAP重建并去畸变图像和相机 +CALL activate_gaussian_splatting_hair.bat +cd %PROJECT_DIR%\src +set CUDA_VISIBLE_DEVICES=%GPU% +python convert.py -s %DATA_PATH% --camera %CAMERA% --max_size 1024 + +REM 运行Matte-Anything +CALL activate_matte_anything.bat +cd %PROJECT_DIR%\src\preprocessing +set CUDA_VISIBLE_DEVICES=%GPU% +python calc_masks.py --data_path %DATA_PATH% --image_format png --max_size 2048 + +REM 使用IQA分数过滤图像 +CALL activate_gaussian_splatting_hair.bat +cd %PROJECT_DIR%\src\preprocessing +set CUDA_VISIBLE_DEVICES=%GPU% +python filter_extra_images.py --data_path %DATA_PATH% --max_imgs 128 + +REM 调整图像大小 +CALL activate_gaussian_splatting_hair.bat +cd %PROJECT_DIR%\src\preprocessing +set CUDA_VISIBLE_DEVICES=%GPU% +python resize_images.py --data_path %DATA_PATH% + +REM 计算方向图 +CALL activate_gaussian_splatting_hair.bat +cd %PROJECT_DIR%\src\preprocessing +set CUDA_VISIBLE_DEVICES=%GPU% +python calc_orientation_maps.py --img_path %DATA_PATH%\images_2 --mask_path %DATA_PATH%\masks_2\hair --orient_dir %DATA_PATH%\orientations_2\angles --conf_dir %DATA_PATH%\orientations_2\vars --filtered_img_dir %DATA_PATH%\orientations_2\filtered_imgs --vis_img_dir %DATA_PATH%\orientations_2\vis_imgs + +REM 运行OpenPose +CALL activate_openpose.bat +cd %PROJECT_DIR%\ext\openpose +mkdir %DATA_PATH%\openpose +set CUDA_VISIBLE_DEVICES=%GPU% +"%PROJECT_DIR%\ext\openpose\build\x64\Release\OpenPoseDemo.exe" --image_dir %DATA_PATH%\images_4 --scale_number 4 --scale_gap 0.25 --face --hand --display 0 --write_json %DATA_PATH%\openpose\json --write_images %DATA_PATH%\openpose\images --write_images_format jpg + +REM 运行Face-Alignment +CALL activate_gaussian_splatting_hair.bat +cd %PROJECT_DIR%\src\preprocessing +set CUDA_VISIBLE_DEVICES=%GPU% +python calc_face_alignment.py --data_path %DATA_PATH% --image_dir "images_4" + +REM 运行PIXIE +CALL activate_pixie-env.bat +cd %PROJECT_DIR%\ext\PIXIE +set CUDA_VISIBLE_DEVICES=%GPU% +python demos\demo_fit_face.py -i %DATA_PATH%\images_4 -s %DATA_PATH%\pixie --saveParam True --lightTex False --useTex False --rasterizer_type pytorch3d + +REM 合并所有PIXIE预测到单个文件 +CALL activate_gaussian_splatting_hair.bat +cd %PROJECT_DIR%\src\preprocessing +set CUDA_VISIBLE_DEVICES=%GPU% +python merge_smplx_predictions.py --data_path %DATA_PATH% + +REM 将COLMAP相机转换为txt +CALL activate_gaussian_splatting_hair.bat +mkdir %DATA_PATH%\sparse_txt +set CUDA_VISIBLE_DEVICES=%GPU% +colmap model_converter --input_path %DATA_PATH%\sparse\0 --output_path %DATA_PATH%\sparse_txt --output_type TXT + +REM 将COLMAP相机转换为H3DS格式 +CALL activate_gaussian_splatting_hair.bat +cd %PROJECT_DIR%\src\preprocessing +set CUDA_VISIBLE_DEVICES=%GPU% +python colmap_parsing.py --path_to_scene %DATA_PATH% + +REM 删除原始文件以节省磁盘空间 +rmdir /s /q %DATA_PATH%\input %DATA_PATH%\images %DATA_PATH%\masks %DATA_PATH%\iqa* + +REM ################## +REM # 重建阶段 # +REM ################## + +set EXP_PATH_1=%DATA_PATH%\3d_gaussian_splatting\%EXP_NAME_1% + +REM 运行3D Gaussian Splatting重建 +CALL activate_gaussian_splatting_hair.bat +cd %PROJECT_DIR%\src +set CUDA_VISIBLE_DEVICES=%GPU% +python train_gaussians.py -s %DATA_PATH% -m "%EXP_PATH_1%" -r 1 --port "888%GPU%" --trainable_cameras --trainable_intrinsics --use_barf --lambda_dorient 0.1 + +REM 运行FLAME网格拟合 +CALL activate_gaussian_splatting_hair.bat +cd %PROJECT_DIR%\ext\NeuralHaircut\src\multiview_optimization + +set CUDA_VISIBLE_DEVICES=%GPU% +python fit.py --conf confs\train_person_1.conf --batch_size 1 --train_rotation True --fixed_images True --save_path %DATA_PATH%\flame_fitting\%EXP_NAME_1%\stage_1 --data_path %DATA_PATH% --fitted_camera_path %EXP_PATH_1%\cameras\30000_matrices.pkl + +python fit.py --conf confs\train_person_1.conf --batch_size 4 --train_rotation True --fixed_images True --save_path %DATA_PATH%\flame_fitting\%EXP_NAME_1%\stage_2 --checkpoint_path %DATA_PATH%\flame_fitting\%EXP_NAME_1%\stage_1\opt_params_final --data_path %DATA_PATH% --fitted_camera_path %EXP_PATH_1%\cameras\30000_matrices.pkl + +python fit.py --conf confs\train_person_1_.conf --batch_size 32 --train_rotation True --train_shape True --save_path %DATA_PATH%\flame_fitting\%EXP_NAME_1%\stage_3 --checkpoint_path %DATA_PATH%\flame_fitting\%EXP_NAME_1%\stage_2\opt_params_final --data_path %DATA_PATH% --fitted_camera_path %EXP_PATH_1%\cameras\30000_matrices.pkl + +REM 裁剪重建的场景 +CALL activate_gaussian_splatting_hair.bat +cd %PROJECT_DIR%\src\preprocessing +set CUDA_VISIBLE_DEVICES=%GPU% +python scale_scene_into_sphere.py --path_to_data %DATA_PATH% -m "%DATA_PATH%\3d_gaussian_splatting\%EXP_NAME_1%" --iter 30000 + +REM 移除与FLAME头部网格相交的头发高斯体 +CALL activate_gaussian_splatting_hair.bat +cd %PROJECT_DIR%\src\preprocessing +set CUDA_VISIBLE_DEVICES=%GPU% +python filter_flame_intersections.py --flame_mesh_dir %DATA_PATH%\flame_fitting\%EXP_NAME_1% -m "%DATA_PATH%\3d_gaussian_splatting\%EXP_NAME_1%" --iter 30000 --project_dir %PROJECT_DIR%\ext\NeuralHaircut + +REM 为训练视图运行渲染 +CALL activate_gaussian_splatting_hair.bat +cd %PROJECT_DIR%\src +set CUDA_VISIBLE_DEVICES=%GPU% +python render_gaussians.py -s %DATA_PATH% -m "%DATA_PATH%\3d_gaussian_splatting\%EXP_NAME_1%" --skip_test --scene_suffix "_cropped" --iteration 30000 --trainable_cameras --trainable_intrinsics --use_barf + +REM 获取FLAME网格头皮图 +CALL activate_gaussian_splatting_hair.bat +cd %PROJECT_DIR%\src\preprocessing +set CUDA_VISIBLE_DEVICES=%GPU% +python extract_non_visible_head_scalp.py --project_dir %PROJECT_DIR%\ext\NeuralHaircut --data_dir %DATA_PATH% --flame_mesh_dir %DATA_PATH%\flame_fitting\%EXP_NAME_1% --cams_path %DATA_PATH%\3d_gaussian_splatting\%EXP_NAME_1%\cameras\30000_matrices.pkl -m "%DATA_PATH%\3d_gaussian_splatting\%EXP_NAME_1%" + +REM 运行潜在头发股线重建 +CALL activate_gaussian_splatting_hair.bat +cd %PROJECT_DIR%\src +set CUDA_VISIBLE_DEVICES=%GPU% +python train_latent_strands.py -s %DATA_PATH% -m "%DATA_PATH%\3d_gaussian_splatting\%EXP_NAME_1%" -r 1 --model_path_hair "%DATA_PATH%\strands_reconstruction\%EXP_NAME_2%" --flame_mesh_dir "%DATA_PATH%\flame_fitting\%EXP_NAME_1%" --pointcloud_path_head "%EXP_PATH_1%\point_cloud_filtered\iteration_30000\raw_point_cloud.ply" --hair_conf_path "%PROJECT_DIR%\src\arguments\hair_strands_textured.yaml" --lambda_dmask 0.1 --lambda_dorient 0.1 --lambda_dsds 0.01 --load_synthetic_rgba --load_synthetic_geom --binarize_masks --iteration_data 30000 --trainable_cameras --trainable_intrinsics --use_barf --iterations 20000 --port "800%GPU%" + +REM 运行头发股线重建 +CALL activate_gaussian_splatting_hair.bat +cd %PROJECT_DIR%\src +set CUDA_VISIBLE_DEVICES=%GPU% +python train_strands.py -s %DATA_PATH% -m "%DATA_PATH%\3d_gaussian_splatting\%EXP_NAME_1%" -r 1 --model_path_curves "%DATA_PATH%\curves_reconstruction\%EXP_NAME_3%" --flame_mesh_dir "%DATA_PATH%\flame_fitting\%EXP_NAME_1%" --pointcloud_path_head "%EXP_PATH_1%\point_cloud_filtered\iteration_30000\raw_point_cloud.ply" --start_checkpoint_hair "%DATA_PATH%\strands_reconstruction\%EXP_NAME_2%\checkpoints\20000.pth" --hair_conf_path "%PROJECT_DIR%\src\arguments\hair_strands_textured.yaml" --lambda_dmask 0.1 --lambda_dorient 0.1 --lambda_dsds 0.01 --load_synthetic_rgba --load_synthetic_geom --binarize_masks --iteration_data 30000 --position_lr_init 0.0000016 --position_lr_max_steps 10000 --trainable_cameras --trainable_intrinsics --use_barf --iterations 10000 --port "800%GPU%" + +rmdir /s /q "%DATA_PATH%\3d_gaussian_splatting\%EXP_NAME_1%\train_cropped" + +REM ################## +REM # 可视化阶段 # +REM ################## + +REM 导出结果的股线为pkl和ply格式 +CALL activate_gaussian_splatting_hair.bat +cd %PROJECT_DIR%\src\preprocessing +set CUDA_VISIBLE_DEVICES=%GPU% +python export_curves.py --data_dir %DATA_PATH% --model_name %EXP_NAME_3% --iter 10000 --flame_mesh_path "%DATA_PATH%\flame_fitting\%EXP_NAME_1%\stage_3\mesh_final.obj" --scalp_mesh_path "%DATA_PATH%\flame_fitting\%EXP_NAME_1%\scalp_data\scalp.obj" --hair_conf_path "%PROJECT_DIR%\src\arguments\hair_strands_textured.yaml" + +REM 渲染可视化 +CALL activate_gaussian_splatting_hair.bat +cd %PROJECT_DIR%\src\postprocessing +set CUDA_VISIBLE_DEVICES=%GPU% +python render_video.py --blender_path "%BLENDER_DIR%" --input_path "%DATA_PATH%" --exp_name_1 "%EXP_NAME_1%" --exp_name_3 "%EXP_NAME_3%" + +REM 渲染股线 +CALL activate_gaussian_splatting_hair.bat +cd %PROJECT_DIR%\src +set CUDA_VISIBLE_DEVICES=%GPU% +python render_strands.py -s %DATA_PATH% --data_dir "%DATA_PATH%" --data_device 'cpu' --skip_test -m "%DATA_PATH%\3d_gaussian_splatting\%EXP_NAME_1%" --iteration 30000 --flame_mesh_dir "%DATA_PATH%\flame_fitting\%EXP_NAME_1%" --model_hair_path "%DATA_PATH%\curves_reconstruction\%EXP_NAME_3%" --hair_conf_path "%PROJECT_DIR%\src\arguments\hair_strands_textured.yaml" --checkpoint_hair "%DATA_PATH%\strands_reconstruction\%EXP_NAME_2%\checkpoints\20000.pth" --checkpoint_curves "%DATA_PATH%\curves_reconstruction\%EXP_NAME_3%\checkpoints\10000.pth" --pointcloud_path_head "%EXP_PATH_1%\point_cloud\iteration_30000\raw_point_cloud.ply" --interpolate_cameras + +REM 制作视频 +CALL activate_gaussian_splatting_hair.bat +cd %PROJECT_DIR%\src\postprocessing +set CUDA_VISIBLE_DEVICES=%GPU% +python concat_video.py --input_path "%DATA_PATH%" --exp_name_3 "%EXP_NAME_3%"