在 Jetson 上跑通 FoundationPose！从零部署 NVIDIA 6D 位姿估计系统实战全流程

介绍

FoundationPose 是 NVIDIA Research（NVLabs）推出的开源 6D 位姿估计与跟踪系统，它能够在无需微调的情况下对从未见过的新物体进行精确的三维姿态识别与实时跟踪。该项目融合了神经隐式表示、Transformer 架构和对比学习等前沿技术，既支持基于模型（有 CAD 模型）也支持无模型（仅参考图像）的场景，广泛适用于机器人抓取、AR/VR 对齐以及具身智能等任务，展现了通用物体 6D 感知的基础模型潜力。

硬件+环境

• Jetson orin nx 16g(reComputer Robotics)
• Jetpack 6.2
• CUDA 12.6
• python3.10

项目部署

1. 克隆项目

git clone https://github.com/NVlabs/FoundationPose.git
cd FoundationPose

2. 安装系统依赖

sudo apt install -y libopenblas-base libopenmpi-dev libjpeg-dev zlib1g-dev

3. 创建conda环境

conda create -n foundationpose python=3.10
conda activate foundationpose

4. 配置CUDA环境变量

在~/.bashrc的最后行，添加自己CUDA的安装路径：

export PATH=/usr/local/cuda/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH
export CUDA_HOME=/usr/local/cuda
export PATH=$CUDA_HOME/bin:$PATH
export LD_LIBRARY_PATH=$CUDA_HOME/lib64:$LD_LIBRARY_PATH
export CUB_HOME=/usr/local/cuda/include/cub

5. 安装GCC/G++

#安装11版本的gcc和g++
sudo apt-get update && sudo apt-get install -y gcc-11 g++-11
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-11 100
sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-11 100
#安装GLIBCXX_3.4.30
conda install -c conda-forge libstdcxx-ng

6. 配置GPU架构

#首先检查你的 GPU 架构
nvidia-smi --query-gpu=compute_cap --format=csv,noheader

# 下面的这两个命令可以写入~/.bashrc里面，最后source ~/.bashrc
export TORCH_CUDA_ARCH_LIST="8.7" # 我的是8.7
# 强制开启 OpenCV 对 OpenEXR 图像格式的读写支持
export OPENCV_IO_ENABLE_OPENEXR=1

7. 安装PyTorch

#安装pytorch
wget https://nvidia.box.com/shared/static/zvultzsmd4iuheykxy17s4l2n91ylpl8.whl -O ~/Downloads/torch-2.3.0-cp310-cp310-linux_aarch64.whl
wget https://nvidia.box.com/shared/static/xpr06qe6ql3l6rj22cu3c45tz1wzi36p.whl -O ~/Downloads/torchvision-0.18.0a0+6043bc2-cp310-cp310-linux_aarch64.whl
wget https://nvidia.box.com/shared/static/9si945yrzesspmg9up4ys380lqxjylc3.whl -O ~/Downloads/torchaudio-2.3.0+952ea74-cp310-cp310-linux_aarch64.whl
pip install ~/Downloads/torch-2.3.0-cp310-cp310-linux_aarch64.whl ~/Downloads/torchvision-0.18.0a0+6043bc2-cp310-cp310-linux_aarch64.whl ~/Downloads/torchaudio-2.3.0+952ea74-cp310-cp310-linux_aarch64.whl

8. 安装PyTorch3D

# 注意安装pytorch3d库之类的时候指定最大的核心数不然会导致内存不够，安装完成后再改回去
export MAX_JOBS=4
#安装pytorch3d
git clone https://github.com/facebookresearch/pytorch3d.git
cd pytorch3d
pip install numpy==1.26.0
pip install -e .

这个过程很漫长，请耐心等待！

9. 安装FoundationPose依赖库

#安装Foundationpose所需依赖库
python -m pip install scipy joblib scikit-learn ruamel.yaml trimesh pyyaml opencv-python imageio open3d transformations warp-lang einops kornia pyrender -i https://pypi.tuna.tsinghua.edu.cn/simple

pip install gdown -i https://pypi.tuna.tsinghua.edu.cn/simple

10. 下载模型和数据

国内网络大概下载不下来，我们直接下载到本地：

https://drive.google.com/drive/folders/1BEQLZH69UO5EOfah-K9bfI3JyP9Hf7wC

https://drive.google.com/drive/folders/12Te_3TELLes5cim1d7F7EBTwUSe7iRBj

创建以下这两个目录下载完后把模型文件放进去：

FoundationPose/weights/2023-10-28-18-33-37 FoundationPose/weights/2024-01-11-20-02-45

下载数据：

mkdir -p demo_data 

https://drive.google.com/drive/folders/1pRyFmxYXmAnpku7nGRioZaKrVJtIsroP

下载下来然后解压到demo_data 文件夹下

11. 安装pybind11

#安装这个是使C++与python相互调用：
cd FoundationPose 
git clone https://github.com/pybind/pybind11
cd pybind11
git checkout v2.10.0
mkdir build && cd build 
cmake .. \
  -DCMAKE_BUILD_TYPE=Release \
  -DPYBIND11_INSTALL=ON \
  -DPYBIND11_TEST=OFF \
-DPYTHON_EXECUTABLE=/home/seeed/miniconda3/envs/foundationpose/bin/python
make -j6
sudo make install

12. 安装Eigen库

#安装Eigen库
cd $HOME && wget -q https://gitlab.com/libeigen/eigen/-/archive/3.4.0/eigen-3.4.0.tar.gz && \
tar -xzf eigen-3.4.0.tar.gz && \
cd eigen-3.4.0 && mkdir build && cd build
cmake .. -Wno-dev -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_FLAGS=-std=c++14 ..
sudo make install
cd $HOME && rm -rf eigen-3.4.0 eigen-3.4.0.tar.gz

13. 安装nvdiffrast

#安装nvdiffrast
#这个也只能从源码安装，如果按照官方给的那个方式安装会出现很多问题（试过很多次），所以这个在jetson里安装：
cd ~/FoundationPose 
git clone https://github.com/NVlabs/nvdiffrast 
cd nvdiffrast && pip install .
#安装mycpp
cd ~/FoundationPose/mycpp
rm -rf build && mkdir -p build && cd build && \
cmake .. && \
make -j$(nproc)

14. 修改requirements.txt

cp requirements.txt rerequirements_copy.txt
vim rerequirements_copy.txt

将rerequirements_copy.txt里原来的所有东西清除，再将下面的内容复制到rerequirements_copy.txt的里：

aiohappyeyeballs==2.4.4
aiohttp==3.10.10
aiosignal==1.3.2
albucore==0.0.17
albumentations==1.4.18
annotated-types==0.7.0
antlr4-python3-runtime==4.9.3
anyio==4.7.0
async-timeout==4.0.3
attrs==24.3.0
beautifulsoup4==4.12.3
blinker==1.9.0
certifi==2025.1.31
charset-normalizer==3.4.1
click==8.1.8
configargparse==1.7
contourpy==1.3.1
cycler==0.12.1
cython==3.0.10
dash==2.18.2
dash-core-components==2.0.0
dash-html-components==2.0.0
dash-table==5.0.0
distro==1.9.0
einops==0.8.0
eval-type-backport==0.2.2
exceptiongroup==1.2.2
fastjsonschema==2.21.1
filelock==3.17.0
flask==3.0.3
fonttools==4.55.3
freetype-py==2.5.1
frozenlist==1.5.0
fsspec==2024.12.0
gdown==5.2.0
glfw==2.7.0
h11==0.14.0
h5py==3.12.1
hickle==5.0.3
httpcore==1.0.7
httpx==0.28.1
idna==3.10
ifaddr==0.2.0
imageio==2.35.1
importlib-metadata==8.6.1
iniconfig==2.0.0
itsdangerous==2.2.0
jinja2==3.1.5
jiter==0.8.2
joblib==1.4.2
jsonschema==4.23.0
jsonschema-specifications==2024.10.1
jupyter-core==5.7.2
kiwisolver==1.4.8
kornia==0.5.10
lazy-loader==0.4
lightning-utilities==0.11.9
loguru==0.7.2
markupsafe==3.0.2
matplotlib==3.9.2
mouseinfo==0.1.3
mpmath==1.3.0
multidict==6.1.0
nbformat==5.10.4
nest-asyncio==1.6.0
netifaces==0.11.0
networkx==3.4.2
ninja==1.11.1.3
numpy==1.26.4
nvdiffrast==0.3.3
omegaconf==2.3.0
open3d==0.18.0
openai==1.51.2
opencv-contrib-python==4.10.0.84
opencv-python==4.10.0.82
opencv-python-headless==4.10.0.84
packaging==24.2
pandas==2.2.3
pillow==11.1.0
platformdirs==4.3.6
plotly==5.24.1
pluggy==1.5.0
portalocker==3.1.1
progressbar33==2.4
propcache==0.2.1
psutil==6.1.1
py-cpuinfo==9.0.0
pyautogui==0.9.54
pydantic==2.10.4
pydantic-core==2.27.2
pygetwindow==0.0.9
pyglet==2.0.20
pymsgbox==1.0.9
pyopengl==3.1.0
pyparsing==3.2.1
pyperclip==1.9.0
pyrect==0.2.0
pyrender==0.1.45
pyscreeze==1.0.1
pyside6==6.8.0.1
pyside6-addons==6.8.0.1
pyside6-essentials==6.8.0.1
pysocks==1.7.1
pytest==8.2.2
python-dateutil==2.9.0.post0
python3-xlib==0.15
pytinyrenderer==0.0.14
pytorch-lightning==2.4.0
pytorch3d==0.7.8
pytweening==1.2.0
pytz==2024.2
pyyaml==6.0.2
referencing==0.35.1
requests==2.32.3
retrying==1.3.4
rpds-py==0.22.3
ruamel-base==1.0.0
ruamel-yaml==0.18.8
ruamel-yaml-clib==0.2.12
ruptures==1.1.9
scikit-image==0.25.0
scikit-learn==1.5.2
scipy==1.14.1
seaborn==0.13.2
setuptools==72.1.0
shiboken6==6.8.0.1
six==1.17.0
sniffio==1.3.1
soupsieve==2.6
sympy==1.13.3
tenacity==9.0.0
threadpoolctl==3.5.0
tifffile==2024.12.12
torch==2.3.0
torchaudio==2.3.0+952ea74
torchmetrics==1.6.1
torchvision==0.18.0a0+6043bc2
tqdm==4.66.5
traitlets==5.14.3
transformations==2025.1.1
trimesh==4.4.7
typing-extensions==4.12.2
tzdata==2024.2
ultralytics==8.3.31
ultralytics-thop==2.0.13
urllib3==2.3.0
warp==1.0.4
warp-lang==1.3.1
websocket-client==1.8.0
websockets==14.2
werkzeug==3.0.6
yacs==0.1.8
yarl==1.18.3
zeroconf==0.143.0
zipp==3.21.0

安装：

python -m pip install -r requirements_copy.txt

15. 构建和安装扩展

## Build and install extensions in repo
#指定自己的路径！
CMAKE_PREFIX_PATH=$CONDA_PREFIX/lib/python3.10/site-packages/pybind11/share/cmake/pybind11

bash build_all_conda.sh

运行Demo

1. 安装额外依赖

pip install psutil
pip install pandas
pip install matplotlib omegaconf h5py
pip install numpy==1.26.0

2. 下载演示数据

mkdir -p demo_data && cd demo_data && gdown --folder https://drive.google.com/drive/folders/1pRyFmxYXmAnpku7nGRioZaKrVJtIsroP -O ./ --remaining-ok

3. 运行演示

export LD_LIBRARY_PATH=$CONDA_PREFIX/lib/python3.10/site-packages/torch/lib:$LD_LIBRARY_PATH && python run_demo.py --mesh_file demo_data/kinect_driller_seq/mesh/textured_mesh.obj --test_scene_dir demo_data/kinect_driller_seq

export LD_LIBRARY_PATH=$CONDA_PREFIX/lib/python3.10/site-packages/torch/lib:$LD_LIBRARY_PATH && python run_demo.py --mesh_file demo_data/kinect_driller_seq/mesh/textured_mesh.obj --test_scene_dir demo_data/kinect_driller_seq --est_refine_iter 3 --track_refine_iter 1

python run_demo.py

性能优化

运行可能爆内存，可以减少一下候选框和迭代次数：

打开/FoundationPose/estimater.py，将27行的内容由：

self.make_rotation_grid(min_n_views=40, inplane_step=60) 

改为：

self.make_rotation_grid(min_n_views=20, inplane_step=90) 

python run_demo.py

运行结果：

这个是官方的运行视频，在jetson上跑的帧率有点低，可能是因为用python推理效率有点低！目前官方有ros c++推理的版本，但是在Jetson Thor上部署的。还不确定能否迁移到orin nx上！

参考链接

• FoundationPose GitHub
• 参考安装教程