aurora

外观

ESAM 部署

约 768 字大约 3 分钟

语义分割

2024-09-27

开启 docker

docker run --ipc=host -i -t -d --name ww_test ww_esam:v1.1 /bin/bash

--ipc=host，否则会报内存不够错误

RuntimeError: DataLoader worker (pid 1006435) is killed by signal: Bus error. It is possible that dataloader's workers are out of shared memory. Please try to raise your shared memory limit.

一些错误

OSError: libGL.so.1: cannot open shared object file: No such file or directory

sudo apt-get install libgl1-mesa-glx

ImportError: libgthread-2.0.so.0: cannot open shared object file: No such file or directory

sudo apt-get install libglib2.0-0

ImportError: Failed to import custom modules from {'imports': ['oneformer3d']}

  • 参考 mmdet3d 导入自定义模块的方法，在这里通过设置 PYTHONPATH 路径的方法没有效果

  • 直接复制到 ~/mmdetection3d/mmdet3d/models，在 __init__.py 中添加 from .oneformer3d import *

准备数据集

准备 3RScan

Step1: 下载 3RScan 数据集 here. 我们在包含46个场景的测试拆分上测试我们的模型。然后将下载的场景放入名为“3RScan”的文件夹中，并将场景重命名为['000', '001', ..., '045']。

或者使用作者预处理好的数据集 here，然后解压为3RScan 文件夹，可以链接到本文件夹

3RScan
├── 3RScan
│   ├── 000
│   │   ├── sequence
│   │   ├── labels.instances.annotated.v2.ply
│   │   ├── mesh.refined_0.png
│   │   ├── mesh.refined.0.010000.segs.v2.json
│   │   ├── mesh.refined.mtl
│   │   ├── mesh.refined.v2.obj
│   │   └── semseg.v2.json
│   ├── 001
│   └── ...
├── batch_load_3rscan_data.py
├── load_scannet_data.py
├── README.md
├── 3RScan.v2 Semantic Classes - Mapping.tsv
└── scannet_utils.py

Step2: 运行以下命令行

python batch_load_3rscan_data.py

然后，您将获得一个名为3rscan_instance_data的新文件夹。

Step3: 返回ESAM的根目录，并通过运行生成.pkl文件：

python tools/create_data.py 3rscan --root-path ./data/3RScan --out-dir ./data/3RScan --extra-tag 3rscan

最终文件结构：

3RScan
├── 3RScan
│   ├── 000
│   │   ├── sequence
│   │   ├── labels.instances.annotated.v2.ply
│   │   ├── mesh.refined_0.png
│   │   ├── mesh.refined.0.010000.segs.v2.json
│   │   ├── mesh.refined.mtl
│   │   ├── mesh.refined.v2.obj
│   │   └── semseg.v2.json
│   ├── 001
│   └── ...
├── instance_mask
│   ├── xxx.bin
│   └── ...
├── points
│   ├── xxx.bin
│   └── ...
├── semantic_mask
│   ├── xxx.bin
│   └── ...
├── super_points
│   ├── xxx.bin
│   └── ...
├── 3rscan_instance_data
│   ├── xxx_aligned_bbox.npy
│   ├── xxx_ins_label.npy
│   ├── xxx_sem_label.npy
│   ├── xxx_sp_label.npy
│   ├── xxx_unaligned_bbox.npy
│   ├── xxx_vert.npy
│   └── ...
├── batch_load_3rscan_data.py
├── load_scannet_data.py
├── 3rscan_oneformer3d_infos_val.pkl
├── README.md
├── 3RScan.v2 Semantic Classes - Mapping.tsv
└── scannet_utils.py

准备 3RScan-mv 和 3RScan-mv_fast

Step1:

下载 3RScan：和上面一样，可以 link 到 3RScan-mv 文件夹中

Link '3RScan' folder to this directory, namely ESAM/data/3RScan-mv. You should get the following directory structure:

3RScan-mv
├── 3RScan
│   ├── 000
│   │   ├── sequence
│   │   ├── labels.instances.annotated.v2.ply
│   │   ├── mesh.refined_0.png
│   │   ├── mesh.refined.0.010000.segs.v2.json
│   │   ├── mesh.refined.mtl
│   │   ├── mesh.refined.v2.obj
│   │   └── semseg.v2.json
│   ├── 001
│   └── ...
├── load_3rscan_mv_data.py
├── load_scannet_data.py
├── README.md
├── 3RScan.v2 Semantic Classes - Mapping.tsv
└── scannet_utils.py

Step2: Run the following commands:

python load_3rscan_mv_data.py

或者通过运行使用FastSAM而不是SAM：

PYTHONPATH='/root/FastSAM' python load_3rscan_mv_data_fast.py

如果您使用FastSAM版本，请将此文件夹从3RScan-mv重命名为3RScan-mv_fast。

然后，您将获得几个新文件夹，包括instance_mask、points、semantic_mask、super_points。

Step3: 返回ESAM的根目录，并通过运行生成.pkl文件：

python tools/create_data.py 3rscan_mv --root-path ./data/3RScan-mv --out-dir ./data/3RScan-mv --extra-tag 3rscan_mv

最终目录结构：

3RScan-mv
├── 3RScan
│   ├── 000
│   │   ├── sequence
│   │   ├── labels.instances.annotated.v2.ply
│   │   ├── mesh.refined_0.png
│   │   ├── mesh.refined.0.010000.segs.v2.json
│   │   ├── mesh.refined.mtl
│   │   ├── mesh.refined.v2.obj
│   │   └── semseg.v2.json
│   ├── 001
│   └── ...
├── instance_mask
│   ├── xxx
│   │   ├── x.bin
│   │   └── ...
│   └── ...
├── points
│   ├── xxx
│   │   ├── x.bin
│   │   └── ...
│   └── ...
├── semantic_mask
│   ├── xxx
│   │   ├── x.bin
│   │   └── ...
│   └── ...
├── super_points
│   ├── xxx
│   │   ├── x.bin
│   │   └── ...
│   └── ...
├── load_3rscan_mv_data.py
├── load_scannet_data.py
├── 3rscan_mv_oneformer3d_infos_val.pkl
├── README.md
├── 3RScan.v2 Semantic Classes - Mapping.tsv
└── scannet_utils.py

Evaluate ESAM on 3RScan-MV (Class Agnostic):

CUDA_VISIBLE_DEVICES=0 python tools/test.py configs/ESAM_CA/ESAM_online_3rscan_CA_test.py work_dirs/ESAM_online_scannet200_CA/epoch_128.pth --work-dir work_dirs/ESAM_online_3rscan_CA_test/

结果

09/30 14:19:50 - mmengine - INFO - Load checkpoint from work_dirs/ESAM_online_scannet200_CA/epoch_128.pth
/root/mmdetection3d/mmdet3d/structures/points/base_points.py:136: UserWarning: point got color value beyond [0, 255]
  warnings.warn('point got color value beyond [0, 255]')
09/30 14:25:20 - mmengine - INFO - 
+---------+---------+---------+--------+
| classes | AP_0.25 | AP_0.50 | AP     |
+---------+---------+---------+--------+
| object  | 0.5808  | 0.2936  | 0.1373 |
+---------+---------+---------+--------+
| Overall | 0.5808  | 0.2936  | 0.1373 |
+---------+---------+---------+--------+
09/30 14:25:20 - mmengine - INFO - Epoch(test) [44/44]    all_ap: 0.1373  all_ap_50%: 0.2936  all_ap_25%: 0.5808  data_time: 0.2144  time: 7.3851

Evaluate ESAM-E on 3RScan-MV (Class Agnostic):

CUDA_VISIBLE_DEVICES=0 python tools/test.py configs/ESAM-E_CA/ESAM-E_online_3rscan_CA_test.py work_dirs/ESAM-E_online_scannet200_CA/epoch_128.pth --work-dir work_dirs/ESAM-E_online_3rscan_CA_test/

结果

09/30 15:55:38 - mmengine - INFO - 
+---------+---------+---------+--------+
| classes | AP_0.25 | AP_0.50 | AP     |
+---------+---------+---------+--------+
| object  | 0.5960  | 0.2964  | 0.1356 |
+---------+---------+---------+--------+
| Overall | 0.5960  | 0.2964  | 0.1356 |
+---------+---------+---------+--------+
09/30 15:55:38 - mmengine - INFO - Epoch(test) [44/44]    all_ap: 0.1356  all_ap_50%: 0.2964  all_ap_25%: 0.5960  data_time: 0.1743  time: 7.7223

wenwei@2025