​作者​：昇腾实战派
​知识地图​：https://blog.csdn.net/Lumos_Lovegood/article/details/161601003

背景概述

在分布式训练场景中，通信算子的正确性与稳定性直接影响模型训练的收敛效果与运行效率。本文记录了在昇腾NPU平台上使用PyTorch适配层（PTA）进行分布式训练过程中遇到的若干典型问题，涵盖通信接口的内存复用、子通信域销毁、算子执行异常、精度偏差及进程同步等多个方面。以下为各问题的详细分析与解决方案。

---

问题1：batch_isend_irecv 接口出现 NaN 值, 切换到 P2P（point-to-point）接口后问题消失

版本信息：PTA 2.6.0.post5

问题描述：
ProcessGroupHCCL::batch_isend_irecv_inner 接口中，仅将 tensors[0] 传递给 collective 操作。collective 在内部仅对 tensors[0] 执行 recordStream 保护，而 batch_isend_irecv 可能包含多个 tensor（每个 op_type 对应一个）。其余 tensor 未受到 recordStream 保护，在多流并行场景下可能导致内存复用问题，进而产生 NaN 值。

问题分类：功能问题

根因分析：
该问题属于异步多流并行优化导致的内存复用问题。具体而言，batch_isend_irecv_inner 只将 tensors[0] 传入 collective，collective 在循环遍历 inputs 执行 recordStream 时，实际只有 tensors[0] 被保护。tensors[1..n] 参与了 HCCL DMA 操作，但 allocator 并不知道 HCCL 流正在使用这些 tensor，因此在多流并行场景下，这些 tensor 被计算流覆盖，导致 NaN 值出现。

解决方案：
新接口已定位并修复该问题，对于无法适配新版 CANN 包的场景，已有修复 patch 适配至 PTA 2.9 版本中。

---

问题2：FSDP2 场景下 reduce scatter 后梯度出现 NaN

版本信息：PTA 2.8.0

问题描述：
部分模型训练中，loss 出现 NaN。定位发现 FSDP2 代码中，经过 reduce scatter 操作后梯度出现 NaN。

问题分类：精度问题

根因分析：
问题根因在于业务侧 accelerator 处，计算流未同步调用 CPU 算子，导致 reduce scatter 前后梯度异常。

解决方案：
通过增加流同步操作解决该问题。

---

问题3：FSDP2 反向通信死锁——collective 不匹配

报错信息：

File "/home/miniconda3/envs/nwulan/lib/python3.11/site-packages/torch/autograd/function.py", line 307, in apply
    return user_fn(self, *args)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
File "/home/miniconda3/envs/nwulan/lib/python3.11/site-packages/torch/distributed/fsdp/_fully_shard/_fsdp_param_group.py", line 747, in backward
    ctx.param_group.post_backward()
File "/home/miniconda3/envs/nwulan/lib/python3.11/site-packages/torch_npu/distributed/fsdp_add_fsdp_patch.py", line 513, in wrapper
    original_post_backward(self)
File "/home/miniconda3/envs/nwulan/lib/python3.11/site-packages/torch/distributed/fsdp/_fully_shard/_fsdp_param_group.py", line 442, in post_backward
    ) = foreach_reduce(
File "/home/miniconda3/envs/nwulan/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
File "/home/miniconda3/envs/nwulan/lib/python3.11/site-packages/torch_npu/distributed/fsdp_add_fsdp_patch.py", line 385, in _patched_foreach_reduce
    dist.reduce_scatter_tensor(
File "/home/miniconda3/envs/nwulan/lib/python3.11/site-packages/torch/distributed/c10d_logger.py", line 81, in wrapper
    return func(*args, **kwargs)
File "/home/miniconda3/envs/nwulan/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py", line 4286, in reduce_scatter_tensor
    work = group._reduce_scatter_base(output, input, opts)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

RuntimeError: Detected mismatch between collectives on ranks. Rank 0 is running collective: CollectiveFingerprint(SequenceNumber=12, OpType=_REDUCE_SCATTER_BASE, TensorShape=[56623104], TensorDtypes=Float Float, TensorDeviceTypes=TensorOptions(dtype=float (default), device=npu, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullptr)), but Rank 1 is running collective: CollectiveFingerprint(SequenceNumber=0OpType=GATHER).Collectives differ in the following aspects: Sequence number: 12vs 0 Op type: REDUCE SCATTER BASEvs GATHER Tensor Tensor shapes: 56623104vs Tensor Tensor dtypes: Float Floatsvs Tensor Tensor devices: TensorOptions(dtype=float (default), device=npu, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullptr))vs TensorOptions(dtype=float (default), device=npu, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullptr))vs Detected mismatch between collectives on ranks. Rank 0 is running collective: CollectiveFingerprint(SequenceNumber=12, OpType=_REDUCE_SCATTER_BASE, TensorShape=[56623104], TensorDtypes=Float Float, TensorDeviceTypes=TensorOptions(dtype=float (default), device=npu, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullptr))), but Rank 1 is running collective: CollectiveFingerprint(SequenceNumber=0OpType=GATHER).Collectives differ in the following aspects: Sequence number: 12vs 0 Op type: REDUCE SCATTER BASEvs GATHER Tensor Tensor shapes: 56623104vs Tensor Tensor dtypes: Float Floatsvs Tensor Tensor devices: TensorOptions(dtype=float (default), device=npu, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullptr))vs TensorOptions(dtype=float (default), device=npu, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullptr))vs Traceback (most recent call last):

版本信息：PTA 2.7.1

问题描述：
FSDP2 反向运行中，两个 rank 上执行的 layer 对象不一致，导致 reduce scatter 通信前的 input shape 不同，无法通信，造成死锁。错误信息显示 Rank 0 执行 _REDUCE_SCATTER_BASE，而 Rank 1 执行 GATHER，序列号与操作类型均不匹配。

问题分类：代码问题

根因分析：
Rank 1 上使用的 MXFP8Tensor 默认 device 为 0，导致 allgather 后的 unsharded param device 为 0，使得 rank 1 上的 param 与 grad 的 device 不一致，赋值累积等操作出现异常。

解决方案：
修复代码逻辑，确保各 rank 上 tensor 的 device 一致。

---

问题4：DeviceMesh.get_group API 功能报错

版本信息：PTA 2.7.1

问题描述：
在 CANN 9.0.0.B120 + FrameworkPTAdapter 26.0.0.B113 环境下，调用 torch.distributed.device_mesh.DeviceMesh.get_group API 出现功能报错。

问题分类：脚本问题

根因分析：
问题由 from torch_npu.contrib import transfer_to_npu 导入引起。

解决方案：
删除该导入语句后，API 可正常运行。

---

问题5：verl 训练超时

问题描述：
运行 verl qwen 训练任务超时。

问题分类：配置问题

根因分析：
未配置环境变量 HCCL_INTRA_ROCE_ENABLE=1。

解决方案：
配置该环境变量后问题解决。

问题6：profiling 多线程调用导致 setdevice 异常

---

报错信息：

runtime_assert:b############################ 19 ############################
runtime_assert:#0 /usr/local/Ascend/cann-9.0.0/lib64/libruntime.so(+0x1e0c0) [0xffff672360c0]
runtime_assert:#1 /usr/local/Ascend/cann-9.0.0/lib64/libruntime.so(rtProfSetProSwitch+0x3c) [0xffff6724e47c]
runtime_assert:#2 /usr/local/Ascend/cann-9.0.0/lib64/libprofimpl.so(_ZN8Analysis4Dvvp14ProfilerCommon19ProfModuleReprotMgr17ProfSetProCommandER19MsprofCommandHandle+0x3b8) [0xffff2b3c3894]
runtime_assert:#3 /usr/local/Ascend/cann-9.0.0/lib64/libprofimpl.so(_ZN8Analysis4Dvvp14ProfilerCommon19ProfModuleReprotMgr17ModuleReportStartEPKjmm+0x1a0) [0xffff2b3c3cd4]
runtime_assert:#4 /usr/local/Ascend/cann-9.0.0/lib64/libprofimpl.so(_ZN10Msprofiler3Api10ProfAcIMgr15ProfStartCommonEPKjj+0x2d0) [0xffff2b3b35c8]
runtime_assert:#5 /usr/local/Ascend/cann-9.0.0/lib64/libprofimpl.so(_ZN8Analysis4Dvvp14ProfilerCommon15ProfConfigStartEjPKvj+0xf8) [0xffff2b3c4c00]
runtime_assert:#6 /usr/local/Ascend/cann-9.0.0/lib64/libprofimpl.so(_ZN8Analysis4Dvvp14ProfilerCommon19ProfNotifySetDeviceEjjb+0x1fc) [0xffff2b3c5d48]
runtime_assert:#7 /usr/local/Ascend/cann-9.0.0/lib64/libprofapi.so(+0xeb4c) [0xffff698c4b4c]
runtime_assert:#8 /usr/local/Ascend/cann-9.0.0/lib64/libruntime_v100.so(+0x34d710) [0xffff5bb85710]
runtime_assert:#9 /usr/local/Ascend/cann-9.0.0/lib64/libruntime_v100.so(+0x6c594) [0xffff5b8a4594]
runtime_assert:#10 /usr/local/Ascend/cann-9.0.0/lib64/libruntime_v100.so(+0xed060) [0xffff5b925060]
runtime_assert:#11 /usr/local/Ascend/cann-9.0.0/lib64/libruntime.so(rtSetDevice+0x5c) [0xffff6725a09c]
runtime_assert:#12 /usr/local/Ascend/cann-9.0.0/lib64/lacl_rt_impl.so(aclrtSetDeviceImpl+0x68) [0xffff621bb468]
runtime_assert:#13 /mnt/sda1/anaconda3/envs/tools_test_py312/lib/python3.12/site-packages/torch_npu/lib/libtorch_npu.so(_ZN7c10_npu9SetDeviceEa+0x8c) [0xffff6c4c2fd0]
runtime_assert:#14 /mnt/sda1/anaconda3/envs/tools_test_py312/lib/python3.12/site-packages/torch_npu/lib/libtorch_npu.so(_ZN8c10d_npu16ProcessGroupHCCL8Watchdog7runLoopEv+0x390) [0xffff6edec010]
runtime_assert:#15 /mnt/sda1/anaconda3/envs/tools_test_py312/lib/python3.12/site-packages/torch_npu/lib/libtorch_npu.so(_ZN8c10d_npu16ProcessGroupHCCL8Watchdog3runEv+0x124) [0xffff6ededc34]
runtime_assert:#16 /usr/local/Ascend/cann-9.0.0/lib64/libhcc1.so(+0x6f1bac) [0xffff6b79abac]
runtime_assert:#17 /usr/lib64/libc.so.6(+0x7f1f0) [0xffff8de8e1f0]
runtime_assert:#18 /usr/lib64/libc.so.6(+0xe571c) [0xffff8def471c]

版本信息：PTA 2.7.1.post4.dev20260315

问题描述：
在 Atlas 800I A2/ Ascend 950 环境下，PTA 接口调用 profiling 时，多线程调用场景下出现线程异常 setdevice。错误栈显示 watchdog 线程在 aclFinalizeImpl 启动后执行了 SetDevice。

问题分类：线程安全问题

根因分析：
两个进程在退出阶段踩到了不同的线程时序窗口。watchdog 是后台线程循环，在 aclFinalizeImpl 启动后，watchdog 中执行了 SetDevice。当前 _destructor_process_group() 只清理 Python 全局状态，清空 Python map 不能保证 C++ ProcessGroupHCCL 立即析构，也不能保证 watchdog 已经 join。因此在用户未显式调用 destroy_process_group() 的场景下，watchdog 可能存活到 profiling stop/finalize 之后，再执行 SetDevice。

解决方案：
建议在退出前显式调用 destroy_process_group()，确保 watchdog 线程正确退出。

---

问题7：多卡通信算子超时——进程退出时序问题

版本信息：PTA 2.7.1 python310

问题描述：
Ascend 950 Pod 环境上，8 卡 torch 调用通信算子长时间运行，规定时间跑完通信算子后，部分卡未结束进程，导致超时。

问题分类：脚本问题

根因分析：
在规定时间内先执行完的卡提前退出，导致其他卡多执行了一次通信，这些卡一直在等待退出的卡，造成超时。

解决方案：
修改循环退出条件为通信次数，或在通信前先校验是否有卡退出。

问题8：HCCL通信域初始化失败（Ranktable检测超时）

报错信息：

File "/home/verl_fsdp_turbo/verl/workers/engine_workers.py", line 175, in reset
    self.engine.initialize()
  File "/home/verl_fsdp_turbo/verl/workers/engine/fsdp/transformer_impl.py", line 176, in initialize
    self._build_model_optimizer()
  File "/home/verl_fsdp_turbo/verl/workers/engine/mindspeed/transformer_impl.py", line 248, in _build_model_optimizer
    module = FSDPTurbo(self.fsdp_turbo_config, module)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/FSDPTurbo/fsdp_turbo/fsdp_turbo.py", line 22, in __init__
    self.apply_ep_modules()
  File "/home/FSDPTurbo/fsdp_turbo/fsdp_turbo.py", line 37, in apply_ep_modules
    self.model = expert_parallelize_modules(self.model, self.parallel_state.get_ep_device_mesh(), self.config.distributed.ep_plan)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/FSDPTurbo/fsdp_turbo/distributed/expert_parallel/expert_parallel.py", line 42, in expert_parallelize_modules
    distribute_experts_module(module, ep_mesh)
  File "/home/FSDPTurbo/fsdp_turbo/distributed/expert_parallel/expert_parallel.py", line 85, in distribute_experts_module
    return distribute_module(module=module, device_mesh=ep_mesh, partition_fn=distribute_expert_weight,)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/python3.11.10/lib/python3.11/site-packages/torch/distributed/tensor/_api.py", line 931, in distribute_module
    partition_fn(name, submod, device_mesh)
  File "/home/FSDPTurbo/fsdp_turbo/distributed/expert_parallel/expert_parallel.py", line 77, in distribute_expert_weight
    dist_param = torch.nn.Parameter(distribute_tensor(param, ep_mesh, [Shard(0)]))
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/python3.11.10/lib/python3.11/site-packages/torch/distributed/tensor/_api.py", line 765, in distribute_tensor
    local_tensor = placement._shard_tensor(
                   ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/python3.11.10/lib/python3.11/site-packages/torch/distributed/tensor/placement_types.py", line 183, in _shard_tensor
    mesh_scatter(
  File "/usr/local/python3.11.10/lib/python3.11/site-packages/torch/distributed/tensor/_collective_utils.py", line 108, in mesh_scatter
    fut = scatter(
          ^^^^^^^^
  File "/usr/local/python3.11.10/lib/python3.11/site-packages/torch/distributed/c10d_logger.py", line 81, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/python3.11.10/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py", line 4369, in scatter
    work = group.scatter(output_tensors, input_tensors, opts)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: create_config:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:130 HCCL function error: hcclCommInitRootInfoConfig(numRanks, &rootInfo, rank, config, &(comm->hcclComm_)), error code is 1
[ERROR] 2026-05-22-12:08:27 (PID:575229, Device:0, RankID:0) ERR02200 DIST call hccl api failed.
[PID: 575229] 2026-05-22-12:08:27.088.244 Ranktable_Detect_Failed(EI0015): Failed to collect cluster information of the communicator based on rootInfo detection. Reason: Within the timeout period, all ranks in the communication domain failed to connect to the server..
        Solution: 1. Check whether all ranks in the communicator have delivered the communicator creation interface. 2. Check the connectivity between the host networks of all nodes and the server node. 3. Check whether the HCCL_SOCKET_IFNAME environment variable of all nodes is correctly configured. 4. Increase the timeout by configuring the HCCL_CONNECT_TIMEOUT environment variable.
        TraceBack (most recent call last):
        Failed to collect cluster information of the communicator based on rootInfo detection. Reason: rank num[8]is different with rank list size[1] in total topo rank info..

版本信息：PTA 2.9.0

问题描述：在分布式训练启动过程中，调用 scatter 操作时出现以下错误：

RuntimeError: create_config:... HCCL function error: hcclCommInitRootInfoConfig(...), error code is 1
[ERROR] ERR02200 DIST call hccl api failed.
[PID: 575229] Ranktable_Detect_Failed(EI0015): Failed to collect cluster information...

错误提示表明，在超时时间内，通信域中的所有Rank未能成功连接到服务器，且检测到的Rank数量（8）与Rank列表大小（1）不一致。

结论：该问题属于环境配置或网络连通性问题。建议用户检查以下内容：

1. 所有Rank是否正确调用了通信域创建接口。
2. 各节点之间的主机网络与服务器节点的连通性。
3. HCCL_SOCKET_IFNAME 环境变量是否正确配置。
4. 可通过配置 HCCL_CONNECT_TIMEOUT 环境变量增加超时时间。

---

问题9：训练结束后HCCL Watchdog线程异常退出（路径失效）

报错信息：

clip=100, train_wall=25.5908, fetch_data=0.0002, cuda_gb_active=59, cuda_gb_allocated=59, cuda_gb_reserved=59.5, cuda_gb_free=2, wall=51.1474
[2026-05-27 19:49:13,414][train_inner][INFO] - epoch 001: 3 / 127485 loss=12.388, ce_loss=12.3729, zloss=151.206, ppl=5304.07, wps=9931.1, ups=0.04, wpb=262144, bsz=64, num_updates=3, lr=6e-07, gnorm=12.293, clip=100, train_wall=26.3889, fetch_data=0.0007, cuda_gb_active=59, cuda_gb_allocated=59, cuda_gb_reserved=59.6, cuda_gb_free=2, wall=77.5745
[2026-05-27 19:49:39,258][train_inner][INFO] - epoch 001: 4 / 127485 loss=12.3652, ce_loss=12.3501, zloss=151.223, ppl=5220.94, wps=10155.6, ups=0.04, wpb=262144, bsz=64, num_updates=4, lr=8e-07, gnorm=11.553, clip=100, train_wall=25.8055, fetch_data=0.0006, cuda_gb_active=59, cuda_gb_allocated=59, cuda_gb_reserved=59.3, cuda_gb_free=2, wall=103.452
[2026-05-27 19:50:05,615][train_inner][INFO] - epoch 001: 5 / 127485 loss=12.3829, ce_loss=12.3678, zloss=151.221, ppl=5285.34, wps=9958.1, ups=0.04, wpb=262144, bsz=64, num_updates=5, lr=1e-06, gnorm=11.621, clip=100, train_wall=26.3174, fetch_data=0.0007, cuda_gb_active=59, cuda_gb_allocated=59, cuda_gb_reserved=59.5, cuda_gb_free=2, wall=129.807
[2026-05-27 19:50:31,496][train_inner][INFO] - epoch 001: 6 / 127485 loss=12.3491, ce_loss=12.3339, zloss=151.228, ppl=5162.81, wps=10148.2, ups=0.04, wpb=262144, bsz=64, num_updates=6, lr=1.2e-06, gnorm=11.414, clip=100, train_wall=25.8242, fetch_data=0.0003, cuda_gb_active=59, cuda_gb_allocated=59, cuda_gb_reserved=59.5, cuda_gb_free=2, wall=155.668
[2026-05-27 19:50:57,244][train_inner][INFO] - epoch 001: 7 / 127485 loss=12.339, ce_loss=12.3239, zloss=151.206, ppl=5127.1, wps=10193.1, ups=0.04, wpb=262144, bsz=64, num_updates=7, lr=1.4e-06, gnorm=11.467, clip=100, train_wall=25.7106, fetch_data=0.0003, cuda_gb_active=59, cuda_gb_allocated=59, cuda_gb_reserved=59.4, cuda_gb_free=2, wall=181.436
[2026-05-27 19:51:23,155][train_inner][INFO] - epoch 001: 8 / 127485 loss=12.2318, ce_loss=12.2167, zloss=151.201, ppl=4759.79, wps=10129.5, ups=0.04, wpb=262144, bsz=64, num_updates=8, lr=1.6e-06, gnorm=12.426, clip=100, train_wall=25.872, fetch_data=0.0003, cuda_gb_active=59, cuda_gb_allocated=59, cuda_gb_reserved=59.5, cuda_gb_free=2, wall=207.348
[2026-05-27 19:51:48,686][train_inner][INFO] - epoch 001: 9 / 127485 loss=12.1868, ce_loss=12.1716, zloss=151.195, ppl=4613.47, wps=10279.7, ups=0.04, wpb=262144, bsz=64, num_updates=9, lr=1.8e-06, gnorm=11.923, clip=100, train_wall=25.4932, fetch_data=0.0007, cuda_gb_active=59, cuda_gb_allocated=59, cuda_gb_reserved=59.3, cuda_gb_free=2, wall=232.881
[2026-05-27 19:52:14,626][train_inner][INFO] - epoch 001: 10 / 127485 loss=12.1536, ce_loss=12.1385, zloss=151.195, ppl=4508.76, wps=10142.3, ups=0.04, wpb=262144, bsz=64, num_updates=10, lr=2e-06, gnorm=12.063, clip=100, train_wall=25.8393, fetch_data=0.0003, cuda_gb_active=59, cuda_gb_allocated=59, cuda_gb_reserved=59.5, cuda_gb_free=2, wall=258.758
[2026-05-27 19:52:14,630][hulk.train][INFO] - Stopping training due to num_updates: 10 >= max_update: 10
[2026-05-27 19:52:14,630][hulk.train][INFO] - before do empty cache
[2026-05-27 19:52:14,631][hulk.train][INFO] - cuda_gb_active=52.79613161087036, cuda_gb_reserved=59.078125
[2026-05-27 19:52:15,057][hulk.train][INFO] - after do empty cache
[2026-05-27 19:52:15,082][hulk.train][INFO] - cuda_gb_active=52.79613161087036, cuda_gb_reserved=59.078125
[2026-05-27 19:52:34,306][hulk.utils.checkpoint_utils][INFO] - Preparing to save checkpoint for epoch 1 @ part_idx 0 @ 10 updates
[2026-05-27 19:52:34,307][hulk.trainer][INFO] - Saving checkpoint to /train28/bitbrain/permanent/gblu2/version_release/v0.8.0-rc1/7b/output/test-0527-cann8.2.rc1-torch2.4-testenv/checkpoints/checkpoint-last/shard-0-0.pt
[2026-05-27 19:52:34,331][hulk.utils.state_dict][INFO] - replaced model state by fp32 param replicas!
[2026-05-27 19:52:34,354][hulk.utils.checkpoint_utils][INFO] - torch_persistent_save filenames:/train28/bitbrain/permanent/gblu2/version_release/v0.8.0-rc1/7b/output/test-0527-cann8.2.rc1-torch2.4-testenv/checkpoints/checkpoint-last/shard-0-0.pt, mindio_enable_acp=False
[default5]:[E527 19:52:41.491119810 compiler_depend.ts:1047] [Rank 0] HCCL watchdog thread terminated with exception: Open shared directory failed. Please check whether input path is valid.
[default5]:[ERROR] 2026-05-27-19:52:41 (PID:227, Device:5, RankID:5) ERR02008
[default2]:[E527 19:52:42.485388340 compiler_depend.ts:1047] [Rank 0] HCCL watchdog thread terminated with exception: Open shared directory failed. Please check whether input path is valid.
[default2]:[ERROR] 2026-05-27-19:52:42 (PID:224, Device:2, RankID:2) ERR02008
[default7]:[E527 19:52:42.488107818 compiler_depend.ts:1047] [Rank 0] HCCL watchdog thread terminated with exception: Open shared directory failed. Please check whether input path is valid.
[default7]:[ERROR] 2026-05-27-19:52:42 (PID:229, Device:7, RankID:7) ERR02008
[default4]:[E527 19:52:42.488646713 compiler_depend.ts:1047] [Rank 0] HCCL watchdog thread terminated with exception: Open shared directory failed. Please check whether input path is valid.
[default4]:[ERROR] 2026-05-27-19:52:42 (PID:226, Device:4, RankID:4) ERR02008
[default1]:[E527 19:52:42.509011422 compiler_depend.ts:1047] [Rank 0] HCCL watchdog thread terminated with exception: Open shared directory failed. Please check whether input path is valid.
[default1]:[ERROR] 2026-05-27-19:52:42 (PID:223, Device:1, RankID:1) ERR02008
[default3]:[E527 19:52:42.484977386 compiler_depend.ts:1047] [Rank 0] HCCL watchdog thread terminated with exception: Open shared directory failed. Please check whether input path is valid.
[default3]:[ERROR] 2026-05-27-19:52:42 (PID:225, Device:3, RankID:3) ERR02008
[default6]:[E527 19:52:42.486642293 compiler_depend.ts:1047] [Rank 0] HCCL watchdog thread terminated with exception: Open shared directory failed. Please check whether input path is valid.
[default6]:[ERROR] 2026-05-27-19:52:42 (PID:228, Device:6, RankID:6) ERR02008
[default6]:[E527 19:52:42.519269937 compiler_depend.ts:1047] [Rank 6] HCCL watchdog thread terminated with exception: Open shared directory failed. Please check whether input path is valid.
[default6]:[ERROR] 2026-05-27-19:52:42 (PID:228, Device:6, RankID:6) ERR02008
[default3]:[E527 19:52:42.666368115 compiler_depend.ts:1047] [Rank 3] HCCL watchdog thread terminated with exception: Open shared directory failed. Please check whether input path is valid.
[default3]:[ERROR] 2026-05-27-19:52:42 (PID:225, Device:3, RankID:3) ERR02008
[default7]:[E527 19:52:42.768249249 compiler_depend.ts:1047] [Rank 7] HCCL watchdog thread terminated with exception: Open shared directory failed. Please check whether input path is valid.
[default7]:[ERROR] 2026-05-27-19:52:42 (PID:229, Device:7, RankID:7) ERR02008
[default2]:[E527 19:52:42.934083889 compiler_depend.ts:1047] [Rank 2] HCCL watchdog thread terminated with exception: Open shared directory failed. Please check whether input path is valid.
[default2]:[ERROR] 2026-05-27-19:52:42 (PID:224, Device:2, RankID:2) ERR02008
[default4]:[E527 19:52:42.234955602 compiler_depend.ts:1047] [Rank 4] HCCL watchdog thread terminated with exception: Open shared directory failed. Please check whether input path is valid.
[default4]:[ERROR] 2026-05-27-19:52:42 (PID:226, Device:4, RankID:4) ERR02008
[default1]:[E527 19:52:42.364461090 compiler_depend.ts:1047] [Rank 1] HCCL watchdog thread terminated with exception: Open shared directory failed. Please check whether input path is valid.
[default1]:[ERROR] 2026-05-27-19:52:42 (PID:223, Device:1, RankID:1) ERR02008
W0527 19:52:50.852000 281467089262240 torch/distributed/elastic/multiprocessing/api.py:858] Sending process 222 closing signal SIGTERM
[default0]:Exception in thread Thread-1(default0)::

版本信息：CANN 8.2.RC1 + torch 2.4.0.post4

问题描述：训练完成后，保存检查点时出现以下错误：

HCCL watchdog thread terminated with exception: Open shared directory failed. Please check whether input path is valid.

所有Rank均报告相同错误，且进程退出时出现 SIGTERM 信号。

问题分类：框架兼容性

结论：问题根因在于PyTorch版本差异。在PyTorch 2.7及以上版本中，destroy_process_group 接口包含 pg.shutdown 逻辑，能够正常关闭通信域并释放资源。而在PyTorch 2.4中，该逻辑缺失，导致进程结束后，共享目录路径已被销毁，但HCCL Watchdog线程未能及时捕获该状态，从而报错。解决方案为在PTA 2.4版本中增加 shutdown 逻辑，确保通信域正常关闭。

---

问题10：DTensor自定义API断言失败（Import顺序问题）

报错信息：

File "/usr/local/python3.11.4/lib/python3.11/site-packages/torch/_compile.py", line 53, in inner
return disable_fn(*args, **kwargs)

File "/usr/local/python3.11.4/lib/python3.11/site-packages/torch/_dynamo/eval_frame.py", line 1044, in _fn
return fn(*args, **kwargs)

File "/usr/local/python3.11.4/lib/python3.11/site-packages/torch/distributed/tensor/_api.py", line 349, in __torch_dispatch__
return DTensor._op_dispatcher.dispatch(

File "/usr/local/python3.11.4/lib/python3.11/site-packages/torch/distributed/tensor/_dispatch.py", line 149, in dispatch
return self._custom_op_handlers[op_call](op_call, args, kwargs) # type: ignore[operator]

File "/usr/local/python3.11.4/lib/python3.11/site-packages/torch_npu/distributed/tensor/_attention.py", line 638, in _npu_fusion_attention_handler
DTensor._op_dispatcher.sharding_propagator.propagate(op_info)

File "/usr/local/python3.11.4/lib/python3.11/site-packages/torch/distributed/tensor/_sharding_prop.py", line 327, in propagate
OutputSharding, self.propagate_op_sharding(op_info.schema)

File "/usr/local/python3.11.4/lib/python3.11/site-packages/torch/distributed/tensor/_sharding_prop.py", line 46, in __call__
return self.cache(*args, **kwargs)

File "/usr/local/python3.11.4/lib/python3.11/site-packages/torch/distributed/tensor/_sharding_prop.py", line 346, in propagate_op_sharding_non_cached
op_strategy = self.op_strategy_funcs[op_schema.op](strategy_schema)

File "/usr/local/python3.11.4/lib/python3.11/site-packages/torch/distributed/tensor/experimental/_register_sharding.py", line 98, in custom_strategy
return expand_to_full_mesh_op_strategy(

File "/usr/local/python3.11.4/lib/python3.11/site-packages/torch/distributed/tensor/_ops/utils.py", line 332, in expand_to_full_mesh_op_strategy
assert len(input_specs) == len(input_args_strategy)

AssertionError

版本信息：PyTorch 2.9.0

问题描述：在使用DTensor自定义API时，出现以下断言错误：

File ".../_sharding_prop.py", line 332, in expand_to_full_mesh_op_strategy
assert len(input_specs) == len(input_args_strategy)
AssertionError

该问题导致4个自定义API用例执行失败。

问题分类：代码逻辑

结论：问题由模块导入顺序引起。在脚本中，import torch_npu.distributed.tensor._attention 与 import torch_npu.distributed.tensor._dtensor_patch 的顺序需要调整。正确的顺序应为先导入 _dtensor_patch，再导入 _attention，以确保策略注册的正确性。