【Ascend】基于四台Atlas800T A2混合部署GLM5.3-w8a8c8

m0_74255282

194人浏览 · 2026-11-08 15:16:34

m0_74255282 · 2026-11-08 15:16:34 发布

1. 基础环境

镜像版本：quay.io/ascend/vllm-ascend:v0.30.0rc1
部署方式：混合部署

2. 操作步骤

2.1. 拉取镜像

docker pull quay.io/ascend/vllm-ascend:v0.30.0rc1

2.2. 启动容器

在每个节点上启动容器

export IMAGE=quay.io/ascend/vllm-ascend:v0.30.0rc1
docker run \
    --name {容器名称} \
    --shm-size=1g \
    --net=host \
    --device /dev/davinci0 \
    --device /dev/davinci1 \
    --device /dev/davinci2 \
    --device /dev/davinci3 \
    --device /dev/davinci4 \
    --device /dev/davinci5 \
    --device /dev/davinci6 \
    --device /dev/davinci7 \
    --device /dev/davinci_manager \
    --device /dev/devmm_svm \
    --device /dev/hisi_hdc \
    -v /usr/local/dcmi:/usr/local/dcmi \
    -v /usr/local/Ascend/driver/tools/hccn_tool:/usr/local/Ascend/driver/tools/hccn_tool \
    -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
    -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
    -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
    -v /etc/ascend_install.info:/etc/ascend_install.info \
    -v /data/model:/data/model \ # 配置的挂载到容器中的模型权重文件地址
    -it $IMAGE bash

2.3. 配置在线服务脚本

使用到的命令，主要用于查询使用的网卡的名称（nic_name）与下面要使用到的ip地址

在宿主机上运行
# 显示所有网卡与ip地址
ifconfig -a

2.3.1. 在容器中创建脚本文件

节点0 - start_model_server.sh（主节点）

# this obtained through ifconfig
# nic_name is the network interface name corresponding to local_ip of the current node
nic_name="查询到的使用的网卡的名称"
local_ip="主节点ip"

# The value of node0_ip must be consistent with the value of local_ip set in node0 (master node)
node0_ip="主节点ip"

export HCCL_OP_EXPANSION_MODE="AIV"
export HCCL_IF_IP=$local_ip
export GLOO_SOCKET_IFNAME=$nic_name
export TP_SOCKET_IFNAME=$nic_name
export HCCL_SOCKET_IFNAME=$nic_name
export VLLM_RPC_TIMEOUT=360000
export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=30000
export HCCL_EXEC_TIMEOUT=200
export HCCL_CONNECT_TIMEOUT=120
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=10
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export ACL_OP_INIT_MODE=1
export CPU_AFFINITY_CONF=1
export VLLM_ASCEND_ENABLE_MLAPO=1
export VLLM_ENGINE_READY_TIMEOUT_S=1200

# Ensure the model path matches the directory recorded during download
vllm serve /data/model/{模型权重文件存储路径} \
    --host 0.0.0.0 \
    --port 8000 \
    --max-model-len 135000 \
    --data-parallel-size 4 \
    --data-parallel-size-local 1 \
    --data-parallel-start-rank 0 \
    --data-parallel-address "${node0_ip}" \
    --data-parallel-rpc-port 12980 \
    --tensor-parallel-size 8 \
    --enable-expert-parallel \
    --seed 1024 \
    --served-model-name glm-5 \
    --reasoning-parser glm47 \
    --safetensors-load-strategy prefetch \
    --max-num-seqs 128 \
    --max-num-batched-tokens 8192 \
    --trust-remote-code \
    --quantization ascend \
    --gpu-memory-utilization 0.92 \
    --speculative-config '{"num_speculative_tokens": 3, "method": "deepseek_mtp", "enforce_eager": true}' \
    --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY"}' \
    --kv-cache-dtype int8 \
    --attention_config.indexer_kv_dtype int8 \
    --additional-config '{"enable_dsa_cp": true, "enable_balance_scheduling": true,  "multistream_overlap_shared_expert": true, "enable_flashcomm1": true}' \
    --enable-prefix-caching \
    --async-scheduling \
    --api-server-count 1

节点1- start_model_server.sh

# this obtained through ifconfig
# nic_name is the network interface name corresponding to local_ip of the current node
nic_name="当前节点查询到的使用的网卡的名称"
local_ip="当前节点的ip"

# The value of node0_ip must be consistent with the value of local_ip set in node0 (master node)
node0_ip="主节点ip"

# node1: dp_start_rank=1, node2: dp_start_rank=2, node3: dp_start_rank=3
dp_start_rank=1

export HCCL_OP_EXPANSION_MODE="AIV"
export HCCL_IF_IP=$local_ip
export GLOO_SOCKET_IFNAME=$nic_name
export TP_SOCKET_IFNAME=$nic_name
export HCCL_SOCKET_IFNAME=$nic_name
export VLLM_RPC_TIMEOUT=360000
export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=30000
export HCCL_EXEC_TIMEOUT=200
export HCCL_CONNECT_TIMEOUT=120
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=10
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export ACL_OP_INIT_MODE=1
export CPU_AFFINITY_CONF=1
export VLLM_ASCEND_ENABLE_MLAPO=1
export VLLM_ENGINE_READY_TIMEOUT_S=1200

# Ensure the model path matches the directory recorded during download
vllm serve /data/model/{模型权重文件存储路径} \
    --host 0.0.0.0 \
    --port 8000 \
    --max-model-len 135000 \
    --data-parallel-size 4 \
    --data-parallel-size-local 1 \
    --data-parallel-start-rank ${dp_start_rank} \
    --data-parallel-address "${node0_ip}" \
    --data-parallel-rpc-port 12980 \
    --tensor-parallel-size 8 \
    --enable-expert-parallel \
    --seed 1024 \
    --served-model-name glm-5 \
    --reasoning-parser glm47 \
    --safetensors-load-strategy prefetch \
    --max-num-seqs 128 \
    --max-num-batched-tokens 8192 \
    --trust-remote-code \
    --quantization ascend \
    --gpu-memory-utilization 0.92 \
    --speculative-config '{"num_speculative_tokens": 3, "method": "deepseek_mtp", "enforce_eager": true}' \
    --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY"}' \
    --kv-cache-dtype int8 \
    --attention_config.indexer_kv_dtype int8 \
    --additional-config '{"enable_dsa_cp": true, "enable_balance_scheduling": true, "multistream_overlap_shared_expert": true, "enable_flashcomm1": true}' \
    --enable-prefix-caching \
    --async-scheduling

节点2 - start_model_server.sh

# this obtained through ifconfig
# nic_name is the network interface name corresponding to local_ip of the current node
nic_name="当前节点查询到的使用的网卡的名称"
local_ip="当前节点ip"

# The value of node0_ip must be consistent with the value of local_ip set in node0 (master node)
node0_ip="主节点ip"

# node1: dp_start_rank=1, node2: dp_start_rank=2, node3: dp_start_rank=3
dp_start_rank=2

export HCCL_OP_EXPANSION_MODE="AIV"
export HCCL_IF_IP=$local_ip
export GLOO_SOCKET_IFNAME=$nic_name
export TP_SOCKET_IFNAME=$nic_name
export HCCL_SOCKET_IFNAME=$nic_name
export VLLM_RPC_TIMEOUT=360000
export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=30000
export HCCL_EXEC_TIMEOUT=200
export HCCL_CONNECT_TIMEOUT=120
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=10
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export ACL_OP_INIT_MODE=1
export CPU_AFFINITY_CONF=1
export VLLM_ASCEND_ENABLE_MLAPO=1
export VLLM_ENGINE_READY_TIMEOUT_S=1200

# Ensure the model path matches the directory recorded during download
vllm serve /data/model/{模型权重文件存储路径} \
    --host 0.0.0.0 \
    --port 8000 \
    --max-model-len 135000 \
    --data-parallel-size 4 \
    --data-parallel-size-local 1 \
    --data-parallel-start-rank ${dp_start_rank} \
    --data-parallel-address "${node0_ip}" \
    --data-parallel-rpc-port 12980 \
    --tensor-parallel-size 8 \
    --enable-expert-parallel \
    --seed 1024 \
    --served-model-name glm-5 \
    --reasoning-parser glm47 \
    --safetensors-load-strategy prefetch \
    --max-num-seqs 128 \
    --max-num-batched-tokens 8192 \
    --trust-remote-code \
    --quantization ascend \
    --gpu-memory-utilization 0.92 \
    --speculative-config '{"num_speculative_tokens": 3, "method": "deepseek_mtp", "enforce_eager": true}' \
    --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY"}' \
    --kv-cache-dtype int8 \
    --attention_config.indexer_kv_dtype int8 \
    --additional-config '{"enable_dsa_cp": true, "enable_balance_scheduling": true, "multistream_overlap_shared_expert": true, "enable_flashcomm1": true}' \
    --enable-prefix-caching \
    --async-scheduling

节点3 - start_model_server.sh

# this obtained through ifconfig
# nic_name is the network interface name corresponding to local_ip of the current node
nic_name="当前节点查询到的使用的网卡的名称"
local_ip="当前节点ip"

# The value of node0_ip must be consistent with the value of local_ip set in node0 (master node)
node0_ip="主节点ip"

# node1: dp_start_rank=1, node2: dp_start_rank=2, node3: dp_start_rank=3
dp_start_rank=3

export HCCL_OP_EXPANSION_MODE="AIV"
export HCCL_IF_IP=$local_ip
export GLOO_SOCKET_IFNAME=$nic_name
export TP_SOCKET_IFNAME=$nic_name
export HCCL_SOCKET_IFNAME=$nic_name
export VLLM_RPC_TIMEOUT=360000
export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=30000
export HCCL_EXEC_TIMEOUT=200
export HCCL_CONNECT_TIMEOUT=120
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=10
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export ACL_OP_INIT_MODE=1
export CPU_AFFINITY_CONF=1
export VLLM_ASCEND_ENABLE_MLAPO=1
export VLLM_ENGINE_READY_TIMEOUT_S=1200

# Ensure the model path matches the directory recorded during download
vllm serve /data/model/{模型权重文件存储路径} \
    --host 0.0.0.0 \
    --port 8000 \
    --max-model-len 135000 \
    --data-parallel-size 4 \
    --data-parallel-size-local 1 \
    --data-parallel-start-rank ${dp_start_rank} \
    --data-parallel-address "${node0_ip}" \
    --data-parallel-rpc-port 12980 \
    --tensor-parallel-size 8 \
    --enable-expert-parallel \
    --seed 1024 \
    --served-model-name glm-5 \
    --reasoning-parser glm47 \
    --safetensors-load-strategy prefetch \
    --max-num-seqs 128 \
    --max-num-batched-tokens 8192 \
    --trust-remote-code \
    --quantization ascend \
    --gpu-memory-utilization 0.92 \
    --speculative-config '{"num_speculative_tokens": 3, "method": "deepseek_mtp", "enforce_eager": true}' \
    --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY"}' \
    --kv-cache-dtype int8 \
    --attention_config.indexer_kv_dtype int8 \
    --additional-config '{"enable_dsa_cp": true, "enable_balance_scheduling": true, "multistream_overlap_shared_expert": true, "enable_flashcomm1": true}' \
    --enable-prefix-caching \
    --async-scheduling

2.4. 启动脚本

bash start_model_server.sh

首先启动主节点容器中的脚本
再依次启动从节点容器中的脚本即可

2.5. 验证服务连通性

curl http://设置的主节点ip:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "{脚本中配置的vllm服务的模型名称}",
        "messages":[
            {
                "role": "user",
                "content": "Who are you?"
            }
        ],
        "temperature": 0
    }'

2.6. 性能测试

使用aisbench进行性能测试
前提：需要下载aisbench源码文件及安装依赖，安装依赖时最好配置独立的依赖环境，避免依赖冲突

修改随机合成数据集配置
配置路径
下载源码路径/ais_bench/benchmark/configs/datasets/synthetic/synthetic_gen_string.py
修改配置：

synthetic_config = {
    "Type": "string",
    "RequestCount": 128,  # 总请求数
    "StringConfig": {
        "Input": {
            "Method": "uniform",
            "Params": {"MinValue": 16384, "MaxValue": 16384}  # 固定输入长度为16384，根据需要修改
        },
        "Output": {
            "Method": "uniform",
            "Params": {"MinValue": 1024, "MaxValue": 1024}   # 固定输出长度为1024，根据需要修改
        }
    }
}

修改模型在线服务配置
配置路径：
下载源码路径/ais_bench/benchmark/configs/models/vllm_api/vllm_api_stream_chat.py
修改配置：

models = [
    dict(
        attr="service",
        type=VLLMCustomAPIChat,
        abbr="vllm-api-stream-chat",
        path="{模型权重路径}",
        model="{vllm启动脚本配置的模型服务名称}",
        stream=True,
        request_rate=0,
        use_timestamp=False,
        retry=2,
        api_key="",
        host_ip="{模型启动的ip地址}",
        host_port={模型启动的端口号},
        url="",
        max_out_len={最大输出数量}, # 根据需要修改
        batch_size={并发量},
        trust_remote_code=False,
        generation_kwargs=dict(
            temperature=0.01,
            ignore_eos=True,
        ),
        pred_postprocessor=dict(type=extract_non_reasoning_content),
    )
]

启动性能测试

ais_bench --models vllm_api_stream_chat \
     --datasets synthetic_gen_string \
     --mode perf

【Ascend】基于四台Atlas800T A2混合部署GLM5.3-w8a8c8

m0_74255282

1. 基础环境

2. 操作步骤

2.1. 拉取镜像

2.2. 启动容器

2.3. 配置在线服务脚本

2.3.1. 在容器中创建脚本文件

2.4. 启动脚本

2.5. 验证服务连通性

2.6. 性能测试

相关参考链接：

所有评论(0)

m0_74255282

【Ascend】基于四台Atlas800T A2混合部署GLM5.3-w8a8c8

m0_74255282

1. 基础环境

2. 操作步骤

2.1. 拉取镜像

2.2. 启动容器

2.3. 配置在线服务脚本

2.3.1. 在容器中创建脚本文件

2.4. 启动脚本

2.5. 验证服务连通性

2.6. 性能测试

相关参考链接：

所有评论(0)

温馨提示：您尚未绑定手机号

m0_74255282