v2.12.0分支用例拉起

.ci/docker/README.md

@@ -0,0 +1,91 @@
+# torch-npu CI Docker Images
+
+本目录管理 torch-npu 项目的 CI Docker 镜像，包括**构建镜像 (builder)** 和**测试镜像 (test)** 两类，每类分别支持 x86_64 和 aarch64 架构。
+
+## 镜像类型
+
+| 类型 | 基座 | 用途 |
+|------|------|------|
+| **builder** | manylinux2_28-builder | 编译构建 torch-npu wheel 包，包含完整编译工具链 |
+| **test** | ubuntu:22.04 | CI 单元测试运行环境，包含 PyTorch CPU、CANN runtime、triton-ascend 和测试框架 |
+
+## 目录结构
+
+```
+.ci/docker/
+├── README.md
+├── requirements-builder.txt      # Builder 镜像 pip 依赖
+├── requirements-test.txt         # Test 镜像 pip 依赖
+├── docker_build.sh               # 构建入口脚本
+├── common/                       # 共享安装脚本
+│   ├── install_cann.sh           # 安装 CANN toolkit (支持 A1/A2/A3)
+│   ├── install_triton.sh         # 安装 triton-ascend (需传 Python 版本)
+│   ├── install_obs.sh            # 安装华为 OBS util
+├── builder/
+│   ├── Dockerfile.x86_64
+│   └── Dockerfile.aarch64
+└── test/
+    ├── Dockerfile.x86_64
+    └── Dockerfile.aarch64
+```
+
+## 快速构建
+
+```bash
+# Builder 镜像 (不含 CANN)
+./docker_build.sh torch-npu-builder-x86_64-torch2.12.0
+./docker_build.sh torch-npu-builder-aarch64-torch2.12.0
+
+# Test 镜像 (含 CANN)
+./docker_build.sh torch-npu-test-x86_64-cann-a1-py3.10-torch2.12.0
+./docker_build.sh torch-npu-test-aarch64-cann-a2-py3.10-torch2.12.0
+```
+
+## Tag 命名规范
+
+参考上游 PyTorch `pytorch-linux-jammy-cuda12.4-cudnn9-py3-gcc11` 模式，tag 即为最终镜像名：
+
+**Builder**（不含 CANN）：
+```
+torch-npu-builder-<ARCH>-torch<PYTORCH_VERSION>
+```
+```
+./docker_build.sh torch-npu-builder-x86_64-torch2.12.0
+#                   ^          ^       ^     ^
+#                   |          |       |     └── PyTorch 版本 (torch2.12.0)
+#                   |          |       └── 架构
+#                   |          └── 镜像类型
+#                   └── 固定前缀
+```
+
+**Test**（含 CANN runtime）：
+```
+torch-npu-test-<ARCH>-cann<CHIP>-py<PYTHON_VERSION>-torch<PYTORCH_VERSION>
+```
+```
+./docker_build.sh torch-npu-test-x86_64-cann-a1-py3.10-torch2.12.0
+#                   ^         ^       ^    ^  ^     ^     ^
+#                   |         |       |    |  |     |     └── PyTorch 版本
+#                   |         |       |    |  |     └── torch 前缀
+#                   |         |       |    |  └── Python 版本
+#                   |         |       |    └── py 前缀
+#                   |         |       └── CANN 芯片 (A1/A2/A3)
+#                   |         └── 架构
+#                   └── 镜像类型
+```
+
+| 字段 | 可选值 |
+|------|--------|
+| IMAGE_TYPE | builder, test |
+| ARCH | x86_64, aarch64 |
+| CHIP | A1 (Ascend 910), A2 (Ascend 910b), A3 (仅 test) |
+| PYTHON_VERSION | 3.10 (仅 test) |
+| PYTORCH_VERSION | 2.12.0 |
+
+## CANN 芯片映射
+
+| CANN_CHIP | 芯片 | CANN 版本 |
+|-----------|------|----------|
+| A1 | Ascend 910 | 9.1.0 |
+| A2 | Ascend 910b | 8.5.0 (x86_64) / 9.1.0 (aarch64) |
+| A3 | Ascend A3 | 9.0.0-beta.1 (x86_64) / 9.0.0-beta.2 (aarch64) |

.ci/docker/builder/Dockerfile.aarch64

@@ -0,0 +1,85 @@
+FROM pytorch/manylinux2_28_aarch64-builder:cpu-aarch64-v2.12.0-rc9
+
+ARG PYTORCH_VERSION=2.12.0
+
+ENV PATH=/usr/local/bin:$PATH
+ENV AUDITWHEEL_PLAT=manylinux_2_28_aarch64
+ENV ETCD_UNSUPPORTED_ARCH=arm64
+ENV PYTORCH_VERSION=${PYTORCH_VERSION}
+
+COPY requirements-builder.txt /opt/buildtools/
+
+# Dynamically discover available cpython versions and create pip/python symlinks.
+# The base manylinux image provides /opt/_internal/cpython-X.Y.Z/ for each Python.
+RUN set -e; cd /usr/local/bin \
+    && for cpython_dir in /opt/_internal/cpython-3.*/; do \
+        py_ver=$(basename "$cpython_dir" | sed 's/cpython-//'); \
+        major_minor=$(echo "$py_ver" | grep -oP '^\d+\.\d+'); \
+        pybin="${cpython_dir}bin/python${major_minor}"; \
+        pipbin="${cpython_dir}bin/pip${major_minor}"; \
+        [ -f "$pybin" ] && ln -sf "$pybin" "python${major_minor}"; \
+        [ -f "$pipbin" ] && ln -sf "$pipbin" "pip${major_minor}"; \
+        echo "Registered Python ${major_minor} (${py_ver})"; \
+    done \
+    && ln -sf python3.10 python3 \
+    && ln -sf pip3.10 pip3 \
+    && echo "Default python: $(python3 --version)" \
+    && echo "Default pip: $(pip3 --version)"
+
+# Set pip source
+RUN mkdir /root/.pip \
+    && echo "[global]" > /root/.pip/pip.conf \
+    && echo "index-url=https://mirrors.huaweicloud.com/repository/pypi/simple" >> /root/.pip/pip.conf \
+    && echo "trusted-host=mirrors.huaweicloud.com" >> /root/.pip/pip.conf \
+    && echo "timeout=120" >> /root/.pip/pip.conf
+
+# Install PyTorch, build deps, and requirements for each python version.
+# Only install on Python 3.10-3.13; skip 3.14+ (torch 2.12 has no wheels for 3.14).
+RUN for pp in $(ls /usr/local/bin/pip3.* 2>/dev/null | grep -oP 'pip\d+\.\d+' | sort -V); do \
+        pyver=${pp#pip}; \
+        case "$pyver" in \
+            3.9|3.10|3.11|3.12|3.13) ;; \
+            *) echo "=== Skipping ${pp} (Python ${pyver} not supported by torch 2.12) ==="; continue ;; \
+        esac; \
+        echo "=== Installing PyTorch ${PYTORCH_VERSION} for ${pp} ==="; \
+        $pp install --no-cache-dir torch==${PYTORCH_VERSION} --extra-index-url https://download.pytorch.org/whl/cpu \
+            -r /opt/buildtools/requirements-builder.txt; \
+    done \
+    && echo "=== PyTorch installation complete ===" \
+    && auditwheel_bin=$(find /opt/_internal/cpython-3.1*/bin/auditwheel 2>/dev/null | tail -1) \
+    && if [ -n "$auditwheel_bin" ]; then \
+        ln -sf "$auditwheel_bin" /usr/local/bin/auditwheel; \
+        echo "auditwheel linked from ${auditwheel_bin}"; \
+    fi
+
+# Install system build tools
+RUN echo "alias ll='ls -l --color=auto'" >> /root/.bashrc \
+    && yum install -y vim-common --disablerepo=ius \
+    && yum install -y ninja-build binutils lld mold dos2unix gcc gcc-c++ make cmake3 wget tar unzip elfutils java-1.8.0-openjdk-devel \
+    && cd /tmp \
+    && wget https://github.com/ccache/ccache/releases/download/v4.10/ccache-4.10.tar.gz \
+    && tar -xzf ccache-4.10.tar.gz \
+    && cd ccache-4.10 \
+    && mkdir build \
+    && cd build \
+    && cmake3 .. \
+    && make -j$(nproc) \
+    && make install \
+    && cd /tmp \
+    && rm -rf ccache-4.10* \
+    && ccache --version \
+    && wget https://github.com/etcd-io/etcd/releases/download/v3.4.3/etcd-v3.4.3-linux-arm64.tar.gz \
+    && tar -zxf etcd-v3.4.3-linux-arm64.tar.gz \
+    && mv etcd-v3.4.3-linux-arm64/etcd /usr/local/bin/ \
+    && pip3.10 install python-etcd \
+    && etcd --version \
+    && yum update -y \
+    && yum clean all
+
+# Set timezone
+RUN rm -f /etc/localtime \
+    && cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime \
+    && echo 'Asia/Shanghai' >/etc/timezone \
+    && echo "export TZ='Asia/Shanghai'" >>/etc/profile
+
+WORKDIR /home

.ci/docker/builder/Dockerfile.x86_64

@@ -0,0 +1,97 @@
+FROM pytorch/manylinux2_28-builder:cpu-v2.12.0-rc9
+
+ARG PYTORCH_VERSION=2.12.0
+
+ENV PATH=/usr/local/bin:$PATH
+ENV AUDITWHEEL_PLAT=manylinux_2_28_x86_64
+ENV PYTORCH_VERSION=${PYTORCH_VERSION}
+
+COPY requirements-builder.txt /opt/buildtools/
+
+# Dynamically discover available cpython versions and create pip/python symlinks.
+# The base manylinux image provides /opt/_internal/cpython-X.Y.Z/ for each Python.
+RUN set -e; cd /usr/local/bin \
+    && for cpython_dir in /opt/_internal/cpython-3.*/; do \
+        py_ver=$(basename "$cpython_dir" | sed 's/cpython-//'); \
+        major_minor=$(echo "$py_ver" | grep -oP '^\d+\.\d+'); \
+        pybin="${cpython_dir}bin/python${major_minor}"; \
+        pipbin="${cpython_dir}bin/pip${major_minor}"; \
+        [ -f "$pybin" ] && ln -sf "$pybin" "python${major_minor}"; \
+        [ -f "$pipbin" ] && ln -sf "$pipbin" "pip${major_minor}"; \
+        echo "Registered Python ${major_minor} (${py_ver})"; \
+    done \
+    && ln -sf python3.10 python3 \
+    && ln -sf pip3.10 pip3 \
+    && echo "Default python: $(python3 --version)" \
+    && echo "Default pip: $(pip3 --version)"
+
+# Set pip source
+RUN mkdir /root/.pip \
+    && echo "[global]" > /root/.pip/pip.conf \
+    && echo "index-url=https://mirrors.huaweicloud.com/repository/pypi/simple" >> /root/.pip/pip.conf \
+    && echo "trusted-host=mirrors.huaweicloud.com" >> /root/.pip/pip.conf \
+    && echo "timeout=120" >> /root/.pip/pip.conf
+
+# Install PyTorch, build deps, and requirements for each python version.
+# x86_64 uses +cpu suffix to avoid pulling CUDA builds from PyPI.
+# Only install on Python 3.10-3.13; skip 3.14+ (torch 2.12 has no wheels for 3.14).
+RUN for pp in $(ls /usr/local/bin/pip3.* 2>/dev/null | grep -oP 'pip\d+\.\d+' | sort -V); do \
+        pyver=${pp#pip}; \
+        case "$pyver" in \
+            3.9|3.10|3.11|3.12|3.13) ;; \
+            *) echo "=== Skipping ${pp} (Python ${pyver} not supported by torch 2.12) ==="; continue ;; \
+        esac; \
+        echo "=== Installing PyTorch ${PYTORCH_VERSION}+cpu for ${pp} ==="; \
+        $pp install --no-cache-dir torch==${PYTORCH_VERSION}+cpu --extra-index-url https://download.pytorch.org/whl/cpu \
+            -r /opt/buildtools/requirements-builder.txt; \
+    done \
+    && echo "=== PyTorch installation complete ===" \
+    && auditwheel_bin=$(find /opt/_internal/cpython-3.1*/bin/auditwheel 2>/dev/null | tail -1) \
+    && if [ -n "$auditwheel_bin" ]; then \
+        ln -sf "$auditwheel_bin" /usr/local/bin/auditwheel; \
+        echo "auditwheel linked from ${auditwheel_bin}"; \
+    fi
+
+# Install system build tools
+RUN yum remove -y ius-release epel-release 2>/dev/null || true \
+    && rm -rf /etc/yum.repos.d/ius*.repo /etc/yum.repos.d/epel*.repo \
+    && yum clean all && rm -rf /var/cache/dnf /var/cache/yum \
+    && echo "alias ll='ls -l --color=auto'" >> /root/.bashrc \
+    && yum install -y vim-common --disablerepo=ius \
+    && yum install -y binutils lld dos2unix gcc gcc-c++ make cmake3 wget tar unzip elfutils java-1.8.0-openjdk-devel \
+    && cd /tmp \
+    && wget -q https://github.com/ninja-build/ninja/releases/download/v1.12.1/ninja-linux.zip \
+    && unzip ninja-linux.zip \
+    && cp ninja /usr/local/bin/ && chmod +x /usr/local/bin/ninja \
+    && cd /tmp \
+    && wget -q https://github.com/rui314/mold/archive/refs/tags/v2.32.1.tar.gz \
+    && tar -xf v2.32.1.tar.gz \
+    && cd mold-2.32.1 \
+    && cmake -DCMAKE_BUILD_TYPE=Release -DMOLD_MOSTLY_STATIC=ON . \
+    && make -j$(nproc) && make install \
+    && cd /tmp \
+    && wget https://github.com/ccache/ccache/releases/download/v4.10/ccache-4.10.tar.gz \
+    && tar -xzf ccache-4.10.tar.gz \
+    && cd ccache-4.10 \
+    && mkdir build \
+    && cd build \
+    && cmake3 .. \
+    && make -j$(nproc) \
+    && make install \
+    && cd /tmp && rm -rf /tmp/* \
+    && ninja --version && mold --version && ccache --version \
+    && wget https://github.com/etcd-io/etcd/releases/download/v3.4.3/etcd-v3.4.3-linux-amd64.tar.gz \
+    && tar -zxf etcd-v3.4.3-linux-amd64.tar.gz \
+    && mv etcd-v3.4.3-linux-amd64/etcd /usr/local/bin/ \
+    && pip3.10 install python-etcd \
+    && etcd --version \
+    && yum clean all \
+    && yum update -y
+
+# Set timezone
+RUN rm -f /etc/localtime \
+    && cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime \
+    && echo 'Asia/Shanghai' >/etc/timezone \
+    && echo "export TZ='Asia/Shanghai'" >>/etc/profile
+
+WORKDIR /home

.ci/docker/common/install_cann.sh

@@ -0,0 +1,109 @@
+#!/usr/bin/bash
+# Install CANN toolkit for Ascend NPU.
+# Usage: CANN_CHIP=A1 ./install_cann.sh
+#   CANN_CHIP: A1 (Ascend 910), A2 (Ascend 910b), A3 (Ascend A3)
+# Automatically detects architecture (x86_64 / aarch64).
+
+set -e
+
+CANN_CHIP="${CANN_CHIP:-A1}"
+ARCH=$(uname -m)
+
+BASE_URL="https://pytorch-package.obs.cn-north-4.myhuaweicloud.com/pta/cann-package"
+CANN_BASE_URL="https://ascend-repo.obs.cn-east-2.myhuaweicloud.com/CANN/CANN%209.1.T1"
+
+case "${ARCH}_${CANN_CHIP}" in
+  # x86_64
+  x86_64_A1)
+    TOOLKIT_URL="${BASE_URL}/20260513/Ascend-cann-toolkit_9.1.0_linux-x86_64.run"
+    OPS_URL="${BASE_URL}/20260513/Ascend-cann-910-ops_9.1.0_linux-x86_64.run"
+    NNAL_URL="${BASE_URL}/20260513/Ascend-cann-nnal_9.1.0_linux-x86_64.run"
+    OPS_GLOB="Ascend-cann-910*"
+    SET_ENV_PATH="/usr/local/Ascend/cann/set_env.sh"
+    ;;
+  x86_64_A2)
+    TOOLKIT_URL="${CANN_BASE_URL}/Ascend-cann-toolkit_9.1.0-beta.1_linux-x86_64.run"
+    OPS_URL="${CANN_BASE_URL}/Ascend-cann-910b-ops_9.1.0-beta.1_linux-x86_64.run"
+    NNAL_URL="${CANN_BASE_URL}/Ascend-cann-nnal_9.1.0-beta.1_linux-x86_64.run"
+    OPS_GLOB="Ascend-cann-910b-ops*"
+    SET_ENV_PATH="/usr/local/Ascend/cann/set_env.sh"
+    ;;
+  x86_64_A3)
+    TOOLKIT_URL="${CANN_BASE_URL}/Ascend-cann-toolkit_9.1.0-beta.1_linux-x86_64.run"
+    OPS_URL="${CANN_BASE_URL}/Ascend-cann-A3-ops_9.1.0-beta.1_linux-x86_64.run"
+    NNAL_URL="${CANN_BASE_URL}/Ascend-cann-nnal_9.1.0-beta.1_linux-x86_64.run"
+    OPS_GLOB="Ascend-cann-A3-ops*"
+    SET_ENV_PATH="/usr/local/Ascend/cann/set_env.sh"
+    ;;
+  # aarch64
+  aarch64_A1)
+    TOOLKIT_URL="${BASE_URL}/20260302/Ascend-cann-toolkit_9.0.0-beta.1_linux-aarch64.run"
+    OPS_URL="${BASE_URL}/20260302/Ascend-cann-910b-ops_9.0.0-beta.1_linux-aarch64.run"
+    NNAL_URL="${BASE_URL}/20260302/Ascend-cann-nnal_9.0.0-beta.1_linux-aarch64.run"
+    OPS_GLOB="Ascend-cann-910b*"
+    SET_ENV_PATH="/usr/local/Ascend/cann/set_env.sh"
+    ;;
+  aarch64_A2)
+    TOOLKIT_URL="${CANN_BASE_URL}/Ascend-cann-toolkit_9.1.0-beta.1_linux-aarch64.run"
+    OPS_URL="${CANN_BASE_URL}/Ascend-cann-910b-ops_9.1.0-beta.1_linux-aarch64.run"
+    NNAL_URL="${CANN_BASE_URL}/Ascend-cann-nnal_9.1.0-beta.1_linux-aarch64.run"
+    OPS_GLOB="Ascend-cann-910b-ops*"
+    SET_ENV_PATH="/usr/local/Ascend/cann/set_env.sh"
+    ;;
+  aarch64_A3)
+    TOOLKIT_URL="${CANN_BASE_URL}/Ascend-cann-toolkit_9.1.0-beta.1_linux-aarch64.run"
+    OPS_URL="${CANN_BASE_URL}/Ascend-cann-A3-ops_9.1.0-beta.1_linux-aarch64.run"
+    NNAL_URL="${CANN_BASE_URL}/Ascend-cann-nnal_9.1.0-beta.1_linux-aarch64.run"
+    OPS_GLOB="Ascend-cann-A3-ops*"
+    SET_ENV_PATH="/usr/local/Ascend/cann/set_env.sh"
+    ;;
+  *)
+    echo "Unsupported combination: ${ARCH} + ${CANN_CHIP}"
+    exit 1
+    ;;
+esac
+
+echo "Installing CANN ${CANN_CHIP} for ${ARCH}..."
+
+echo "=== Creating HwHiAiUser user and group ==="
+groupadd -f HwHiAiUser
+id -u HwHiAiUser >/dev/null 2>&1 || useradd -g HwHiAiUser -d /home/HwHiAiUser -m HwHiAiUser -s /bin/bash
+
+rm -rf cann
+mkdir -p cann && cd cann
+
+echo "=== Downloading CANN packages ==="
+curl -O "${TOOLKIT_URL}"
+curl -O "${OPS_URL}"
+curl -O "${NNAL_URL}"
+echo "Download complete."
+
+chmod +x Ascend-cann*.run
+
+echo "=== Installing CANN toolkit ==="
+./Ascend-cann-toolkit*.run --full --quiet --install-path=/usr/local/Ascend
+source "${SET_ENV_PATH}"
+echo "toolkit install success"
+
+echo "=== Installing CANN ops ==="
+./${OPS_GLOB}.run --install --quiet --install-path=/usr/local/Ascend
+echo "ops install success"
+
+echo "=== Installing CANN nnal ==="
+./Ascend-cann-nnal*.run --install --quiet --install-path=/usr/local/Ascend
+source /usr/local/Ascend/nnal/atb/set_env.sh
+echo "nnal install success"
+
+# Some CANN versions install to versioned paths (e.g. cann-9.0.0-beta.2)
+# instead of /usr/local/Ascend/cann/. Fix broken symlinks so runtime
+# sourcing of set_env.sh works.
+if [ ! -f /usr/local/Ascend/cann/set_env.sh ]; then
+  CANN_REAL_DIR=$(ls -d /usr/local/Ascend/cann-* 2>/dev/null | head -1)
+  if [ -n "${CANN_REAL_DIR}" ]; then
+    ln -sf "${CANN_REAL_DIR}" /usr/local/Ascend/cann
+    echo "Fixed: linked ${CANN_REAL_DIR} -> /usr/local/Ascend/cann"
+  fi
+fi
+
+rm -rf *
+echo "CANN ${CANN_CHIP} installation complete."