当前位置：工控论坛 > 产品 > 自动化软件

模型插入 NV12 预处理节点精度问题排查流程 点击：20 | 回复：0



206yjf

    SSI ļʱ
发表于：2025-10-10 15:16:36
楼主

模型插入 NV12 预处理节点精度问题排查流程

近期工具链实践过程中，频繁出现 BC 模型在插入 NV12 预处理节点后精度崩溃的现象。经分析，此类问题可分为两类：其一为用户侧 BGR/RGB 转 NV12 的代码实现缺陷；其二为 BGR/RGB 与 NV12 格式转换过程中固有的不可逆误差所致。后者的根本原因在于模型训练阶段存在严重过拟合现象，导致模型泛化能力薄弱，无法容忍格式转换带来的微小数据偏差。

此类问题的传统排查流程耗时较长，且往往需要重新训练浮点模型，对用户开发进度造成显著影响。基于此，本文提供一套标准化的快速排查方案，旨在缩短问题定位周期，降低对开发节奏的干扰。

二、问题确定与现象
2.1.1 关键术语定义

  • qat_nv12.bc/quantized_nv12.bc：插入 NV12 预处理节点的伪量化 / 定点 BC 模型

  • qat_feat.bc/quantized_feat.bc：未插入 NV12 预处理节点的伪量化 / 定点 BC 模型

2.1.2 典型现象

问题通常表现为：在对插入 NV12 预处理节点的 HBM 或 qat_nv12.bc/quantized_nv12.bc 进行精度验证时，出现 quantized_feat.bc 精度正常而 quantized_nv12.bc 精度异常的特征性差异。

2.1.3 NV12 节点插入规范

在 qat.bc 中插入 NV12 预处理节点的标准代码如下：

resizer_input = ["resize"]      # 部署时数据来源于resizer的输入节点名称列表
pyramid_input = ["pym"]         # 部署时数据来源于pyramid的输入节点名称列表

# 为和历史版本保持兼容，建议使用flatten_inputs将输入展开，如下代码同时兼容新旧版本模型：
for input in func.flatten_inputs[::-1]:
    if input.name in pyramid_input:
        # pyramid&resizer 只支持 NHWC 的 input layout，若原始输入layout为NHWC，则无需插入transpose
        node = input.insert_transpose(permutes=[0, 3, 1, 2])
        # 插入前处理节点，具体可参考下一节的说明
        node = node.insert_image_preprocess(mode=None, divisor=1, mean=[128, 128, 128], std=[128, 128, 128])
        node.insert_image_convert("nv12")
            
for input in func.flatten_inputs[::-1]:
    if input.name in resizer_input:
        # pyramid&resizer 只支持 NHWC 的 input layout，若原始输入layout为NHWC，则无需插入transpose
        node = input.insert_transpose(permutes=[0, 3, 1, 2])
        # 插入前处理节点，具体可参考下一节的说明
        node = node.insert_image_preprocess(mode=None, divisor=1, mean=[128, 128, 128], std=[128, 128, 128])
        node.insert_roi_resize("nv12")
2.2 核心验证思路

通过单帧图像分别推理两类模型（含 NV12 节点与不含 NV12 节点），对比输出结果的可视化效果及数值相似度。预期异常特征为：

  • quantized_nv12.bc 与 quantized_feat.bc 推理结果差异显著

  • qat_nv12.bc 与 quantized_nv12.bc 推理结果一致性差

推理时需要注意两个问题：

  1. quantized_nv12.bc 已经插入了 NV12->bgr/rgb 和归一化节点了，所以其输入为 NV12 图像；quantized_feat.bc 的输入需要对齐浮点模型，为手动做过归一化操作的 bgr/rgb 图像；

  2. bgr/rgb->NV12 图像的方式有多种，推荐使用以下方式；

def generate_nv12(img):
    w,h = img.size
    # Convert images to YUV format
    yuv_img = img.convert('YCbCr')
    y_data, u_data, v_data = yuv_img.split()

    # Convert Y, U, and V channel data to byte streams
    y_data_bytes = y_data.tobytes()
    u_data_bytes = u_data.resize((u_data.width // 2, u_data.height // 2)).tobytes()
    v_data_bytes = v_data.resize((v_data.width // 2, v_data.height // 2)).tobytes()

    # Arrange the UV data in the form of UVUVUVUV... 
    uvuvuv_data = bytearray()
    for u_byte, v_byte in zip(u_data_bytes, v_data_bytes):
        uvuvuv_data.extend([u_byte, v_byte])

    # Input for the hbir model
    y = np.frombuffer(y_data_bytes, dtype=np.uint8).reshape(1, h, w, 1).astype(np.uint8)
    # np.save("y_data.npy", y)
    uv = np.frombuffer(uvuvuv_data, dtype=np.uint8).reshape(1, h//2, w//2, 2).astype(np.uint8)
    # np.save("uv_data.npy", uv)
    return y, uv
三、原因验证与示例
3.1 验证原理

当确定是插入 NV12 预处理节点导致的 bc/hbm 精度下降后，且确定输入数据处理过程完全正确，为了进一步确定原因，可以通过以下方式来证明：在输入数据中加入 2-3 个像素值的随机噪声推理浮点模型。

rgb->NV12 的过程是有损的，所以我们可以在输入图像中加小幅的随机噪声（2-3 个像素值即可），然后输入到浮点模型或者原始浮点 onnx，如果加小噪声前后的模型输出相差很大（可以观察 cos 相似度、L1 等），证明确实为浮点模型过拟合导致。

3.2 参考实现代码

以下为对比输入数据加噪前后，比较浮点 onnx 推理结果的示例代码：

from hbdk4.compiler import load,compile,hbm_perf,visualize,save,convert
import numpy as np
from horizon_tc_ui.hb_runtime import HBRuntime
def cosine_similarity(vec1, vec2):
    """计算两向量余弦相似度"""
    vec1_flat = vec1.flatten()
    vec2_flat = vec2.flatten() 
    dot_product = np.dot(vec1_flat, vec2_flat)
    norm_vec1 = np.linalg.norm(vec1_flat)
    norm_vec2 = np.linalg.norm(vec2_flat)
    return dot_product / (norm_vec1 * norm_vec2) if (norm_vec1 * norm_vec2) != 0 else 0
def compare_noise_impact(float_model_path):
    """对比加噪前后浮点模型输出差异"""
    input_data = np.load("data.npy")
    # 添加0-2像素值范围的随机噪声
    noise = np.random.uniform(0, 2, size=input_data.shape).astype(np.float32)
    input_data_noise = input_data + noise
    sess = HBRuntime(float_model_path)
    input_names = sess.input_names
    output_names = sess.output_names
    
    # 原始输入推理
    input_dict = {input_names[0]: input_data}
    outputs_original = sess.run(output_names, input_dict)
    
    # 加噪输入推理
    input_dict_noise = {input_names[0]: input_data_noise}
    outputs_noise = sess.run(output_names, input_dict_noise)
    https://www.bilibili.com/video/BV1yx47zkEeu/
https://www.bilibili.com/video/BV1yx47zkEeu
https://www.bilibili.com/video/BV1yx47zkEm7/
https://www.bilibili.com/video/BV1yx47zkEm7
https://www.bilibili.com/video/BV11x47zkEK7/
https://www.bilibili.com/video/BV11x47zkEK7
https://www.bilibili.com/video/BV1Nx47zCEjr/
https://www.bilibili.com/video/BV1Nx47zCEjr
https://www.bilibili.com/video/BV11x47zkENp/
https://www.bilibili.com/video/BV11x47zkENp
https://www.bilibili.com/video/BV1yx47zkEbK/
https://www.bilibili.com/video/BV1yx47zkEbK
https://www.bilibili.com/video/BV11x47zkENx/
https://www.bilibili.com/video/BV11x47zkENx
https://www.bilibili.com/video/BV1yx47zkEkh/
https://www.bilibili.com/video/BV1yx47zkEkh
https://www.bilibili.com/video/BV1Nx47zCEek/
https://www.bilibili.com/video/BV1Nx47zCEek
https://www.bilibili.com/video/BV1Px47zCEUt/
https://www.bilibili.com/video/BV1Px47zCEUt
https://www.bilibili.com/video/BV1Px47zCEZJ/
https://www.bilibili.com/video/BV1Px47zCEZJ
https://www.bilibili.com/video/BV1yx47zkEmG/
https://www.bilibili.com/video/BV1yx47zkEmG
https://www.bilibili.com/video/BV1yx47zkEkU/
https://www.bilibili.com/video/BV1yx47zkEkU
https://www.bilibili.com/video/BV1Px47zCEUk/
https://www.bilibili.com/video/BV1Px47zCEUk
https://www.bilibili.com/video/BV11x47zCEVd/
https://www.bilibili.com/video/BV11x47zCEVd
https://www.bilibili.com/video/BV1Nx47zCEPT/
https://www.bilibili.com/video/BV1Nx47zCEPT
https://www.bilibili.com/video/BV1Nx47zCE7e/
https://www.bilibili.com/video/BV1Nx47zCE7e
https://www.bilibili.com/video/BV1Nx47zCEPG/
https://www.bilibili.com/video/BV1Nx47zCEPG
https://www.bilibili.com/video/BV1yx47zkEHN/
https://www.bilibili.com/video/BV1yx47zkEHN
https://www.bilibili.com/video/BV1Px47zCERq/
https://www.bilibili.com/video/BV1Px47zCERq
https://www.bilibili.com/video/BV1yx47zkETZ/
https://www.bilibili.com/video/BV1yx47zkETZ
https://www.bilibili.com/video/BV1Nx47zCEF3/
https://www.bilibili.com/video/BV1Nx47zCEF3
https://www.bilibili.com/video/BV1yx47zkEHE/
https://www.bilibili.com/video/BV1yx47zkEHE
https://www.bilibili.com/video/BV1yx47zkEHJ/
https://www.bilibili.com/video/BV1yx47zkEHJ
https://www.bilibili.com/video/BV1Nx47zCEis/
https://www.bilibili.com/video/BV1Nx47zCEis
https://www.bilibili.com/video/BV1yx47zkEmo/
https://www.bilibili.com/video/BV1yx47zkEmo
https://www.bilibili.com/video/BV1Nx47zCEGn/
https://www.bilibili.com/video/BV1Nx47zCEGn
https://www.bilibili.com/video/BV1yx47zkEC4/
https://www.bilibili.com/video/BV1yx47zkEC4
https://www.bilibili.com/video/BV11x47zkEKC/
https://www.bilibili.com/video/BV11x47zkEKC
https://www.bilibili.com/video/BV1yx47zkE8x/
https://www.bilibili.com/video/BV1yx47zkE8x
https://www.bilibili.com/video/BV1yx47zkEeC/
https://www.bilibili.com/video/BV1yx47zkEeC
https://www.bilibili.com/video/BV1yx47zkECi/
https://www.bilibili.com/video/BV1yx47zkECi
https://www.bilibili.com/video/BV1Nx47zCEGy/
https://www.bilibili.com/video/BV1Nx47zCEGy
https://www.bilibili.com/video/BV1Nx47zCEvk/
https://www.bilibili.com/video/BV1Nx47zCEvk
https://www.bilibili.com/video/BV11x47zkEKm/
https://www.bilibili.com/video/BV11x47zkEKm
https://www.bilibili.com/video/BV1yx47zkE86/
https://www.bilibili.com/video/BV1yx47zkE86
https://www.bilibili.com/video/BV1Nx47zCEAZ/
https://www.bilibili.com/video/BV1Nx47zCEAZ
https://www.bilibili.com/video/BV12H47zLEib/
https://www.bilibili.com/video/BV12H47zLEib
https://www.bilibili.com/video/BV1TH47zjENE/
https://www.bilibili.com/video/BV1TH47zjENE
https://www.bilibili.com/video/BV1KH47zjEkL/
https://www.bilibili.com/video/BV1KH47zjEkL
https://www.bilibili.com/video/BV12H47zLERP/
https://www.bilibili.com/video/BV12H47zLERP
https://www.bilibili.com/video/BV1TH47zjEE1/
https://www.bilibili.com/video/BV1TH47zjEE1
https://www.bilibili.com/video/BV12H47zLEVa/
https://www.bilibili.com/video/BV12H47zLEVa
https://www.bilibili.com/video/BV12H47zLE4k/
https://www.bilibili.com/video/BV12H47zLE4k
https://www.bilibili.com/video/BV1TH47zjEc7/
https://www.bilibili.com/video/BV1TH47zjEc7
https://www.bilibili.com/video/BV1TH47zjEgA/
https://www.bilibili.com/video/BV1TH47zjEgA
https://www.bilibili.com/video/BV12H47zLE4f/
https://www.bilibili.com/video/BV12H47zLE4f
https://www.bilibili.com/video/BV12H47zLEax/
https://www.bilibili.com/video/BV12H47zLEax
https://www.bilibili.com/video/BV12H47zLEvd/
https://www.bilibili.com/video/BV12H47zLEvd
https://www.bilibili.com/video/BV12H47zLEGU/
https://www.bilibili.com/video/BV12H47zLEGU
https://www.bilibili.com/video/BV14H47zjEba/
https://www.bilibili.com/video/BV14H47zjEba
https://www.bilibili.com/video/BV1mH47zLEG7/
https://www.bilibili.com/video/BV1mH47zLEG7
https://www.bilibili.com/video/BV12H47zLEGo/
https://www.bilibili.com/video/BV12H47zLEGo
https://www.bilibili.com/video/BV12H47zLESt/
https://www.bilibili.com/video/BV12H47zLESt
https://www.bilibili.com/video/BV12H47zLEVA/
https://www.bilibili.com/video/BV12H47zLEVA
https://www.bilibili.com/video/BV1TH47zjENg/
https://www.bilibili.com/video/BV1TH47zjENg
https://www.bilibili.com/video/BV12H47zLEVG/
https://www.bilibili.com/video/BV12H47zLEVG
https://www.bilibili.com/video/BV1mH47zLEV9/
https://www.bilibili.com/video/BV1mH47zLEV9
https://www.bilibili.com/video/BV12H47zLEqz/
https://www.bilibili.com/video/BV12H47zLEqz
https://www.bilibili.com/video/BV1TH47zjEc4/
https://www.bilibili.com/video/BV1TH47zjEc4
https://www.bilibili.com/video/BV1TH47zjEgJ/
https://www.bilibili.com/video/BV1TH47zjEgJ
https://www.bilibili.com/video/BV12H47zLEaZ/
https://www.bilibili.com/video/BV12H47zLEaZ
https://www.bilibili.com/video/BV1mH47zLEVk/
https://www.bilibili.com/video/BV1mH47zLEVk
https://www.bilibili.com/video/BV12H47zLEon/
https://www.bilibili.com/video/BV12H47zLEon
https://www.bilibili.com/video/BV12H47zLE8M/
https://www.bilibili.com/video/BV12H47zLE8M
https://www.bilibili.com/video/BV12H47zLE8V/
https://www.bilibili.com/video/BV12H47zLE8V
https://www.bilibili.com/video/BV1TH47zjEwk/
https://www.bilibili.com/video/BV1TH47zjEwk
https://www.bilibili.com/video/BV12H47zLEeF/
https://www.bilibili.com/video/BV12H47zLEeF
https://www.bilibili.com/video/BV12H47zLEQy/
https://www.bilibili.com/video/BV12H47zLEQy
https://www.bilibili.com/video/BV12H47zLEZB/
https://www.bilibili.com/video/BV12H47zLEZB
https://www.bilibili.com/video/BV12H47zLEhg/
https://www.bilibili.com/video/BV12H47zLEhg
https://www.bilibili.com/video/BV1TH47zjEVb/
https://www.bilibili.com/video/BV1TH47zjEVb
https://www.bilibili.com/video/BV1TH47zjEV8/
https://www.bilibili.com/video/BV1TH47zjEV8
https://www.bilibili.com/video/BV12H47zLEZG/
https://www.bilibili.com/video/BV12H47zLEZG
https://www.bilibili.com/video/BV11p47zyEUx/
https://www.bilibili.com/video/BV11p47zyEUx
https://www.bilibili.com/video/BV11p47zyEXd/
https://www.bilibili.com/video/BV11p47zyEXd
https://www.bilibili.com/video/BV12p47zyE8p/
https://www.bilibili.com/video/BV12p47zyE8p
https://www.bilibili.com/video/BV11p47zyEZt/
https://www.bilibili.com/video/BV11p47zyEZt
https://www.bilibili.com/video/BV1xp47z1EQN/
https://www.bilibili.com/video/BV1xp47z1EQN
https://www.bilibili.com/video/BV11p47zyE8M/
https://www.bilibili.com/video/BV11p47zyE8M
https://www.bilibili.com/video/BV1Kp47z1EGJ/
https://www.bilibili.com/video/BV1Kp47z1EGJ
https://www.bilibili.com/video/BV1Kp47z1Ex1/
https://www.bilibili.com/video/BV1Kp47z1Ex1
https://www.bilibili.com/video/BV11p47zyECh/
https://www.bilibili.com/video/BV11p47zyECh
https://www.bilibili.com/video/BV12p47z1EJE/
https://www.bilibili.com/video/BV12p47z1EJE
https://www.bilibili.com/video/BV11p47zyE88/
https://www.bilibili.com/video/BV11p47zyE88
https://www.bilibili.com/video/BV11p47zyE8W/
https://www.bilibili.com/video/BV11p47zyE8W
https://www.bilibili.com/video/BV12p47zyE8F/
https://www.bilibili.com/video/BV12p47zyE8F
https://www.bilibili.com/video/BV1Kp47z1E7T/
https://www.bilibili.com/video/BV1Kp47z1E7T
https://www.bilibili.com/video/BV12p47zyEhL/
https://www.bilibili.com/video/BV12p47zyEhL
https://www.bilibili.com/video/BV12p47zyEhu/
https://www.bilibili.com/video/BV12p47zyEhu
https://www.bilibili.com/video/BV1Tp47z1EFK/
https://www.bilibili.com/video/BV1Tp47z1EFK
https://www.bilibili.com/video/BV1xp47z1ECG/
https://www.bilibili.com/video/BV1xp47z1ECG
https://www.bilibili.com/video/BV11p47zyES2/
https://www.bilibili.com/video/BV11p47zyES2
https://www.bilibili.com/video/BV12p47zyEGm/
https://www.bilibili.com/video/BV12p47zyEGm
https://www.bilibili.com/video/BV1Kp47z1Ery/
https://www.bilibili.com/video/BV1Kp47z1Ery
https://www.bilibili.com/video/BV11p47zyE1b/
https://www.bilibili.com/video/BV11p47zyE1b
https://www.bilibili.com/video/BV11p47z1EZA/
https://www.bilibili.com/video/BV11p47z1EZA
https://www.bilibili.com/video/BV11p47zyEyk/
https://www.bilibili.com/video/BV11p47zyEyk
https://www.bilibili.com/video/BV11p47zyEhn/
https://www.bilibili.com/video/BV11p47zyEhn
https://www.bilibili.com/video/BV1xp47z1EQa/
https://www.bilibili.com/video/BV1xp47z1EQa
https://www.bilibili.com/video/BV1xT4EzyEtu/
https://www.bilibili.com/video/BV1xT4EzyEtu
https://www.bilibili.com/video/BV1xT4EzyE2F/
https://www.bilibili.com/video/BV1xT4EzyE2F
https://www.bilibili.com/video/BV1sT4EzyEdt/
https://www.bilibili.com/video/BV1sT4EzyEdt
https://www.bilibili.com/video/BV1xT4EzyEmu/
https://www.bilibili.com/video/BV1xT4EzyEmu
https://www.bilibili.com/video/BV1xT4EzyEgQ/
https://www.bilibili.com/video/BV1xT4EzyEgQ
https://www.bilibili.com/video/BV1xT4EzyEL7/
https://www.bilibili.com/video/BV1xT4EzyEL7
https://www.bilibili.com/video/BV1xT4EzyEQ2/
https://www.bilibili.com/video/BV1xT4EzyEQ2
https://www.bilibili.com/video/BV1xT4EzyEQk/
https://www.bilibili.com/video/BV1xT4EzyEQk
https://www.bilibili.com/video/BV11T4EzyEzo/
https://www.bilibili.com/video/BV11T4EzyEzo
https://www.bilibili.com/video/BV11T4EzyEZ1/
https://www.bilibili.com/video/BV11T4EzyEZ1
https://www.bilibili.com/video/BV1xT4EzyEwH/
https://www.bilibili.com/video/BV1xT4EzyEwH
https://www.bilibili.com/video/BV1xT4EzyE3T/
https://www.bilibili.com/video/BV1xT4EzyE3T
https://www.bilibili.com/video/BV1xT4EzyEKQ/
https://www.bilibili.com/video/BV1xT4EzyEKQ
https://www.bilibili.com/video/BV1xT4EzyE2z/
https://www.bilibili.com/video/BV1xT4EzyE2z
https://www.bilibili.com/video/BV11T4EzyEoX/
https://www.bilibili.com/video/BV11T4EzyEoX
https://www.bilibili.com/video/BV12T4EzyEL2/
https://www.bilibili.com/video/BV12T4EzyEL2
https://www.bilibili.com/video/BV1sT4EzyEop/
https://www.bilibili.com/video/BV1sT4EzyEop
https://www.bilibili.com/video/BV1xT4EzyE5f/
https://www.bilibili.com/video/BV1xT4EzyE5f
https://www.bilibili.com/video/BV1xT4EzyEgv/
https://www.bilibili.com/video/BV1xT4EzyEgv
https://www.bilibili.com/video/BV1xT4EzyEQa/
https://www.bilibili.com/video/BV1xT4EzyEQa
https://www.bilibili.com/video/BV1xT4EzyEgr/
https://www.bilibili.com/video/BV1xT4EzyEgr
https://www.bilibili.com/video/BV1sT4EzyEot/
https://www.bilibili.com/video/BV1sT4EzyEot
https://www.bilibili.com/video/BV11u4EzVEe6/
https://www.bilibili.com/video/BV11u4EzVEe6
https://www.bilibili.com/video/BV1xu4EzVEN2/
https://www.bilibili.com/video/BV1xu4EzVEN2
https://www.bilibili.com/video/BV1su4EzVEsM/
https://www.bilibili.com/video/BV1su4EzVEsM
https://www.bilibili.com/video/BV1xu4EzVEKF/
https://www.bilibili.com/video/BV1xu4EzVEKF
https://www.bilibili.com/video/BV1su4EzVE3M/
https://www.bilibili.com/video/BV1su4EzVE3M
https://www.bilibili.com/video/BV1su4EzVEsJ/
https://www.bilibili.com/video/BV1su4EzVEsJ
https://www.bilibili.com/video/BV11u4EzVEeA/
https://www.bilibili.com/video/BV11u4EzVEeA
 
    # 计算相似度
    return [cosine_similarity(orig, noise) for orig, noise in zip(outputs_original, outputs_noise)]
    
# 执行验证
similarities = compare_noise_impact(float_model_path="float_model.onnx")
for i, sim in enumerate(similarities):
    print(f"输出{i}余弦相似度: {sim:.6f}")


分享到：
收藏 邀请回答 回复楼主 举报


楼主最近还看过

SSI ļʱ