This folder contains Olive recipes for optimizing Qwen-Qwen3.5-2B targeting the CUDA EP.
- Execution Provider: CUDA EP
- Typical precision: INT4 precision by default
- Example recipe filename: Qwen-Qwen3.5-2B_cuda_int4.json
- Install the main branch of Olive:
- pip install git+https://github.com/microsoft/olive.git
- Install the appropriate runtime package for this backend:
- onnxruntime-genai-cuda (CUDA build)
- Run Olive to build/optimize the model
- olive run --config Qwen-Qwen3.5-2B_cuda_int4.json
Additional notes:
- Pipeline:
SelectiveMixedPrecision(kld_gradient) →GPTQ→RTN(8-bit lm_head/embeddings) →ModelBuilder→TieWordEmbeddings - GPTQ group size: 128
- Uses text-only mode (exclude_embeds=false, prune_lm_head=true) for standalone LLM inference without multimodal pipeline.
- Requires NVIDIA GPU with CUDA support.
- Ensure CUDA toolkit and cuDNN are properly installed.
This README was auto-generated for the CUDA EP of Qwen-Qwen3.5-2B.