Skip to content

README.md

Latest commit

 

History

History
72 lines (50 loc) · 1.86 KB

README.md

File metadata and controls

72 lines (50 loc) · 1.86 KB
Raw

Q3INF Model Export Scripts

Python scripts for exporting Qwen3 models to the Q8_0 quantized binary format used by q3inf.

Requirements

  • Python 3.11+
  • uv package manager (recommended)

Setup

cd scripts
uv sync

Export Model

Export Qwen3-0.6B with optimized 32K English vocabulary:

uv run export_q8.py ../model_q8.bin Qwen/Qwen3-0.6B --vocab-size 32000

This generates:

  • model_q8.bin (604MB) - Quantized INT8 model weights
  • model_q8.bin.tokenizer (403KB) - BPE tokenizer vocabulary

Options

export_q8.py <output_path> <hf_model_path> [options]

Arguments:
  output_path     Output binary file path
  hf_model_path   HuggingFace model path (e.g., Qwen/Qwen3-0.6B)

Options:
  --vocab-size N  Target vocabulary size for pruning (default: full vocab)
  --group-size N  Quantization group size (default: 64)

Examples

# Export with pruned vocabulary (recommended for q3inf)
uv run export_q8.py ../model_q8.bin Qwen/Qwen3-0.6B --vocab-size 32000

# Export with full vocabulary (larger file, ~151K tokens)
uv run export_q8.py ../model_q8_full.bin Qwen/Qwen3-0.6B

# Export with different group size
uv run export_q8.py ../model_q8_g128.bin Qwen/Qwen3-0.6B --vocab-size 32000 --group-size 128

Files

File Description
export_q8.py Main export script with Q8_0 quantization and vocabulary pruning
model.py Qwen3 model architecture definition
pyproject.toml Python dependencies

Vocabulary Pruning

The --vocab-size option enables intelligent vocabulary pruning that:

  • Preserves all special tokens (BOS, EOS, etc.)
  • Prioritizes English function words and common patterns
  • Keeps essential punctuation and formatting tokens
  • Maintains number tokens for arithmetic

This reduces the model size while preserving English language capability.