Lightweight transformer inference engine for Qwen3 0.6B, designed for both host development and baremetal RISC-V deployment with hardware acceleration via a 16x16 INT8 systolic array.
- Qwen3 0.6B model with INT8 quantization (604MB)
- 28 transformer layers, 16 heads, 1024 dimensions, 32K vocabulary
- Header-only C implementation for maximum portability
- Arena-based memory allocation (no malloc/free on baremetal)
- 16x16 INT8 systolic array hardware acceleration (GEMMA on VEGA AT1051)
- Dual-target: host (x86/ARM) and baremetal (RISC-V)
The following model files are required but not included in the repository:
model_q8.bin(604MB) - Quantized INT8 model weightsmodel_q8.bin.tokenizer(403KB) - BPE tokenizer vocabulary
Generate these files using the export scripts in scripts/:
cd scripts
uv sync
uv run export_q8.py ../model_q8.bin Qwen/Qwen3-0.6B --vocab-size 32000See scripts/README.md for more options.
just hc # Build and run on hostjust b # Build for baremetal RISC-V (requires Docker)qwen3c/
├── main.c # Entry point, model loading, tokenizer, inference loop
├── include/
│ ├── config.h # Build configuration, memory layout, feature flags
│ ├── platform.h # Platform abstraction (host/baremetal detection)
│ ├── memory.h # Arena-based memory management (3 arenas)
│ ├── types.h # Core data structures (tensors, model config)
│ ├── nn.h # Neural network operations, transformer forward pass
│ ├── gemma.h # Hardware accelerator interface (systolic array)
│ ├── log.h # Logging macros and performance timing
│ ├── baremetal.h # UART, printf, libc implementations for baremetal
│ ├── encoding.h # RISC-V CSR encoding definitions
│ ├── tests.h # Test framework dispatcher
│ ├── test_diagnostic.h # Early system validation tests
│ ├── test_accelerator.h # Hardware accelerator tests
│ ├── test_memory.h # Memory management tests
│ ├── test_gemma.h # GEMMA IP performance tests
│ └── test_debug.h # Debug system tests
├── crt.S # C runtime startup for baremetal
├── mbl.lds # Linker script for baremetal memory layout
├── justfile # Build commands (just hc, just b)
├── build.sh # Baremetal build script (Docker)
├── makefile # Alternative build system
├── scripts/ # Python scripts for model export
│ ├── export_q8.py # Q8_0 quantization and vocabulary pruning
│ ├── model.py # Qwen3 model architecture
│ └── pyproject.toml # Python dependencies
├── build/ # Build artifacts (baremetal ELF, bin, dump)
└── logs/ # Sample execution logs
| Region | Address | Size | Purpose |
|---|---|---|---|
| Application | 0x80000000 | 4MB | Code and static data |
| General Arena | 0x80400000 | 248MB | Runtime allocations, activations |
| Cache Arena | 0x8FC00000 | 128MB | KV cache (persistent) |
| Accelerator | 0x97C00000 | 8MB | Systolic array workspace (non-cacheable) |
| Model Data | 0x98400000 | 635MB | Pre-loaded model weights (read-only) |
| Tokenizer | 0xBFF00000 | 1MB | Vocabulary data (read-only) |
The engine integrates with a 16x16 INT8 systolic array accelerator (GEMMA) for matrix multiplication:
- Accelerator registers memory-mapped at
0x20060000 - Systolic array workspace at
0x97C00000(non-cacheable) - Intelligent routing: hardware used when
total_ops >= 65536,d >= 16,group_size >= 64 - Automatic fallback to software on timeout or failure
[INFO] === Q3INF INFERENCE ENGINE v0.13 ===
[INFO] STAGE 4/7: Loading model and initializing runtime...
[ACCL] Hardware accelerator: HOST BUILD (software only)
[ NN ] CONFIG: layers=28 heads=16 dim=1024 vocab=32000
[INFO] User prompt: What is 2 + 2?
[STEP] === STARTING INFERENCE ===
[PERF] Inference started
[ NN ] Forward pass pos=0, token=31975
[ NN ] Progress: [ 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 ]
[PERF] LAYER AVG: 5.848ms
[STEP] === RESPONSE GENERATION ===
[ TK ] Token 17 -> '2'
[ TK ] Token 488 -> ' +'
[ TK ] Token 220 -> ' '
[ TK ] Token 17 -> '2'
[ TK ] Token 284 -> ' ='
[ TK ] Token 220 -> ' '
[ TK ] Token 19 -> '4'
[STEP] === FINAL RESPONSE ===
[INFO] >>> 2 + 2 = 4
[PERF] Inference timing breakdown:
[PERF] Pure inference: 28 tokens in 555.098ms (50.4 tok/s)
booting
[INFO] === Q3INF MAIN ENTRY ===
[INFO] STAGE 1/7: System initialization...
[INFO] === Q3INF INFERENCE ENGINE v0.13 ===
[INFO] Setting up non-cacheable region: 0x97c00000 - 0xc0000000
[ACCL] Detecting GEMMA accelerator at 0x20060000
[ACCL] GEMMA accelerator detected and configured successfully
[ACCL] Hardware accelerator: ENABLED
[ACCL] Implementation: GEMMA VEGA AT1051 (16x16 Systolic Array)
[INFO] [BM] Using pre-loaded model at 0x98400000 (635MB)
[ NN ] CONFIG: layers=28 heads=16 dim=1024 vocab=32000
[INFO] Hardware acceleration: ENABLED (GEMMA VEGA AT1051 (16x16 Systolic Array))
[INFO] User prompt: What is 2 + 2?
[STEP] === STARTING INFERENCE ===
[ NN ] Forward pass pos=0, token=31975
[ NN ] Progress: [ 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 ]
[STEP] === RESPONSE GENERATION ===
[ TK ] Token 17 -> '2'
[ TK ] Token 488 -> ' +'
[ TK ] Token 220 -> ' '
[ TK ] Token 17 -> '2'
[ TK ] Token 284 -> ' ='
[ TK ] Token 220 -> ' '
[ TK ] Token 19 -> '4'
[STEP] === FINAL RESPONSE ===
[INFO] >>> 2 + 2 = 4
[INFO] === Q3INF INFERENCE ENGINE COMPLETE ===
Key defines in config.h:
| Define | Description |
|---|---|
HOST_BUILD |
Defined automatically for host compilation |
DEBUG_MODE |
Enable verbose logging |
ENABLE_INFERENCE |
Enable token generation |
ENABLE_TESTS |
Enable test framework |
CONTEXT_LENGTH |
Maximum sequence length (default: 64) |
USE_ACCELERATOR |
Enable hardware acceleration (baremetal only) |
- Host build: GCC or Clang with C99 support
- Baremetal build: Docker (contains RISC-V toolchain)
- Model export: Python 3.11+, uv package manager
MIT