🇨🇳 pizza-analysis-stconvert

Simplified/Traditional Chinese conversion plugin for INFINI Pizza

---

High-performance Simplified/Traditional Chinese converter for pizza-engine.

Features

• 7 conversion directions: T2S, S2T, TW2CN, CN2TW, T2HK, HK2T, JP2T
• 57,587 dictionary entries from OpenCC + supplementary data
• Zero-allocation fast path for ASCII content (>3 GB/s)
• ~18 MB/s throughput on pure CJK text (Apple Silicon, release build)
• Buffer reuse API (convert_to) for amortized zero-allocation in hot loops
• Cow API for zero-copy passthrough when input needs no conversion
• Optional pizza-engine integration (Normalizer + Tokenizer + TokenFilter)

Performance

Benchmarked on Apple Silicon (M-series), --release, 1 MB input, 100 iterations:

Workload	Throughput	Notes
CJK conversion (T2S)	~18 MB/s	Unified lookup + hash-based prefix
ASCII passthrough	>3 GB/s	Byte-level fast scan
Mixed CJK+ASCII	~50+ MB/s	Depends on ASCII ratio

Design choices for speed

1. Fast ASCII bypass: scans raw bytes; ASCII spans are bulk-copied without char decoding or dictionary lookup.
2. Unified lookup table: single binary search returns both the char mapping and the multi-start flag (replaces two separate lookups).
3. FxHash prefix check: instead of binary searching a &[&str] table (expensive pointer chasing), prefix membership uses a sorted array of pre-computed 64-bit hashes — integer comparison only.
4. Stack-resident buffer: greedy multi-char matching uses a fixed 48-byte [u8; 48] buffer on the stack (no heap allocation).
5. Build-time code generation: all tables are generated by build.rs as sorted static arrays for cache-friendly binary search.

Usage

use pizza_stconvert::{convert, ConvertType};

// Traditional → Simplified
assert_eq!(convert("計算機科學與技術", ConvertType::T2S), "计算机科学与技术");

// Simplified → Traditional
assert_eq!(convert("计算机科学与技术", ConvertType::S2T), "計算機科學與技術");

Buffer reuse (high-throughput)

use pizza_stconvert::{STConverter, ConvertType};

let converter = STConverter::new(ConvertType::T2S);
let mut buf = String::new();

for text in &["傳統", "簡體", "中文"] {
    buf.clear();
    converter.convert_to(text, &mut buf);
    println!("{text} → {buf}");
}

Zero-copy check

use pizza_stconvert::{STConverter, ConvertType};

let converter = STConverter::new(ConvertType::T2S);
let result = converter.convert_cow("hello world"); // returns Cow::Borrowed

Conversion directions

Type	From	To
T2S	Traditional Chinese	Simplified Chinese
S2T	Simplified Chinese	Traditional Chinese
TW2CN	Taiwan vocab	Mainland vocab
CN2TW	Mainland vocab	Taiwan vocab
T2HK	Traditional	Hong Kong variant
HK2T	Hong Kong variant	Traditional
JP2T	Japanese Shinjitai	Traditional Chinese

pizza-engine integration

With the engine feature (enabled by default), the crate provides three analysis components for use in pizza-engine pipelines:

• STConvertNormalizer — normalizes the full input text in-place (Simplified ↔ Traditional) before tokenization.
• STConvertTokenizer — converts the entire input and emits it as a single token.
• STConvertTokenFilter — converts each token's term individually. Supports keep_both mode to emit both original and converted forms.

use pizza_stconvert::{STConvertNormalizer, ConvertConfig, ConvertType};
use pizza_engine::analysis::Normalizer;

let normalizer = STConvertNormalizer::new(ConvertConfig::new(ConvertType::T2S));
let mut text = String::from("計算機科學與技術");
normalizer.normalize(&mut text);
assert_eq!(text, "计算机科学与技术");

Feature flags

• engine (default): enables STConvertNormalizer, STConvertTokenizer, and STConvertTokenFilter for integration with pizza-engine analysis pipelines.

License

Apache-2.0