Hardware-accelerated subword tokenizer with automatic AVX2/512 SIMD, CUDA, and ROCm HIP execution. Instant cartridge vocabulary swapping without rebuilding tokenizer states.
Replaces pointer-heavy lookups with aligned BASE, CHECK, and VALUES arrays compiled into binary zero-copy memory-mapped (.dat) cartridges.
Scans 32 bytes per cycle. Bypasses UTF-8 overhead on pure ASCII, enabling loop unrolling and ultra-low latency lookups.
Maps document sequences directly to GPU threads with dynamic capacity management, avoiding out-of-memory errors on massive batches.
Zero-copy mmap loading of DAT files allows hot-swapping vocabulary profiles in sub-millisecond timeframes.
Easily initialize the tokenizer with automatic hardware detection and swap between standard and lite profile cartridges.
from crayon import CrayonVocab
# Auto-detect best engine (AVX2/CUDA/HIP)
tokenizer = CrayonVocab(device="auto")
# 1. Load Standard 206k Cartridge Profile
tokenizer.load_profile("standard")
print(f"--- Turbo Engine: {tokenizer._turbo_backend.get_hardware_info() if tokenizer._turbo_backend else tokenizer.device} ---")
tokens_std = tokenizer.tokenize("that is a test for the standard profile and lite profile and god")
print(f"Standard Tokens: {tokens_std}")
print(f"Decoded: {tokenizer.decode(tokens_std)}")
# 2. Instantly Swap to Lite 50k Cartridge Profile
tokenizer.load_profile("lite")
tokens_lite = tokenizer.tokenize("my daughter")
print(f"Lite Tokens: {tokens_lite}")
print(f"Decoded: {tokenizer.decode(tokens_lite)}")
Diverse multi-domain corpus | vocab._turbo_backend | Minimal repetition | GC enabled | vs tiktoken cl100k
| Size | Tokenizer | Tokens | Throughput | MB/s | Latency (ms) |
|---|---|---|---|---|---|
| 0.5 KB | Crayon Turbo | 243 | 73.94 M tok/s | 148.6 | 0.00 ± 0.00 |
| 0.5 KB | tiktoken cl100k | 129 | 1.74 M tok/s | 6.6 | 0.07 ± 0.03 |
| 1 KB | Crayon Turbo | 475 | 82.60 M tok/s | 169.8 | 0.01 ± 0.00 |
| 1 KB | tiktoken cl100k | 258 | 1.55 M tok/s | 5.9 | 0.17 ± 0.07 |
| 16 KB | Crayon Turbo | 7,769 | 75.31 M tok/s | 151.5 | 0.10 ± 0.01 |
| 16 KB | tiktoken cl100k | 4,358 | 2.00 M tok/s | 7.2 | 2.18 ± 0.36 |
| 64 KB | Crayon Turbo | 31,661 | 67.62 M tok/s | 133.5 | 0.47 ± 0.05 |
| 64 KB | tiktoken cl100k | 17,858 | 1.94 M tok/s | 6.8 | 9.23 ± 0.12 |
| 1 MB | Crayon Turbo | 503,013 | 47.47 M tok/s | 94.9 | 10.60 ± 1.40 |
| 1 MB | tiktoken cl100k | 266,962 | 1.83 M tok/s | 6.9 | 145.95 ± 3.73 |
| 4 MB | Crayon Turbo | 2,037,057 | 47.73 M tok/s | 94.3 | 42.68 ± 1.97 |
| 4 MB | tiktoken cl100k | 1,054,219 | 1.31 M tok/s | 5.0 | 807.74 ± 231.40 |
pip install -U xerv-crayon==5.7.16
PYPI