Fastest
Tokenization

Zero-copy vocabulary swapping engine written in C++17. Eliminates preprocessing bottlenecks across heterogeneous GPU runtimes and SIMD clusters.

0.0M

Tokens / Sec

0.0MiB/s

Memory Throughput

<0.0s

100M Char Stress Test

Explore Pipeline

Engine Spec 01 — Sub-Word BPE

Vector Architecture

Linked-list Byte Pair Encoding (BPE) optimized in C++17. Seamlessly scales from AVX2/AVX-512 CPU execution to native GPU kernel concurrency.

CUDA / HIP

NVIDIA & AMD Acceleration

C++17

Lock-Free BPE Engine

< 200 µs

Kernel Launch Overhead

Launch Live Benchmarks

System Spec 02 — Zero-Copy Swapping

Cartridge System

Pre-compiled binary vocabulary profiles loaded instantly via zero-copy memory mapping (`mmap`). Switch model tokenizers in production with zero deserialization pause.

206,373

Multi-Domain Subwords

~5.23 MB

DAT Binary Footprint

0 ms

Deserialization Pause

View Cartridge Specs

System Spec 03 — 100M Character Stress Test

Extreme Telemetry

Evaluated against a massive 100 million character corpus (~95.37 MiB) in Tesla T4 GPU & bare-metal environments using the unified Double-Array Trie (DAT).

23.84 M

Tokens / Second

107.16

MiB / Second

0.8899 s

Total Process Time

Run Live Profiler

Deploy CRAYON

git clone https://github.com/Electroiscoding/CRAYON.git