27.9KB language-agnostic¹ WebGPU syntax highlighter

Shu Ding at Vercel Labs

import { parse } from 'gpu-lexer'

const spans = await parse('source code')
// {
//   type: 'plain' | 'comment' | 'string' | 'number' | 'keyword' | 'type' | 'function' | 'constant' | 'operator'
//   start: number
//   end: number
// }[]

gpu-lexer splits source code into simple parts—words, whitespace, newlines, and symbols. Then a tiny WebGPU model combines local and whole-file context to label each part. It is designed for any language: instead of choosing a grammar, it guesses each part's type from the surrounding source, even when it never saw that language or syntax during training. Adjacent labels become the syntax spans returned to your code.

This is an experiment, not a grammar-equivalent highlighter (see the agreement chart below). The chart measures agreement with Shiki—not objective correctness—and results may vary on unseen languages or real-world code.

¹ "Language-agnostic" means one shared tokenizer and classifier, not equal accuracy for every language. 83.02% is the share of held-out token labels that matched Shiki. Mixed-language code is supported too, including embedded <script> and <style> regions in HTML, Vue, and Svelte.

Held-out label agreement with Shiki by language
agreement	verified languages (98)
95–100%	angular-html, angular-ts, apex, jsx
90–<95%	actionscript, actionscript-3, cython, dart, go, hcl, javascript, proto, protobuf, python, rust, solidity, svelte, swift, toml, tsx
80–<90%	astro, c, cmake, coffeescript, cpp, csharp, css, docker, gdscript, glsl, haxe, hlsl, html, java, json, odin, php, plist, powershell, ruby, shell, shellscript, typescript, wgsl, xml, xsl
70–<80%	bat, bicep, c3, fortran-fixed-form, haskell, kotlin, make, nginx, nim, nix, objective-cpp, r, rss, scala, vue, yaml, zig
60–<70%	ada, d, diff, fortran-free-form, groovy, hack, less, lua, markdown, matlab, prisma, puppet, qml, scss, system-verilog, systemverilog
50–<60%	asm, awk, emacs-lisp, gherkin, graphql, julia, mdx, objective-c, perl, shaderlab, sql, viml, wasm, webassembly
<50%	jinja, liquid, razor, vala, vb
training epoch	4,679,585 tokens
model	41,321 parameters

Each language appears in one band; results with fewer held-out labels are less stable. Training tokens include context-only tokens and replay.

Highlight 10× three.min.jswarmed browser time · lower is better

Highlights75.3ms

gpu-lexer220.5ms

Sugar High820.0ms

Prism.js1.24s

Highlight.js1.36s

Starry Night11.4s

Shiki30.4s

Median of three samples on October 22, 2026. Each sample ran in a fresh dedicated worker and timed one call after one warm-up call; the engine order rotated between rounds. The input was 10 concatenated copies of three.min.js (5.56M characters). MacBook Pro, Apple M4 Pro, 20-core GPU, 24GB, macOS 26.7.1, headless Chrome 156. DOM rendering was excluded. gpu-lexer returned spans, Highlights and Shiki returned themed tokens, Starry Night returned a HAST tree, while Sugar High, Prism.js, and Highlight.js returned highlighted HTML. Highlights 0.2.0, Sugar High 2.3.1, Prism.js 1.30.0, Highlight.js 11.12.0, Starry Night 3.11.0, and Shiki 4.4.3.

Loaded library sizeruntime + selected language coverage · lower is better

Sugar High (major 6 web languages)5.4KB

Prism.js (major 6 web languages)8.6KB

Sugar High (all 29 languages)8.8KB

Highlight.js (major 6 web languages)14.9KB

gpu-lexer (one model)27.9KB

Highlights (all 80 languages)95.9KB

Prism.js (all 297 languages)162.1KB

Starry Night (major 6 web languages)185.3KB

Shiki (major 6 web languages)213.9KB

Highlight.js (all 193 languages)240.4KB

Shiki (all 242 grammars)991.5KB

Starry Night (all 710 grammars)1.46MB

Minified and Brotli-compressed browser bundles, all measured on October 22, 2026. Major web includes javascript, typescript, css, html, json, and markdown. gpu-lexer uses the same bundle for every language. Starry Night totals include its Oniguruma WASM payload. Highlights ships every lexer in one WebAssembly module; themes are excluded.

Top-25 weighted agreementpopularity-weighted agreement with Shiki · higher is better

Shiki100%

gpu-lexer86.77%

Prism.js84.05%

Sugar High73.61%

Starry Night72.25%

Highlight.js70.12%

Shiki is the 100% normalization reference. Each library's token names are mapped to the same nine classes: plain, comment, string, number, keyword, type, function, constant, and operator. Scores compare non-whitespace source parts across 1,103 held-out files in the GitHub Innovation Graph top 25 for 2026-Q1, weighted by each language's pusher count. Unsupported languages score zero; corpus size does not affect the weights.

[ live demos ]

75 languages

react.development.js0 bytes

https://unpkg.com/react@19.2.8/cjs/react.development.js—

27.9KB language-agnostic1 WebGPU syntax highlighter

27.9KB language-agnostic¹ WebGPU syntax highlighter