The only prompt compressor that's lossless, model-free, and built for structured data.
No inference cost, no hallucinations, no byte-level tricks an LLM can't see -- just fewer tokens for the same payload.
27-69% savings on JSON / YAML / CSV / logs.
PAKT (Pipe-Aligned Kompact Text) is a lossless-first compression format that converts JSON, YAML, CSV, and mixed markdown content into a compact pipe-delimited syntax optimized for LLM token efficiency. Structured payloads often see 30-50% token savings, with higher gains on repetitive and tabular data, while preserving data fidelity across core lossless layers L1-L3. L4 is separately opt-in, budgeted, and lossy.
LLMs charge by the token. Structured data wastes tokens on syntax: braces, quotes, repeated keys, whitespace. PAKT eliminates the waste.
- LLMLingua / LLMLingua-2? Neural compressors. They run a model to rewrite your prompt, which is lossy, model-dependent, and adds inference cost and latency. PAKT is deterministic, model-free, and free to run.
- TOON format? TOON is the core inspiration for PAKT's Layer 1 pipe-delimited syntax. PAKT extends it with a dictionary layer (L2), tokenizer-aware packing (L3), delta encoding for tabular arrays, multi-format input (JSON/YAML/CSV/Markdown/Text), and an MCP server for agents.
- gzip / brotli? They compress bytes, but the LLM API bills you on tokens after BPE tokenization. A gzipped prompt still costs full tokens once decoded. PAKT reshapes the text so the tokenizer itself produces fewer tokens.
- Just minify JSON? Free and worth doing -- but it only removes whitespace. PAKT minifies, then layers dictionary substitution and tokenizer-aware choices on top, typically doubling the savings.
ClipForge is the product suite built around PAKT. In this repository, that means:
- @sriinnu/pakt -- The core library, CLI, and MCP server. This is the stable release surface for Node.js and TypeScript projects, plus agent hosts that need stdio tools for compress, auto, and inspect.
- ClipForge Playground -- A lightweight local web UI for trying JSON, YAML, CSV, and mixed markdown compression before wiring PAKT into a real workflow. It is a browser lab, not a release integration. Hosted playground: pakt-4f9.pages.dev.
- ClipForge Desktop -- A Tauri desktop shell for clipboard compression workflows. The current release validation is macOS menu bar first; Windows and Linux tray targets exist in source but are not part of the validated release path yet. The 0.9 frontend is hand-split into vendor / pakt-core / tauri-api / app chunks plus lazy-loaded Settings and History overlays so vendor code caches independently across releases.
- ClipForge Browser Extension (experimental) -- A Chrome extension with a popup, a dedicated full-tab Options page, context-menu actions, and input helpers for supported web LLM UIs: ChatGPT, Claude, Gemini, Slack web, and Gmail. Includes opt-in auto-compress-on-paste with a per-host allowlist.
The goal is simple: every token you send to an LLM should carry meaning, not syntax.
For agent workflows, the MCP server is the integration bridge. pakt serve --stdio exposes pakt_compress, pakt_auto, and pakt_inspect through the standard MCP transport, so stdio-based MCP clients can call the same toolset without custom protocol glue. The generic stdio path is verified in-repo; named hosts like Claude Desktop and Cursor are integration targets rather than a certification matrix. pakt_inspect is the recommended first call when deciding whether compression is worth it.
The app surfaces now align on shared layer profiles: Structure only (L1), Standard (L1+L2), Tokenizer-aware (L1+L2+L3), and opt-in Semantic (L1+L2+L3+L4). Semantic mode requires a positive semanticBudget and is explicitly lossy.
JSON (28 tokens) PAKT (15 tokens)
------------------------------ --------------------------
{ @from json
"users": [ @dict
{ "name": "Alice", $a: dev
"role": "dev" }, @end
{ "name": "Bob",
"role": "dev" } users [2]{name|role}:
] Alice|$a
} Bob|$a
This is a pnpm workspace monorepo.
clipforge-PAKT/
packages/
pakt-core/ Core compression engine, CLI, and MCP server
apps/
playground/ Local web playground for trying PAKT inputs
desktop/ ClipForge tray app (Tauri v2 + React)
extension/ Experimental Chrome extension for supported LLM UIs
docs/ Format spec and guides
assets/
pakt-logo.svg Logo assets
| Package | npm | Description |
|---|---|---|
@sriinnu/pakt |
 |
PAKT compression engine -- the core library with API and CLI |
npm install @sriinnu/pakt@sriinnu/pakt supports Node 18+. Monorepo development for this repository uses Node 22+.
import { compress, decompress, detect } from '@sriinnu/pakt';
// Compress JSON to PAKT
const result = compress('{"users": [{"name": "Alice", "role": "dev"}, {"name": "Bob", "role": "dev"}]}');
console.log(result.compressed);
console.log(`Saved ${result.savings.totalPercent}% tokens`);
// Decompress back to JSON
const original = decompress(result.compressed, 'json');
console.log(original.text);
// Detect input format
const detected = detect('name,role\nAlice,dev');
console.log(detected.format); // 'csv'import { estimateCompressibility } from '@sriinnu/pakt';
const score = estimateCompressibility(myJson);
console.log(score.score); // 0.72
console.log(score.label); // 'high'
console.log(score.profile); // 'tokenizer' — recommended layer profile
console.log(score.breakdown); // { repetitionDensity, structuralOverhead, schemaUniformity, valueLengthScore }Delta encoding activates automatically on tabular arrays with 30%+ repeated adjacent values. No code change needed — compress() applies it as a post-pass on L1.
@from json
@compress delta
users [5]{name|role|dept|city}:
Alice|engineer|platform|NYC
Bob|~|~|~
Charlie|~|~|SF
Diana|designer|product|~
Eve|~|~|~
The ~ sentinel replaces unchanged fields. Fully reversible via decompress().
See the pakt-core README for comprehensive API documentation, CLI usage, format specification, and examples.
Core CLI example for opt-in lossy packing:
npx @sriinnu/pakt compress data.json --semantic-budget 120Release-facing benchmark numbers live in docs/BENCHMARK-SNAPSHOT.md.
For LLM round-trips, the core package now also exposes interpretModelOutput() so your app can auto-detect PAKT in a model response, repair minor syntax issues, and decompress valid replies back to JSON/YAML/CSV.
Try the hosted playground: pakt-4f9.pages.dev.
From the repo root, you can install, build, and boot each surface directly:
pnpm install
pnpm build
pnpm build:all
pnpm build:core
pnpm build:playground
pnpm build:extension
pnpm build:desktop:web
pnpm build:desktop
pnpm build:apps
pnpm test:core
pnpm test:playground
pnpm dev:playground
pnpm dev:extension
pnpm dev:desktop:web
pnpm dev:desktop
pnpm start:mcpLocal surface entrypoints:
pnpm dev:playground # local playground
pnpm dev:extension # extension dev build
pnpm dev:desktop:web # desktop frontend only
pnpm dev:desktop # real Tauri desktop shell
pnpm start:mcp # core MCP server over stdioPlayground notes for release testing:
- Mixed-content restores embedded structured blocks semantically; exact original formatting may normalize.
- CSV is not always a win; some already-compact CSV can expand.
- Compare mode now includes an auto-pack lab; table-aware variants unlock for top-level CSV and top-level JSON arrays.
- The playground runs locally in the browser session and does not upload payloads.
- For mixed-content decompress, paste the PAKT-marked output back into the input area, then run
Decompress.
CLI/MCP note:
semanticBudgetnow cleanly opts into lossyL4; if you stay onL1-L3, the pipeline remains lossless.pakt serve --stdionow uses the official MCP SDK stdio transport, and embedders can register the same tools programmatically viaregisterPaktTools().
Track token savings across sessions with persistent, multi-agent support.
MCP tool: pakt_stats returns compression metrics. Use scope: 'session' for the current process (fast, default) or scope: 'all' to aggregate across all agents from disk.
CLI: pakt stats has two modes:
pakt stats data.json # single-shot stats for one file
pakt stats # aggregate from persistent storage
pakt stats --today # filter to today
pakt stats --week # filter to last 7 days
pakt stats --agent research # filter by agent name
pakt stats --active # only running agents
pakt stats --compact # archive old sessions
pakt stats --reset # clear all statsNamed agents: pakt serve --stdio --agent-name research names the session for filtering.
Stats are persisted as per-agent JSONL files in ~/.pakt/stats/. Each MCP server writes to its own file -- zero contention across 10+ concurrent agents. Old sessions are lazily compacted into daily summaries.
PAKT automatically compresses data on every MCP tool call to reduce conversation context size. Compressed data stays in context and saves tokens on every subsequent turn.
- Content-addressed dedup -- SHA-256 hash cache (10MB byte-budget LRU) avoids re-compressing identical data
- Text compression -- line dedup + word n-gram dictionary. 57% savings on logs, 38% on repetitive text, 69% on identical lines
- Whitespace normalization -- trailing spaces, blank line runs stored as metadata for lossless restoration
- Configurable tool description --
registerPaktTools(server, { autoDescription: '...' })controls how aggressively the LLM uses auto-compression - Safe detection --
@from Johnand@warning chemicalscorrectly detected as text, not PAKT. No false positives on@username,@Override, email headers - Input size caps -- 512KB for auto, 1MB for explicit compress, 100KB for text compression. Prevents CPU DoS
| Input Type | Savings | Round-trip |
|---|---|---|
| JSON 10 records | 27% | Lossless |
| JSON 50 records | 33% | Lossless |
| Log lines (duplicates) | 57% | Lossless |
| Repetitive text | 38-69% | Lossless |
| Normal prose (no repetition) | 0% (passthrough) | Safe |
- Lossless by default -- L1 (Structural), L2 (Dictionary), and L3 (Tokenizer-Aware) round-trip exactly. L4 (Semantic) is opt-in and explicitly lossy, gated by a
semanticBudget. - Multi-format -- JSON, YAML, CSV, Markdown, and Plain Text with auto-detection, so the same pipeline handles structured, tabular, and mixed content.
- Tokenizer-aware via real BPE -- Uses
gpt-tokenizerto measure and pick tokens LLMs actually merge, not byte-level heuristics that vanish at the API boundary. - MCP + CLI + library, zero model dependency --
pakt serve --stdioexposespakt_compress,pakt_auto,pakt_inspect, andpakt_statsto agents; the same logic ships as a CLI and a typed TypeScript library. - 27-69% savings with public benchmarks -- 27-33% on JSON payloads, 57% on duplicate log lines, 38-69% on repetitive text. Full reproducible numbers in docs/BENCHMARK-SNAPSHOT.md.
git clone https://github.com/sriinnu/clipforge-PAKT.git
cd clipforge-PAKT
pnpm install
# Opt in to repo-managed git hooks (rejects Co-authored-by trailers, etc.)
git config core.hooksPath .githooks# Build all packages
pnpm build
# Run all tests
pnpm test
# Run tests in watch mode
pnpm test:watch
# Run benchmarks
pnpm bench
# Clean build artifacts
pnpm cleanPAKT's core pipe-delimited syntax (Layer 1) is directly inspired by TOON Format v1.3 -- the original compact notation for structured data, created by Nicholas Charlton (@nichochar). TOON demonstrated that structured data can be represented without the syntactic overhead of JSON while remaining unambiguous and machine-parseable. PAKT builds on this foundation by adding multi-format support, a dictionary compression layer, and guaranteed lossless round-tripping. TOON has implementations across Python, TypeScript, Go, Rust, .NET, Elixir, Java, and Julia -- a testament to the strength of its design.
- LLMLingua-2 (Microsoft, arXiv:2403.12968, 2024) -- The closest neural competitor. PAKT achieves comparable savings on structured data without running a model.
- Gist Token Study (Deng et al., arXiv:2412.17483, Dec 2024) -- Lossy compression fails on exact recall, which motivates PAKT's lossless-first L1-L3 design.
- DeltaKV (Hao et al., arXiv:2602.08005, Feb 2026) -- Residual similarity compression, adapted as PAKT's delta encoding for tabular arrays.
Full citation list and survey: docs/research.md.
PAKT sits alongside a couple of sibling tools that round out the token economy — shrink, measure, monitor:
- tokmeter -- Token usage tracker for AI coding agents. Tracks consumption across 16+ agents through five surfaces: CLI, TUI, React web dashboard, MCP server (Drishti), and a macOS menu bar. Includes cost digests, cache-efficiency analytics, and a model advisor.
- Runic -- macOS menubar app monitoring AI usage, costs, and quotas across 26 providers in real time. Charts, forecasts, budget alerts, CSV/JSON export, widgets, and a bundled CLI. Local-only, zero telemetry.
PAKT reduces what you send; tokmeter and Runic tell you what you spent.
PAKT is maintained independently. If it saves you tokens and you'd like to back continued development, you can sponsor via GitHub Sponsors or Buy Me a Coffee. Issues, PRs, and stars help just as much.
MIT -- Srinivas Pendela