Codon

A bidirectional compiler and translator that converts between Codon (.cdn) and JavaScript/React (ESTree / Babel AST).

Codon is designed from first principles to be maximally token-efficient for Large Language Models (LLMs), shrinking context window consumption, boosting inference generation speed, and eliminating boilerplate hallucination while maintaining 100% round-trip AST fidelity.

---

⚡ Quickstart

Codon is primarily used through the CLI, distributed on NPM as codon-lang.

1. Install codon-lang Globally with Bun

Install the package globally using Bun:

bun add -g codon-lang

Verify the installation:

codon --help

(Alternatively with npm: npm install -g codon-lang)

2. Bootstrap a Project

Bootstrap a new, ready-to-run Codon project using codon init:

# Bootstrap a new project in my-project/
codon init my-project
cd my-project

codon init scaffolds a complete ECMAScript module (ESM) application:

my-project/
├── codon.json          # Codon compiler configuration (src/ -> dist/, runtime auto-import)
├── package.json        # Configured with "type": "module" and build/dev/start scripts
└── src/
    └── index.cdn       # Starter entrypoint with sample Codon idioms

3. Build & Run

Compile your Codon source code into standard JavaScript and run it:

# Compile project (src/**/*.cdn -> dist/**/*.js)
codon build

# Run the compiled application
bun start
# or: bun dist/index.js

4. Development Watch Mode

Start the incremental compiler in watch mode during development:

codon build -w
# or: bun run dev

---

📚 Documentation

Comprehensive documentation is available in the docs/ directory:

• 🏗️ Multi-File Projects Guide: Complete manual for authoring multi-file applications, codon init, codon build (tsc-like compiler), codon.json, relative imports, npm packages, and watch mode.
• 🚀 Macros System Guide: Deep dive into Codon's Subtree-Hashing Macro System (M!), empirical Level-2 code clone compaction (dict.json), wyhash AST matching, and catalog.
• 📖 Syntax Specification: Complete language reference covering tacit trains, folds, monadic verbs, objects, classes, JSX, and control flow.
• ⚡ Runtime Reference: API reference for codon-runtime.ts combinators (fork, hook, atop, rank) and array primitives.
• 🛠️ Programmatic API: TypeScript guide for CodonToASTParser, CodonToJSCompiler, JSToCodonTranslator, and MacroDictionary.
• 💻 CLI Manual: Complete guide to the Codon command-line interface, scripts, and workflows.

---

Key Features

• Subtree-Hashing Macro System (M!name(...)):
  • Automatically identifies Level-2 code clones from an empirical dictionary of 721 popularity-ranked patterns mined across thousands of open-source repositories (dict.json).
  • Replaces 12–127 AST node subtrees (10–50 tokens) with compact 2–4 token macro calls (e.g., M!m_call_expr_11(items)), cutting boilerplate token consumption by 60%–85%.
  • Zero runtime penalty: expands directly to native JavaScript AST during compilation.
• Bidirectional Compilation:
  • Compile .cdn files to standard .js.
  • Translate .js and .jsx files to .cdn with automatic macro compaction.
  • 100% round-trip AST fidelity verified via Babel parser validation.
• Array-Oriented Strengths Translated to JS:
  • Tacit Trains: Forks (fork f g h) / (f g h) and Hooks (hook f g) / (f g).
  • Reductions & Scans: Sum fold +/, product fold */, scans +\, fold, scan.
  • Monadic Verbs: Monadic await @ expr and @: expr, tally #, reciprocal %, square root %:, absolute value |.
  • Right-to-Left order of operations (no operator precedence table needed).
  • Array Primitives: tally, range, head, tail, behead, curtail, take, drop, from, gradeUp, gradeDown.
• Modern Web & Fullstack Expressed in Codon:
  • ES6 Classes: class. Name [extends Super] do. ... end., constructors, methods, async methods, getters, setters.
  • ES Modules: Concise import. specifiers from 'mod' and export. declarations.
  • Concise Objects: Colon pairs O(key:val), property shorthand O(prop), prefix spread O(...rest).
  • Native Arrays: Brackets [], [elem], [a, b], and spreads [...prev, item].
  • Direct Definition Lambdas: {{params => expr}}, {{params => do. ... end.}}, {{fn ...}}.
  • React & JSX: Native support for tags, attributes, interpolations, and fragments <> ... </>.
• Zero-Dependency Runtime (codon-runtime.ts):
  • Implements Codon combinators and array verbs for running Codon idioms natively in any JS environment.

---

Token Efficiency: JavaScript vs Codon

Pattern	Modern JavaScript	Codon Notation	Token Savings
Random Array Index	Math.floor(Math.random() * items.length)	M!m_call_expr_11(items)	~75%
Prototype Inheritance	Child.prototype = Object.create(Parent.prototype);	M!m_expression_stmt_17(Child;Parent)	~65%
String Capitalization	word.charAt(0).toUpperCase() + word.slice(1)	M!m_binary_expr_32(word)	~75%
Monadic Await	const res = await fetch(url);	res =: @ (fetch url)	~50%
Object Shorthand & Spread	{ ...options, active: true, port }	O(...options;active:true;port)	~40%
Arrow Closure	const double = (x) => x * 2;	double =: {{x => (x * 2)}}	~50%
Average Fork	const avg = (arr) => sum(arr) / arr.length;	avg =: (fork sum div tally)	~40%
Sum Reduction	const total = nums.reduce((a, b) => a + b, 0);	total =: +/ nums	~70%
Numeric Vector	const primes = [2, 3, 5, 7, 11];	primes = 2 3 5 7 11	~50%
Function Declaration	function add(a, b) { return a + b; }	fn. add a,b do. return. (a + b) end.	~35%
Module Import	import { useState, useRef } from "react";	import. {useState,useRef} from 'react'	~45%

---

Usage

Build Multi-File Project (build)

Compiles an entire project directory from src/ to dist/ with directory hierarchy preservation, import specifier rewriting, static asset copying, and runtime bundling:

# Full build
codon build

# Watch mode for fast incremental recompilation
codon build -w

# Clean build
codon build --clean

Single File Compilation (compile)

Converts a .cdn file into standard JavaScript:

codon compile path/to/file.cdn

Output: Generates path/to/file.js in the same directory.

Translate JavaScript to Codon (translate)

Parses a .js (or .jsx) file and emits Codon notation with automatic Level-2 code clone compaction:

codon translate path/to/file.js

Output: Generates path/to/file.cdn in the same directory.

Verify Roundtrip Suite (verify)

Runs verification benchmarks to ensure round-trip AST fidelity:

codon verify

---

Development & Contributing

To contribute to or develop the Codon compiler repository directly from source:

# Clone the repository
git clone https://github.com/your-org/codon.git
cd codon

# Install dependencies
bun install

# Run test suite
bun test

---

Architecture

The project is structured into modular components:

• index.ts: The CLI entrypoint. Handles commands (build, init, compile, translate, verify), flags, and configuration loading.
• project-compiler.ts: Project-wide compiler engine (compileProject, watchProject, initProject, loadCodonConfig), static asset syncer, and runtime packager.
• codon-parser.ts: Recursive-descent parser that tokenizes and parses Codon notation, control structures, tacit trains, classes, and pattern destructuring into an ESTree-compatible Abstract Syntax Tree. Includes parseMacroInvocation.
• macro-dict.ts: Implements MacroDictionary, lexical scope resolution (resolveScopes), canonical wyhash subtree normalization (normalizeSubtree), and template expansion (expandToAST).
• dict.json: Empirical library of 721 Level-2 code clones mined from open-source repositories.
• transpilers.ts:
  • JSToCodonTranslator: Walks modern JavaScript/JSX AST, recognizes clone subtrees, and emits concise Codon notation.
  • CodonToJSCompiler: Normalizes AST nodes, maps tacit trains and Codon verbs, rewrites import specifiers, and uses @babel/generator to emit clean JavaScript.
• types.ts: Comprehensive TypeScript definitions for all supported AST nodes.
• codon-runtime.ts: Zero-dependency library of combinators and array primitives supporting Codon semantics in JavaScript.
• docs/: Comprehensive documentation suite (README, Projects, Macros, Syntax, Runtime, API, CLI).

---

License

MIT