nescript/docs/architecture.md

Compiler Architecture

An overview of the NEScript compiler internals for contributors and maintainers.

---

Pipeline

Source (.ne) --> Lexer --> Parser --> Analyzer --> IR Lowering --> Optimizer --> Codegen --> Peephole --> Linker --> ROM (.nes)

Each phase is a pure function (input to output) with no global state, making every stage independently testable.

Phase Summary

Phase	Input	Output	Responsibility
Lexer	Source text	Token stream	Tokenization, number/string literal parsing
Parser	Token stream	AST	Syntax validation, tree construction
Analyzer	AST	Annotated AST	Type checking, scope resolution, call graph analysis
IR Lowering	Annotated AST	NEScript IR	Flatten expressions, expand u16 ops, desugar
Optimizer	IR	Optimized IR	Constant folding, dead code, strength reduction, inlining
Codegen	Optimized IR	6502 instruction list	Slot allocation, instruction selection
Peephole	6502 instructions	6502 instructions	Dead-load elimination, branch folding, INC/DEC fold
Linker	Instructions + assets	.nes file	Bank layout, vectors, iNES header

---

Modules

Each module has a mod.rs (implementation) and a co-located tests.rs with unit tests.

lexer/

mod.rs, token.rs, tests.rs. Tokenizes NEScript source into a stream of typed tokens with source spans. Handles decimal/hex/binary integer literals, string literals, keywords, operators, and the raw-capture mode for asm { ... } bodies.

parser/

mod.rs, ast.rs, preprocess.rs, tests.rs. Recursive descent parser that converts the token stream into an AST. ast.rs defines every AST node type (expressions, statements, declarations). preprocess.rs inlines include "path.ne" directives before parsing.

analyzer/

mod.rs, tests.rs. Performs semantic analysis on the AST: type checking, scope and symbol table management, call graph construction with depth analysis, state reachability, unused-variable detection, and dead-code-after-terminator warnings. Emits all user-facing diagnostics beyond lexer/parser syntax errors.

ir/

mod.rs (types), lowering.rs (AST → IR), tests.rs. The IR is a flat, register-agnostic representation built from virtual temps and basic blocks. Lowering flattens nested expressions, expands 16-bit operations, desugars for into while, and resolves constant expressions early.

optimizer/

mod.rs, tests.rs. Runs passes over the IR in order: strength reduction (mul/div by powers of two, mod by powers of two, ShiftVar → ShiftLeft/Right), function inlining (trivial-function elimination), constant folding (arithmetic + comparisons + shifts), and dead code elimination.

codegen/

ir_codegen.rs, peephole.rs, mod.rs. ir_codegen.rs walks the optimized IR and emits 6502 instructions; variables land in allocated addresses and IR temps land in a recycling zero-page slot pool. peephole.rs runs after codegen to clean up the temp-heavy output (dead-load elimination, branch folding, INC/DEC fold, copy propagation).

asm/

mod.rs, opcodes.rs, inline_parser.rs, tests.rs. The built-in assembler and the inline-asm parser. opcodes.rs defines the 6502 opcode table with addressing modes. inline_parser.rs parses the body of asm { ... } blocks so codegen can splice real instructions in-line.

linker/

mod.rs, tests.rs. Assigns addresses to code and data segments, resolves label/symbol fixups, lays out banks for banked mappers (MMC1/UxROM/MMC3), and emits the final iNES byte stream via rom::RomBuilder.

rom/

mod.rs, tests.rs. Builds the final iNES ROM file. Generates the 16-byte iNES header and places the NMI/RESET/IRQ vector table.

runtime/

mod.rs, tests.rs. Contains built-in runtime code emitted into every ROM: NES hardware init, NMI handler, controller reads, OAM DMA, software multiply/divide, the frame-walking audio driver (gen_audio_tick, gen_period_table, gen_data_block), and the MMC1/UxROM/MMC3 mapper init and bank-switch helpers.

assets/

mod.rs, chr.rs, palette.rs, resolve.rs, audio.rs, tests.rs. The asset pipeline. chr.rs converts PNGs to CHR tile data. palette.rs maps RGB to NES palette indices. resolve.rs resolves sprite declarations into tile-indexed CHR blocks. audio.rs compiles sfx/music declarations into ROM-ready envelope and note-stream byte tables, plus the builtin effect/track tables used when programs reference audio names they haven't declared.

errors/

mod.rs, diagnostic.rs, render.rs. Defines the Diagnostic struct (error codes, severity, spans, labels, help text). Renders diagnostics with color and source context for terminal output using ariadne.

---

Testing

Test Organization

Tests are co-located with each module in tests.rs files under src/:

src/lexer/tests.rs        -- lexer unit tests
src/parser/tests.rs       -- parser unit tests
src/analyzer/tests.rs     -- semantic analysis tests
src/ir/tests.rs           -- IR lowering tests
src/optimizer/tests.rs    -- optimizer tests
src/asm/tests.rs          -- assembler tests
src/linker/tests.rs       -- linker tests
src/rom/tests.rs          -- ROM builder tests
src/runtime/tests.rs      -- runtime code emission tests
src/assets/tests.rs       -- asset pipeline tests

End-to-end and error-code tests live in tests/integration_test.rs, which compiles representative .ne snippets through the full pipeline and asserts on ROM/diagnostic shape. The emulator smoke test (tests/emulator/run_examples.mjs) runs every example through jsnes and byte-compares the resulting screenshot and audio hash against goldens in tests/emulator/goldens/.

Running Tests

# Run all tests
cargo test

# Run tests for a specific module
cargo test --lib lexer
cargo test --lib parser
cargo test --lib analyzer

# Run integration tests only
cargo test --test integration_test

# Run a specific test by name
cargo test test_name

Test Strategy

Each compiler phase is a pure function, so unit tests provide isolated input and verify output without side effects. Integration tests compile complete .ne programs and either assert on shape (length, presence of specific labels) or byte-compare output against checked-in goldens. Emulator goldens catch regressions that pass type-check but corrupt the final executable image.