imjasonh/nescript
1
0
Fork 0
mirror of https://github.com/imjasonh/nescript synced 2026-07-08 17:06:04 +00:00
Code Activity

cleanup: fix silent miscompiles and delete dead code exposed by code review

Browse source 
Two correctness bugs were silently producing wrong ROMs:

  - `x << n` / `x >> n` always shifted by 1, regardless of `n`, because
    the IR lowering for `BinOp::ShiftLeft`/`ShiftRight` hardcoded the
    count. Now eval_const the RHS into a compile-time count; fall back
    to a new `IrOp::ShiftLeftVar` / `ShiftRightVar` (runtime loop) when
    the amount isn't constant. Strength reduction folds the variable
    form back to a fixed count once the optimizer knows the value.

  - `x / n` / `x % n` always returned 0, because the lowering emitted
    `LoadImm(t, 0)` for `BinOp::Div`/`Mod` with a comment saying the
    runtime call was "TODO for now". Added real `IrOp::Div` and
    `IrOp::Mod`, wired them through use-counting and DCE, gave codegen
    `__divide`-based implementations, and taught strength reduction to
    rewrite power-of-two divisors into shifts and modulo-by-2ⁿ into
    AND masks. Constant folding now handles `Mul`/`Div`/`Mod`/shifts
    too, which were previously left for the codegen to emit inefficient
    software calls.

Dead code removed (no backward-compat shims kept):

  - `src/debug/` entirely. `DebugSymbols`, `SourceMap`, and the
    Mesen/.sym emitters had no callers outside their own tests;
    `main.rs` never wrote a symbol file. Documented the intent in
    `docs/future-work.md` so it comes back intentionally if needed.
  - `ErrorCode::E0202` (invalid cast) and `E0403` (unreachable state):
    defined, formatted, and marked `#[allow(dead_code)]` but never
    emitted. W0104 now carries the unreachable-state semantics too.
  - `Level::Info`: never constructed.
  - `load_background` / `set_palette` statements and their
    `BackgroundDecl` / `PaletteDecl` parser support: parsed and
    silently dropped by IR lowering (`// TODO: implement in asset
    pipeline`). Removed keywords, AST variants, parser paths, analyzer
    arms, and tests. `docs/future-work.md` documents the runtime
    palette/nametable design for when it comes back.

Doc cleanup:

  - `docs/architecture.md` was describing files that don't exist
    (`analyzer/types.rs`, `optimizer/const_fold.rs`, `codegen/regalloc.rs`,
    `rom/header.rs`, `debug/symbols.rs`, …). Rewrote it to match the
    real flat `mod.rs` + `tests.rs` layout and the real pipeline order.
  - `docs/future-work.md` was a hybrid of open work and "recently
    completed" entries that duplicated the active stubs at the top of
    the file. Collapsed to just the gaps that are actually still open.
  - `README.md` claimed Mesen symbol export and 210 tests; updated both.
  - `docs/language-guide.md` and `spec.md` described `palette` decls,
    `set_palette` / `load_background`, `debug.overlay`, and error codes
    that were never emitted. Trimmed.
  - Stale comments on `Statement::Play`/`StartMusic`/`StopMusic`
    claimed the audio subsystem was "a no-op at codegen time".

Tests:

  - Regression tests for every fix above (`lower_shift_left_with_literal
    _count_uses_that_count`, `lower_shift_right_with_variable_count
    _uses_runtime_variant`, `lower_divide_emits_div_op_not_load_imm
    _zero`, `lower_modulo_emits_mod_op_not_load_imm_zero`,
    `strength_reduce_div_by_power_of_two`, `strength_reduce_mod_by
    _power_of_two`, `strength_reduce_shift_var_with_constant_amount`).
  - Renamed the `program_with_sprites_and_palette` integration test
    (which was exercising the now-removed `load_background`/`set_palette`)
    to `program_with_inline_sprite_chr`.

`examples/sprites_and_palettes.ne` lost its `palette`/`set_palette`
usage. Nothing in the emulator test presses A, so the headless
jsnes render shouldn't move, but the golden may need regeneration
via `UPDATE_GOLDENS=1` if it does.

https://claude.ai/code/session_012fKB251HvEUQwG3tizFyqt
This commit is contained in:
Claude 2026-04-13 02:47:37 +00:00
parent 971a128d0b
commit fdb1ec7c91
No known key found for this signature in database
29 changed files with 699 additions and 1157 deletions
Download patch file Download diff file Expand all files Collapse all files

65
docs/architecture.md
View file

@ -7,19 +7,11 @@ An overview of the NEScript compiler internals for contributors and maintainers.
## Pipeline
```
Source (.ne) --> Lexer --> Parser --> Analyzer --> IR Lowering --> Optimizer --> Codegen --> Assembler --> Linker --> ROM (.nes)
Source (.ne) --> Lexer --> Parser --> Analyzer --> IR Lowering --> Optimizer --> Codegen --> Peephole --> Linker --> ROM (.nes)
```
Each phase is a pure function (input to output) with no global state, making every stage independently testable.
```
.ne source ---------> Lexer -----> Parser -----> Analyzer ------>
(tokens) (AST) (annotated AST)
IR Lowering -----> Optimizer -----> Codegen -----> Linker -----> ROM
(IR) (optimized IR) (6502 insns) (.nes file)
```
### Phase Summary
| Phase | Input | Output | Responsibility |
@ -28,53 +20,52 @@ IR Lowering -----> Optimizer -----> Codegen -----> Linker -----> ROM
| **Parser** | Token stream | AST | Syntax validation, tree construction |
| **Analyzer** | AST | Annotated AST | Type checking, scope resolution, call graph analysis |
| **IR Lowering** | Annotated AST | NEScript IR | Flatten expressions, expand u16 ops, desugar |
| **Optimizer** | IR | Optimized IR | Constant folding, dead code, ZP promotion, inlining |
| **Codegen** | Optimized IR | 6502 instruction list | Register allocation, instruction selection |
| **Assembler** | 6502 instructions | Byte sequences + fixups| Opcode encoding, address resolution |
| **Linker** | Bytes + assets | .nes file | Bank layout, vectors, iNES header |
| **Optimizer** | IR | Optimized IR | Constant folding, dead code, strength reduction, inlining |
| **Codegen** | Optimized IR | 6502 instruction list | Slot allocation, instruction selection |
| **Peephole** | 6502 instructions | 6502 instructions | Dead-load elimination, branch folding, INC/DEC fold |
| **Linker** | Instructions + assets | .nes file | Bank layout, vectors, iNES header |
---
## Modules
Each module has a `mod.rs` (implementation) and a co-located `tests.rs` with unit tests.
### `lexer/`
Tokenizes NEScript source text into a stream of typed tokens with source spans. Handles decimal, hex, and binary integer literals, string literals, all keywords, and operators.
`mod.rs`, `token.rs`, `tests.rs`. Tokenizes NEScript source into a stream of typed tokens with source spans. Handles decimal/hex/binary integer literals, string literals, keywords, operators, and the raw-capture mode for `asm { ... }` bodies.
### `parser/`
Recursive descent parser that converts the token stream into an AST. Defines all AST node types (expressions, statements, declarations) in `ast.rs`.
`mod.rs`, `ast.rs`, `preprocess.rs`, `tests.rs`. Recursive descent parser that converts the token stream into an AST. `ast.rs` defines every AST node type (expressions, statements, declarations). `preprocess.rs` inlines `include "path.ne"` directives before parsing.
### `analyzer/`
Performs semantic analysis on the AST: type checking (`types.rs`), scope and symbol table management (`scope.rs`), and call graph construction with depth analysis (`call_graph.rs`). Detects recursion, type mismatches, undefined references, and call depth violations.
`mod.rs`, `tests.rs`. Performs semantic analysis on the AST: type checking, scope and symbol table management, call graph construction with depth analysis, state reachability, unused-variable detection, and dead-code-after-terminator warnings. Emits all user-facing diagnostics beyond lexer/parser syntax errors.
### `ir/`
Defines the intermediate representation and the lowering pass (`lowering.rs`) that translates the annotated AST into IR. Flattens nested expressions, expands 16-bit operations into 8-bit sequences, and resolves syntactic sugar.
`mod.rs` (types), `lowering.rs` (AST → IR), `tests.rs`. The IR is a flat, register-agnostic representation built from virtual temps and basic blocks. Lowering flattens nested expressions, expands 16-bit operations, desugars `for` into `while`, and resolves constant expressions early.
### `optimizer/`
Runs optimization passes over the IR: constant folding (`const_fold.rs`), dead code elimination (`dead_code.rs`), zero-page promotion analysis (`zp_promote.rs`), and function inlining (`inliner.rs`).
`mod.rs`, `tests.rs`. Runs passes over the IR in order: strength reduction (mul/div by powers of two, mod by powers of two, ShiftVar → ShiftLeft/Right), function inlining (trivial-function elimination), constant folding (arithmetic + comparisons + shifts), and dead code elimination.
### `codegen/`
Translates optimized IR into 6502 instructions. Includes register allocation for the A/X/Y registers (`regalloc.rs`) and instruction pattern matching for idiomatic 6502 code (`patterns.rs`).
`ir_codegen.rs`, `peephole.rs`, `mod.rs`. `ir_codegen.rs` walks the optimized IR and emits 6502 instructions; variables land in allocated addresses and IR temps land in a recycling zero-page slot pool. `peephole.rs` runs after codegen to clean up the temp-heavy output (dead-load elimination, branch folding, INC/DEC fold, copy propagation).
### `asm/`
The built-in assembler. Encodes 6502 instructions (`encode.rs`) with all addressing modes (`addressing.rs`), using a complete opcode table (`opcodes.rs` -- 56 instructions across all modes).
`mod.rs`, `opcodes.rs`, `inline_parser.rs`, `tests.rs`. The built-in assembler and the inline-asm parser. `opcodes.rs` defines the 6502 opcode table with addressing modes. `inline_parser.rs` parses the body of `asm { ... }` blocks so codegen can splice real instructions in-line.
### `linker/`
Assigns addresses to code and data segments, resolves fixups/relocations (`fixups.rs`), and handles bank allocation (`banks.rs`) for banked mappers.
`mod.rs`, `tests.rs`. Assigns addresses to code and data segments, resolves label/symbol fixups, lays out banks for banked mappers (MMC1/UxROM/MMC3), and emits the final iNES byte stream via `rom::RomBuilder`.
### `rom/`
Builds the final iNES ROM file. Generates the 16-byte iNES header (`header.rs`) and places the NMI/RESET/IRQ vector table (`vectors.rs`).
`mod.rs`, `tests.rs`. Builds the final iNES ROM file. Generates the 16-byte iNES header and places the NMI/RESET/IRQ vector table.
### `runtime/`
Contains built-in runtime code that the compiler emits into every ROM: NES hardware initialization, NMI handler generation, controller read routines, OAM DMA setup, software multiply/divide, and the frame-walking audio driver (`gen_audio_tick`, `gen_period_table`, `gen_data_block`).
`mod.rs`, `tests.rs`. Contains built-in runtime code emitted into every ROM: NES hardware init, NMI handler, controller reads, OAM DMA, software multiply/divide, the frame-walking audio driver (`gen_audio_tick`, `gen_period_table`, `gen_data_block`), and the MMC1/UxROM/MMC3 mapper init and bank-switch helpers.
### `assets/`
The asset pipeline. Converts PNG images to CHR tile data (`chr.rs`), maps RGB colors to the NES palette (`palette.rs`), resolves `sprite`/`background` declarations into tile-indexed CHR blocks (`resolve.rs`), and compiles `sfx`/`music` declarations into ROM-ready envelope and note-stream byte tables (`audio.rs`) — plus the builtin effect/track tables used as fallbacks when programs reference audio names without declaring them.
### `debug/`
Debug instrumentation output. Generates source maps relating ROM addresses to source locations (`source_map.rs`), symbol tables compatible with Mesen (`symbols.rs`), and runtime check code for debug builds (`checks.rs`).
`mod.rs`, `chr.rs`, `palette.rs`, `resolve.rs`, `audio.rs`, `tests.rs`. The asset pipeline. `chr.rs` converts PNGs to CHR tile data. `palette.rs` maps RGB to NES palette indices. `resolve.rs` resolves `sprite` declarations into tile-indexed CHR blocks. `audio.rs` compiles `sfx`/`music` declarations into ROM-ready envelope and note-stream byte tables, plus the builtin effect/track tables used when programs reference audio names they haven't declared.
### `errors/`
Error reporting infrastructure. Defines the `Diagnostic` struct with error codes, severity levels, source spans, labels, help text, and notes (`diagnostic.rs`). Renders diagnostics with color and source context for terminal output (`render.rs`).
`mod.rs`, `diagnostic.rs`, `render.rs`. Defines the `Diagnostic` struct (error codes, severity, spans, labels, help text). Renders diagnostics with color and source context for terminal output using ariadne.
---
@ -82,7 +73,7 @@ Error reporting infrastructure. Defines the `Diagnostic` struct with error codes
### Test Organization
Tests are co-located with each module in `tests.rs` files:
Tests are co-located with each module in `tests.rs` files under `src/`:
```
src/lexer/tests.rs -- lexer unit tests
@ -90,20 +81,14 @@ src/parser/tests.rs -- parser unit tests
src/analyzer/tests.rs -- semantic analysis tests
src/ir/tests.rs -- IR lowering tests
src/optimizer/tests.rs -- optimizer tests
src/codegen/tests.rs -- code generation tests
src/asm/tests.rs -- assembler tests
src/linker/tests.rs -- linker tests
src/rom/tests.rs -- ROM builder tests
src/runtime/tests.rs -- runtime code emission tests
src/assets/tests.rs -- asset pipeline tests
```
Integration tests live in the `tests/` directory:
```
tests/integration/ -- full pipeline tests with .ne source files
tests/error_tests/ -- tests that verify specific error codes
tests/asm_tests/ -- 6502 opcode and addressing mode tests
```
End-to-end and error-code tests live in `tests/integration_test.rs`, which compiles representative `.ne` snippets through the full pipeline and asserts on ROM/diagnostic shape. The emulator smoke test (`tests/emulator/run_examples.mjs`) runs every example through `jsnes` and byte-compares the resulting screenshot and audio hash against goldens in `tests/emulator/goldens/`.
### Running Tests
@ -117,7 +102,7 @@ cargo test --lib parser
cargo test --lib analyzer
# Run integration tests only
cargo test --test '*'
cargo test --test integration_test
# Run a specific test by name
cargo test test_name
@ -125,6 +110,4 @@ cargo test test_name
### Test Strategy
Each compiler phase is designed as a pure function, so unit tests provide isolated input and verify output without side effects. Integration tests compile complete `.ne` source files and verify the output ROM matches expected golden files in `tests/integration/expected/`.
Error tests in `tests/error_tests/` contain intentionally broken programs and verify that the correct error code is produced (e.g., `recursion.ne` should produce `E0402`).
Each compiler phase is a pure function, so unit tests provide isolated input and verify output without side effects. Integration tests compile complete `.ne` programs and either assert on shape (length, presence of specific labels) or byte-compare output against checked-in goldens. Emulator goldens catch regressions that pass type-check but corrupt the final executable image.