Spec M4 — tokenizer in .mc (a free cross-check)

Prerequisite: M3. Goal: the first real program in the language, and a cross-check of the lexer: src/lex.mc must be a transliteration of stage0/lex.c, function by function.

Deliverable (mc-dev) #

• src/lex.mc: the same token table (same insertion order, same ids), the same longest-prefix matching, the same literals/escapes/comments, the same line counting. No #rule/prelude. Flat data: tokens in an arena array with #define TOK_ID 0, TOK_START 8, ... and accessors.
• src/lexdump.mc: main opens the file given in argv[1] (read ld64(argv + 8)), tokenizes to EOF and prints exactly the format of mc0 --dump-tokens (LINE ID TEXT).
• scripts/check-lex.sh: for each tests/*.mc and src/*.mc, compares mc0 --dump-tokens F with build/lexdump F via diff; exit 1 on any difference.

Acceptance #

scripts/check-lex.sh green for all files. Report the line count of src/lex.mc.

Edit this page