About the Scanner: In a typical compiler, the scanner (also called lexer or tokenizer) reads source code character-by-character on-demand, producing tokens one at a time as the parser requests them. This is called "lazy scanning" or "on-demand tokenization."
This visualizer takes a different approach: we scan the entire source file up front, producing all tokens at once before compilation begins. This makes it easier to show off the complete token stream in the carousel above. The highlighted token shows where the parser is currently reading.
Each token contains: a type (NUMBER, IDENTIFIER, PLUS, etc.), the original lexeme (source text), a line number for error reporting, and an optional literal value for numbers and strings.
Pratt parsing (also called "top-down operator precedence parsing") was invented by Vaughan Pratt in 1973. See his original paper: Top Down Operator Precedence.
There's proofs in that paper, so watchout, but to tease I quote:
Thus Theorem 1 takes care of what the programmer needs to know, and Theorem 2 what the computer needs to know.
This visualizer is based on the Pratt parser implementation from Crafting Interpreters by Robert Nystrom, specifically the clox compiling chapter 17.
Note: The book's implementation compiles expressions to bytecode for later execution by a virtual machine. This visualizer is more like an interpreter (evaluate-as-you-go) deal. This makes it easier to see how the parser constructs the expression tree and handles operator precedence, but it's not how the actual clox compiler works. Use this tool to understand Pratt parsing pedagogy, but Nystrom's book for comprehending compilers.
Crafting Interpreter's quotes Russell and Dunsany, so watchout!
Pratt's Top Down Operator Precedence Parser works by traversing the sequence of tokens, along the way referring to a table of 'precedence' rules which inform the parser how much binding power the current token has. Comparing precedence rules combined with incrementing the precedence informs whether to keep recursively calling the main parse function, therefore putting off a binary operation for later, versus doing the operation now (for bytecode compilers emitting the operation's bytecode).
Note "Pratt's Parser" isn't a parser so much as a strategy for writing parsers. The work of figuring out and encoding the different precedence classes and their numerical ranking is up to you, the parser author. The result should be a distinct Precedence table of prefix (nud) and infix/postfix (led) functions, and precedence values that encode the grammar you want parsed. The two proofs from Pratt's paper offer a comforting garuntee that so long as your grammar follows a few rules, there will be such a table possible, and writing a parser for unambiguous grammar can be this 'easy'.
Grammar constraints for Pratt parsing:
f x y)if a then b else c, don't also allow if a, b, c)
The Precedence Stack: Pratt parsing works by recursively calling parsePrecedence(minPrec)
with increasing minimum precedence levels. The call stack visualizes these recursive calls.
Each frame shows the minimum precedence being enforced at that level. When we encounter an operator, we check if its precedence is high enough to be consumed at the current level. If not, we return and let a lower frame handle it. This creates the natural operator precedence hierarchy - where some parsers generate an Abstract Syntax Tree, Pratt parsing avoids this by holding the equivalent 'grammar' stucture both in the stack of recursive parse calls and the rest of the structure is not necessary because it has been evaluated as soon as possible.
Scope Depth and Locals: The compiler tracks lexical scope depth (0 for global scope, increasing with each block) and local variables in the current function.
When you declare a variable inside a block ({ var x = 1; }), it becomes a local variable
stored on the stack at runtime. The compiler tracks which locals are in scope and at what depth, so it
can generate correct bytecode to access them (OP_GET_LOCAL, OP_SET_LOCAL)
instead of using the slower global lookup.
Local variables are indexed by their stack slot position, making them very fast to access. When a block
ends, locals at that depth are discarded (via OP_POP instructions).
The Constants Pool: Bytecode instructions are small (typically 1-3 bytes), but values like numbers and strings can be arbitrarily large. Rather than embedding large values directly in the bytecode stream, we store them in a separate constants array.
When the compiler encounters a literal value (like 42 or "hello"), it adds
the value to the constants pool and emits an OP_CONSTANT instruction with the index.
At runtime, the VM uses this index to fetch the actual value from the pool and push it onto the stack.
This keeps the bytecode compact and makes the instruction set simpler - we only need one constant instruction instead of separate instructions for different value types and sizes.
Line Number Tracking: When a runtime error occurs, we want to tell the user which line of source code caused it. The lines array maps each bytecode instruction back to its source line.
This is pure debugging metadata - it doesn't affect program execution at all. In a production compiler, you might use a compressed encoding (run-length encoding) to save space, since consecutive instructions often come from the same source line.
The Bytecode Sequence: Bytecode is the compiled representation of your source code -
a linear sequence of instructions that the virtual machine executes. Each instruction is an opcode
(like OP_ADD, OP_PRINT) optionally followed by operands (like constant indices
or jump offsets).
The instruction pointer (IP) tracks which instruction is currently executing. It starts at the beginning
and advances sequentially, except when jump instructions (OP_JUMP, OP_JUMP_IF_FALSE,
OP_LOOP) redirect control flow for if/else statements and loops.
The black arrows show jump targets - where control flow redirects to. A forward jump implements if/else (skip the else branch), while a backward jump creates a loop (jump back to repeat code). Note as the compiler builds the jump bytecode the arrow reaches off the page to indicate the compiler still doesn't know how large the jump offset should be, yet. Only after compiling to the end of code which might get jumped to does the compiler go back to that jump bytecode to 'patch' the offset.
This linear structure makes bytecode simple and fast to execute: just fetch, decode, execute in a loop.
The Stack-Based VM: Our virtual machine uses a stack architecture. Most instructions
operate on values at the top of the stack: OP_ADD pops two values, adds them, and pushes
the result back. This is simpler than register-based VMs and maps naturally to expression evaluation.
Value Stack: Shows the current stack contents. The stack grows upward as values are pushed
(from constants, variable loads, expression results) and shrinks as values are popped (consumed by operators,
stored in variables, or discarded by OP_POP). I think of this as the running program's "short
term memory". (Not the best metaphor, I know, as we already call computer memory "memory")
Globals: Global variables are stored in a hash table (shown as a simple map here) keyed
by variable name. OP_DEFINE_GLOBAL creates a new global, while OP_GET_GLOBAL
and OP_SET_GLOBAL read and write them. Globals are slower than locals but accessible from
anywhere in the program.