Skip to content

perf-004: Static tokens for single-char lexemes #229

Description

@python-processing-unit

Problem

Single-char tokens (, ), [, ], {, }, <, >, ,, =, ., :, @, ~, #, \n\ each heap-allocate and copy a 1-byte string. They dominate token count.

Location: \src/lexer.c:162-177\ (\make_token)

Proposed Fix

Add ownership flag to \Token. Use static literal pool for single-char tokens. \ oken_free\ skips \ ree\ for \TOKEN_LITERAL_STATIC.

Status

Baseline still present — \make_token\ at \src/lexer.c:162-177\ still does \safe_malloc\ + \memcpy\ for every token. No ownership flag in \Token\ struct at \src/token.h:59-64.

Metadata

Metadata

Labels

interpreterRequires a code change in the interpreter.patchRequires a patch version change.performancePerformance improvement.

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions