memory.txt

Brackets Code - Memory & Specification
=============================================

1. WHAT IT IS
--------------
Brackets Code is a CLI code generator for the Brackets (L1VM) assembly language.
It converts natural-language prompts into working .l1com programs.
Written in C, split across 4 source files (~6900 lines total). Author: Stefan.

2. SOURCE FILE ORGANIZATION
--------------------------
  brackets-code.h  -- master header: type defs (Variable, Function, Program, TaskProfile, etc.),
                      constants, global variable externs, and all forward declarations
                      (emitters, templates, utilities, CLI).
  brackets-code.c  -- main translation unit (~3280 lines): utility funcs, data structures,
                       all 166 emitters, 31 templates, parse_task, generate_code, CLI.
  dsl.c            -- DSL rule engine (~1080 lines): dsl_parse_file(), dsl_match_*,
                       dsl_generate_code(), dsl_generate_from_task(). Separately compiled.
  embed.c          -- LLM inference engine (~535 lines): word embeddings, softmax,
                       vector search, llm_select_emitter(), split_prompt_steps().
  learn.c          -- learned pattern system (~690 lines): learn from .l1com files,
                       persist/reload .l1lp files, match + emit learned patterns.
  dsl.h            -- DSL header: types (DslRule, DslToken, DslVarDecl), function declarations.

  dsl.c is SEPARATELY COMPILED (not #included) and linked.

3. ARCHITECTURE OVERVIEW
-----------------------
The program has a three-tier generation pipeline:

  Prompt -> is_question? -> answer_question() + generate_code()
       |
  smart_generate() --> multi-step split + per-step TaskProfile generation
       |                  |
       |            generate_from_task():
       |              a) dsl_generate_from_task() -- DSL rules (flag + keyword matching)
       |              b) llm_select_emitter() -- scores 166 emitters via softmax
       |              c) Regular emitter dispatch (input_sort, median, factorial, etc.)
       | (fallback)
  match_template() --> 31 template functions
       | (fallback)
  nearest-neighbor vector search --> gen_hello_world() with NN comment
       | (fallback)
  gen_hello_world() (default)

Key principle: "planner-first ordering" -- the plan-based smart_generate()
runs BEFORE template matching, so compound/novel prompts are handled
by the planner rather than being captured by shorter template keywords.

DSL rules take priority: if dsl_generate_from_task() returns 1, the
regular emitter is never called for that step.

4. DATA STRUCTURES
------------------
Program:
  - filename, funcs[] (up to 24), includes[] (up to 24), includes_post[] (up to 24), globals

Function:
  - name, is_local, has_vars, has_vardef, vardef_name
  - vars[] (up to 48 Variable), body (string buffer, up to 64KB)

Variable:
  - name, type, count, values[], num_values

TaskProfile (the plan struct):
  - has_input, input_count, count_value
  - has_literals, literals[], num_literals
  - has_operation, op[] ("add"/"sub"/"mul"/"div"/"mod")
  - has_algorithm, algorithm[], algo_param
  - Enhanced flags: has_sum_range, has_print_even, has_find_max,
    has_fib_seq, has_countdown_from, has_input_sort, has_median,
    has_input_fact, has_string_cat, has_string_compare,
    has_array_assign, has_array_reverse, has_array_find, has_average,
    has_read_file, has_write_file, has_array_vmath,
    has_string_to_num, has_timer,
    has_array_min_max, has_bool_demo, has_bit_check,
    has_max, has_min, has_descending, has_sort_stats,
    has_input, has_factorial, has_power, has_fizzbuzz, has_primes, ...
  - has_loop, loop_start/end/step
  - has_condition, has_output, has_output_str
  - type[] ("int64", "double", "int32", "int16", "byte")
  - title[]
  - inherit_var[], inherit_count, num_inherit_vars
  - inherit_var_types[][], inherit_var_names[][], inherit_var_counts[]

Template struct:
  - keywords (space-separated), description, function pointer

5. DSL RULE SYSTEM
-------------------
  DSL rules are stored in dsl/*.l1dsl files. Each rule has:
    parser: "keywords, ..."        -- tokenization keywords for scoring
    match: has_flag_name           -- required TaskProfile flag (optional)
    token: type name               -- parsed numeric tokens from prompt
    result: type name              -- the output variable
    var: type name value           -- explicit variable declaration
    include: header.l1h            -- file to #include
    desc: "description"            -- human-readable description
    code:                          -- L1VM code body (indented lines)

  Generation flow (dsl_generate_from_task in dsl.c):
    1. Flag matching: for each rule, check if task->has_* flag matches
       rule->match. If match, call dsl_generate_code().
    2. Keyword matching: dsl_match_all_rules() scores all rules against
       prompt text. Rules with >=3 keywords need >=2 matches.
       Skip already-applied flag-matched rules.
    3. Token value update: update_token_from_prompt() fills token defaults
       from numbers in the prompt.

  dsl_generate_code() in dsl.c:
    1. Add includes (pre + post)
    2. Declare var:/token:/result: variables via add_var_to_func() (dedup'd by name)
    3. Append code lines to f->body, dedup'd by exact string match:
       if strncmp("(set ", 5) == 0 && strstr(f->body, line) != NULL → skip
    4. Code lines get \t prefix for L1VM indentation

  dsl.c is compiled separately with -Wall -Wextra (no warnings).

6. EMITTER BLOCKS (NUM_EMITTERS = 166)
--------------------------------------
  Each emits L1VM code for a specific pattern:

  Block                    Purpose
  ------------------------ ------------------------------
  emit_math                Pure math with 2-3 literals
  emit_input_loop          Read N numbers, optionally sum
  emit_for_sum             Sum 1..N
  emit_print_even          Print even numbers <= N
  emit_input_find_max       Read N, find maximum
  emit_countdown_from       Countdown N to 0
  emit_fib_seq             Fibonacci sequence (first N)
  emit_input_sort           Read N, bubble sort, print (supports descending, max, min)
  emit_median               Read N, sort, print median (uses "two" const 2 for divide)
  emit_string_cat           Input 2 strings, concatenate
  emit_string_compare       Input 2 strings, compare
  emit_array_assign         Input index+value, assign to array
  emit_array_reverse        Read N, reverse array, print (parameterized array_count)
  emit_array_find           Read N + search key, linear search
  emit_input_factorial      Read N, call factorial, print
  emit_array_vmath          Array min/max/average using vmath
  emit_read_file            Read int64 value from file
  emit_write_file           Write int64 value to file
  emit_string_to_num        Parse string input to integer
  emit_timer                Measure execution time (intr0 24/25)
  emit_factorial            Recursive factorial (no input)
  emit_fizzbuzz             FizzBuzz 1..100
  emit_primes               Prime finder up to N
  emit_even_odd             Check if number is even or odd
  emit_power                Power function (loop)
  emit_multiplication_table 1..10 times table
  emit_guess_number         Number guessing game
  emit_gcd                  Recursive Euclid GCD
  emit_random_number        Generate N random numbers
  emit_hello_name           Ask name and greet
  emit_array_min_max        Find min/max/average via vmath library
  emit_bool_demo            Boolean variable demo
  emit_bit_check            Bit check with << and &
  emit_leap_year            Leap year detection
  emit_temp_convert         Celsius-to-Fahrenheit conversion
  emit_circle_area          Circle area calculation
  emit_average              Average of input numbers
  emit_selection_sort       Selection sort
  emit_fann_create          Create FANN neural network
  emit_fann_train           Train FANN neural network
  emit_fann_run             Run FANN neural network
  emit_palindrome           Palindrome number check
  emit_lcm                  Least Common Multiple
  emit_collatz              Collatz sequence generator
  emit_sum_of_digits        Sum of digits of a number
  emit_reverse_string       Reverse a string
  emit_armstrong            Armstrong number check
  emit_perfect_number       Perfect number check
  emit_count_vowels         Count vowels in a string
  emit_anagram_check        Anagram string comparison
  emit_string_to_upper      Convert string to uppercase
  emit_string_to_lower      Convert string to lowercase
  emit_caesar_cipher        Caesar cipher encode/decode
  emit_palindrome_string    Palindrome string check
  emit_bubble_sort          Bubble sort with hardcoded array
  emit_binary_search        Binary search on sorted array
  emit_square_root          Integer square root via Newton-Raphson
  emit_prime_factorization  Prime factor decomposition
  emit_standard_deviation   Standard deviation of N numbers
  emit_compound_interest    Compound interest calculation
  emit_decimal_to_binary    Decimal to binary conversion
  emit_dice_roll            Dice roll simulation (1-6)
  emit_double_math          Double-precision math (add/sub/mul/div)
  emit_double_circle_area   Double-precision circle area
  emit_double_average       Double-precision average of 5 numbers
  emit_double_compound_interest Double-precision compound interest
  emit_double_pythagoras    Double-precision Pythagoras
  emit_double_temp_convert  Double-precision Celsius to Fahrenheit
  emit_double_sqrt          Double-precision Newton-Raphson sqrt
  emit_function             User-defined function call demo
  emit_string_length        Length of input string
  emit_stack                Stack: push/pop/size/print-all via sp array
  emit_queue                Queue: enqueue/dequeue/print via head/tail array
  emit_insertion_sort       Insertion sort with nested for/while loops
  emit_calculator           Calculator: add/sub/mul/div/mod via menu
  emit_unit_converter       Unit converter: km/mi/kg/lb/cm/in via menu
  emit_rock_paper_scissors  RPS game vs computer
  emit_pyramid              Print pyramid number pattern 1..n
  emit_temp_converter_menu  Temperature converter with menu (C/F/K)
  emit_sort_stats           Sort + statistics (min/max/avg/median)
  emit_string_analyzer      String character analysis
  emit_number_analyzer      Number analysis (digits/sum/even/odd/prime)
  emit_filter_numbers       Filter numbers from list (even/odd/prime)
  emit_random_generator     Random number generator with range/count
  emit_math_menu            Math operations menu
  emit_quiz_game            Math quiz with random questions
  emit_bmi_calculator       BMI calculator
  emit_statistics_suite     Full stats suite
  emit_linked_list          Linked list insert/search/delete/print
  emit_binary_search_tree   BST insert/search/traverse
  emit_tree_traversal       Binary tree inorder/preorder/postorder
  emit_graph_bfs_dfs        Graph BFS/DFS traversal
  emit_n_queens             N-Queens backtracking solver
  emit_sudoku               Simple sudoku constraint propagation solver
  emit_levenshtein_distance Levenshtein edit distance
  emit_maze_generator       Maze generation via DFS
  emit_maze_solver          Maze solver BFS
  emit_monte_carlo_pi       Monte Carlo pi estimation
  emit_matrix_multiplication Matrix multiply
  emit_matrix_transpose     Matrix transpose
  emit_numerical_integration Numerical integration (trapezoidal)
  emit_complex_numbers      Complex number arithmetic
  emit_linear_regression    Linear regression (least squares)
  emit_base_converter       Base converter (decimal to base N)
  emit_freq_analysis        Letter frequency analysis
  emit_shuffle              Array shuffle (Fisher-Yates)
  emit_weighted_random      Weighted random selection
  emit_ascii_table          ASCII table printer
  emit_bignum_math          Bignum arithmetic
  emit_password_card        Password card generator
  emit_chess_problem        Chess puzzle setup
  emit_shell_repl           Interactive shell REPL
  emit_webserver            TCP web server
  emit_sdl_window           SDL pixel-graphics window
  emit_sdl_button           SDL button GUI
  emit_thread               Two-thread demo
  emit_scheduler            Task scheduler
  emit_shell_exec           Shell command execution
  emit_json                 JSON string get/set
  emit_crypto               Encrypt/decrypt
  emit_bluetooth_ble        Bluetooth LE init stub
  emit_serial_rs232         Serial RS-232 init stub
  emit_gpio                 GPIO pin I/O
  emit_gps                  GPS position read
  emit_timer_date           Timer elapsed ms + date/time
  emit_sdl_sound            SDL sound playback
  emit_sdl_joystick         SDL joystick axis/button
  emit_sdl_mouse            SDL mouse motion/click
  emit_fractal              Mandelbrot fractal ASCII render
  emit_cluster_3x1          3x1 cluster communication stub
  emit_reload               Self-reload stub
  emit_coordinate_grid      2D coordinate grid operations
  emit_turmite              Turmite (ant) simulation
  emit_crossword            Crossword puzzle grid generator
  emit_linter               Code linter stub
  emit_hello_world          Hello World program
  emit_string_find          Find substring in a string
  emit_string_split         Split string by delimiter
  emit_switch_demo          Switch/case control flow demo
  emit_type_convert         Type conversion
  emit_iterative_factorial  Iterative factorial with while loop
  emit_random_walk          Random walk simulation
  emit_bar_chart            ASCII bar chart from data array
  emit_hanoi_tower          Towers of Hanoi step printer
  emit_ascii_art            Simple ASCII art shapes
  emit_number_to_words      Convert numbers to English words
  emit_temperature_table    Celsius/Fahrenheit conversion table
  emit_loop_demo            Loop constructs demo
  emit_pointer_demo         Pointer access demo
  emit_struct_demo          Struct (dotted vars) demo
  emit_hex_binary           Hex and binary literal demo
  emit_shell_args           Command-line arguments demo
  emit_time_demo            Current time + epoch ms
  + template emitters (31 matched by match_template, not smart_generate)

7. GENERATION FLOW (generate_code)
-----------------------------------
1. Check if input is a question -> answer via FAQ
2. Try smart_generate() which:
   a. Splits prompt on conjunctions (multi-step detection)
   b. Validates/merges non-viable steps
   c. Per-step: generate_from_task()
      i.   dsl_generate_from_task() -- DSL flag + keyword matching
      ii.  llm_select_emitter() + dispatch to emitter block
   d. Falls through to single-step plan-based generation
3. If plan fails, try match_template() -- scores all 31 templates
4. If template fails, generate default "hello world"

8. MULTI-STEP COMPOSITION (split_prompt_steps in embed.c)
----------------------------------------------------------
  - Split patterns (ordered longest-first):
    " und dann ", " und danach ", " anschliessend ",
    " and then ", " . ", ". ", "; ", " then ",
    " danach ", " and " (math-guarded), " und " (math-guarded)
    "Schritt N:" / "step N:" patterns (phase 2)
  - Math guard: if conjunction surrounded by digits, skip split
  - Comma guard: ", and" / ", und" / ", or" / ", oder" → false positive for split
  - Each split segment parsed as independent TaskProfile
  - Pre-validation: non-viable steps auto-merge backward
  - Additional merge: "print the median/average/.../result/them/it"
    steps merge with preceding step
  - Per-step generation via generate_from_task (same Function f)
  - Variable inheritance: non-const vars from prior steps passed to next step

9. DSL KEYWORD MATCHING
-----------------------
  dsl_match_all_rules() in dsl.c:
    - Tokenizes prompt into words
    - For each rule, tokenizes parser keywords into individual words
    - Counts how many keyword words appear in the prompt
    - Rules with >=3 keywords require >=2 matches (minimum match count)
    - Scores are (match_count / total_keyword_count) normalized
    - Results sorted by score descending
    - Threshold: score >= 0.3f to be considered a match

  dsl_match_rule() in dsl.c:
    - Dedicated single-rule keyword matching
    - Same minimum-match logic: >=2 matches for >=3 keyword rules

10. write_program DEDUP
------------------------
  write_program() in brackets-code.c writes a Function's vars + body:
    - Vars section: writes all fn->vars[] as (set ...) declarations
      and tracks emitted variable names in emitted_names[] buffer
    - Body section: for each body line:
      - Skip lines starting with "(set " whose variable name is already
        in emitted_names (dedup with " %s " space-delimited match)
      - Otherwise emit line and add variable name to emitted_names
    - Purpose: prevent duplicate variable declarations when multiple
      DSL rules generate code for the same function
    - False-positive guard: " %s " format avoids matching "c " inside "fac "

11. KEY DESIGN DECISIONS
------------------------
  - Planner-first ordering; DSL takes priority over emitters
  - Multi-step: all steps share same Function f (body accumulates)
  - Variable inheritance: non-const, non-zero/one/two/four/five vars inherited
  - DSL code body lines get \t prefix (matching emitter code format)
  - emit_array_reverse takes array_count parameter (not hardcoded 5)
  - has_sort_stats set when sort + (min|max|average|avg) in prompt
  - has_input_sort activated if has_descending set even without has_input
  - Literal values always use named variables (zero, one, two)
  - Complex nested expressions split into single-operation lines
  - vars.l1h added per-emitter rather than globally
  - Labels use colon syntax: (:label) for definition, :label for reference
  - "exit" in string constants triggers l1pre macro conflict
  - :input_i fails on piped stdin (interactive terminal reads)
  - if+/else requires (f if+) not (f if) before (else)
  - Array element writes: (value array [ index ] =) NOT expression as target
  - Double operations: operator BETWEEN operands, reset-reg before :print_d

12. COMPILATION PIPELINE
------------------------
  brackets-code "<prompt>"  -->  writes program.l1com
  l1pre program.l1com out.l1com <include_path>/  -->  expands includes
  l1com out  -->  compiles to out.l1asm
  l1asm out.l1asm  -->  assembles to out.l1obj
  l1vm out -q  -->  runs (out.l1obj loaded)

  l1pre requires trailing slash on include directory argument.
  l1com reads <basename>.l1com (not the full filename).
  l1vm loads <basename>.l1obj; -q flag runs quietly.

13. TEST SUITE
--------------
  Shell-based test suite at tests/run_tests.sh (282 lines).
  Tests generation + L1VM pipeline (l1pre → l1com).
  Two modes: --quick (22 tests) and full (87 tests).
  Categories: template tests, emitter tests, multi-step tests.
  Known failures: none
  Latest run: 192 passed, 0 failed, 0 skipped.

14. FILE LOCATIONS
------------------
  /home/stefan/opencode/brackets-code/brackets-code.c  (~3294 lines)
  /home/stefan/opencode/brackets-code/brackets-code.h  (~534 lines)
  /home/stefan/opencode/brackets-code/dsl.c            (~1008 lines)
  /home/stefan/opencode/brackets-code/dsl.h            (~131 lines)
  /home/stefan/opencode/brackets-code/embed.c          (~1276 lines)
  /home/stefan/opencode/brackets-code/learn.c          (~712 lines)
  /home/stefan/opencode/brackets-code/synonyms.txt     (fuzzy synonyms)
  /home/stefan/opencode/brackets-code/Makefile         (build, readline auto-detect)
  /home/stefan/opencode/brackets-code/memory.txt       (this file)
  /home/stefan/opencode/brackets-code/tests/test_unit.c (~417 lines, 40 tests)
  /home/stefan/opencode/brackets-code/tests/run_tests.sh (~405 lines, 192 tests)

  Total: ~7800 lines across 8 source files.
  Compilation: clang-19 brackets-code.c dsl.c -lm (separate compilation)