Python: library callback summaries -- a library calling back into client code (stacked on #1873)

bin/axiomcode

@@ -295,6 +295,13 @@ case "$cmd" in
     echo "▶ parsing $src → IR"
     "$0" parser "$src" "$ir" ${popts[@]+"${popts[@]}"} > "$int/parse.log" 2>&1 || { echo "❌ parser failed — see $int/parse.log" >&2; tail -5 "$int/parse.log" >&2; exit 1; }
     head -1 "$int/parse.log"
+    # WHAT AN INSTALLED PYTHON LIBRARY HANDS BACK to the objects the client gives it (graph/python/libsum): read from
+    # the project's environment, cached by package and version, staged beside the client IR. Never fatal -- without
+    # it the engine sees no library hand-backs, exactly as before. AXIOMCODE_LIB_CALLBACKS=0 turns it off.
+    if [ -d "$ir/python" ] && [ "${AXIOMCODE_LIB_CALLBACKS:-1}" != 0 ]; then
+      python3 "$ROOT/graph/python/libsum/libsum.py" --src "$src" --out "$ir/python" > "$int/libsum.log" 2>&1 \
+        && tail -1 "$int/libsum.log" || echo "⚠️  library callback summaries failed (see $int/libsum.log); continuing without them" >&2
+    fi
     # which languages the IR holds — one folder each — and which of them have an engine
     found=(); for d in "$ir"/*/; do l="$(basename "$d")"; [ -d "$ROOT/graph/$l/engine" ] && found+=("$l"); done
     [ ${#found[@]} -gt 0 ] || { echo "❌ no language with an engine in the IR (folders: $(ls "$ir" | tr '\n' ' '))" >&2; exit 1; }

graph/bundle/SCHEMA.md

@@ -442,7 +442,7 @@ One row per place a call is written (or, for a synthesised edge, the construct t
 - **typescript** — end_line / end_column come from the expression row; the call-site row itself records only the start.
 - **javascript** — caller_id is the parser's enclosing method, or the module initializer for top-level code. end_line / end_column come from the expression row. `require()` is a module edge, not a call site.
 - **typescript** — PROPERTY_READ and PROPERTY_WRITE rows are accessor invocations with no written call: the site is the property-access expression that runs the getter or setter, positioned from the expressions table, and callee_name is NULL because nothing was written; the accessor's name is on the callee's methods row. Filter them out with kind NOT IN (…) when counting calls.
-- **python** — PROPERTY_READ, CONTEXT_MANAGER, ITERATION_PROTOCOL, METACLASS_CREATION and DYNAMIC_CALL rows are protocol or indirect edges with no written call: their site is the expression that triggers them, and callee_name is always NULL because nothing was written. SUBSCRIPT_CALL is NULL only when the subscript is not a written name (measured 206 of 337 rows on a Python subject). Filter them out with kind NOT IN (…) when counting calls.
+- **python** — PROPERTY_READ, PROPERTY_WRITE, CONTEXT_MANAGER, ITERATION_PROTOCOL, METACLASS_CREATION and DYNAMIC_CALL rows are protocol or indirect edges with no written call: their site is the expression that triggers them, and callee_name is always NULL because nothing was written. SUBSCRIPT_CALL is NULL only when the subscript is not a written name (measured 206 of 337 rows on a Python subject). Filter them out with kind NOT IN (…) when counting calls.
 - **python** — The id is an EXPRESSION hash for a written call; a DECORATOR hash (PY_DECORATOR_…) for DECORATOR_APPLICATION and DECORATOR_* sites, positioned at the decorator line; and the class's TYPE hash for METACLASS_CREATION, positioned at the class declaration.
 
 ### `call_edges`
@@ -531,6 +531,7 @@ THE GRAPH. One row per (site, resolved target). A site with N possible targets h
 | `DECORATOR_*` | python | Applying an unparenthesised decorator; the suffix is the parser's decorator kind: BARE, ATTRIBUTE, SUBSCRIPT, EXPRESSION (and CALL/ATTRIBUTE_CALL when the factory expression is not itself a call site). The site is the decorator hash. |
 | `METACLASS_CREATION` | python | A class statement invokes its metaclass's `__new__` / `__init__` at import time, whether the metaclass is written on the statement (`class X(metaclass=M)`) or inherited from a base, and the nearest base's `__init_subclass__`. No written call; the site is the class's type hash. |
 | `PROPERTY_READ` | python | Reading `obj.attr` where `attr` is a `@property` runs the getter; reading `Cls.attr` where the METACLASS defines `attr` as a property runs that getter. No written call; the site is the attribute-access expression. |
+| `PROPERTY_WRITE` | python | Assigning `obj.attr = v` where `attr` is a `@property` with a setter runs the setter; `del obj.attr` runs its deleter. No written call; the site is the attribute-access expression. |
 | `CONTEXT_MANAGER` | python | `with expr:` runs `__enter__` / `__exit__` (or the async pair). No written call; the site is the context-manager expression. |
 | `ITERATION_PROTOCOL` | python | `for x in expr:` (and comprehensions) runs `__iter__` / `__next__` (or the async pair). No written call; the site is the iterated expression. |
 | `SUBSCRIPT_PROTOCOL` | python | `x[k]` runs `__getitem__` (and `x[k] = v` / `del x[k]` the setter and deleter) of the receiver's class. No written call; the site is the subscript expression. Its own kind so it is never counted as a written call. |

graph/bundle/schema.ts

@@ -667,6 +667,7 @@ export const VOCAB: readonly VocabSpec[] = [
   { table: 'call_edges', column: 'kind', value: 'DECORATOR_*', languages: P, meaning: 'Applying an unparenthesised decorator; the suffix is the parser\'s decorator kind: BARE, ATTRIBUTE, SUBSCRIPT, EXPRESSION (and CALL/ATTRIBUTE_CALL when the factory expression is not itself a call site). The site is the decorator hash.' },
   { table: 'call_edges', column: 'kind', value: 'METACLASS_CREATION', languages: P, meaning: 'A class statement invokes its metaclass\'s `__new__` / `__init__` at import time, whether the metaclass is written on the statement (`class X(metaclass=M)`) or inherited from a base, and the nearest base\'s `__init_subclass__`. No written call; the site is the class\'s type hash.' },
   { table: 'call_edges', column: 'kind', value: 'PROPERTY_READ', languages: P, meaning: 'Reading `obj.attr` where `attr` is a `@property` runs the getter; reading `Cls.attr` where the METACLASS defines `attr` as a property runs that getter. No written call; the site is the attribute-access expression.' },
+  { table: 'call_edges', column: 'kind', value: 'PROPERTY_WRITE', languages: P, meaning: 'Assigning `obj.attr = v` where `attr` is a `@property` with a setter runs the setter; `del obj.attr` runs its deleter. No written call; the site is the attribute-access expression.' },
   { table: 'call_edges', column: 'kind', value: 'CONTEXT_MANAGER', languages: P, meaning: '`with expr:` runs `__enter__` / `__exit__` (or the async pair). No written call; the site is the context-manager expression.' },
   { table: 'call_edges', column: 'kind', value: 'ITERATION_PROTOCOL', languages: P, meaning: '`for x in expr:` (and comprehensions) runs `__iter__` / `__next__` (or the async pair). No written call; the site is the iterated expression.' },
   { table: 'call_edges', column: 'kind', value: 'SUBSCRIPT_PROTOCOL', languages: P, meaning: '`x[k]` runs `__getitem__` (and `x[k] = v` / `del x[k]` the setter and deleter) of the receiver\'s class. No written call; the site is the subscript expression. Its own kind so it is never counted as a written call.' },
@@ -748,7 +749,7 @@ export const NOTES: readonly NoteSpec[] = [
   { language: 'typescript', table: 'overrides', note: 'EMPTY — this table is Java-shaped. The TypeScript dispatch envelope is in dispatch_candidates, with basis `nominal` or `structural`.' },
   { language: 'typescript', table: 'type_instantiated', note: 'Every row has how = `new`. Not restricted to client provenance: a type the library constructs is still a type that exists at run time, and dropping it would narrow the envelope unsoundly.' },
   { language: 'typescript', table: 'call_sites', note: 'PROPERTY_READ and PROPERTY_WRITE rows are accessor invocations with no written call: the site is the property-access expression that runs the getter or setter, positioned from the expressions table, and callee_name is NULL because nothing was written; the accessor\'s name is on the callee\'s methods row. Filter them out with kind NOT IN (…) when counting calls.' },
-  { language: 'python', table: 'call_sites', note: 'PROPERTY_READ, CONTEXT_MANAGER, ITERATION_PROTOCOL, METACLASS_CREATION and DYNAMIC_CALL rows are protocol or indirect edges with no written call: their site is the expression that triggers them, and callee_name is always NULL because nothing was written. SUBSCRIPT_CALL is NULL only when the subscript is not a written name (measured 206 of 337 rows on a Python subject). Filter them out with kind NOT IN (…) when counting calls.' },
+  { language: 'python', table: 'call_sites', note: 'PROPERTY_READ, PROPERTY_WRITE, CONTEXT_MANAGER, ITERATION_PROTOCOL, METACLASS_CREATION and DYNAMIC_CALL rows are protocol or indirect edges with no written call: their site is the expression that triggers them, and callee_name is always NULL because nothing was written. SUBSCRIPT_CALL is NULL only when the subscript is not a written name (measured 206 of 337 rows on a Python subject). Filter them out with kind NOT IN (…) when counting calls.' },
   { language: 'python', table: 'call_sites', note: 'The id is an EXPRESSION hash for a written call; a DECORATOR hash (PY_DECORATOR_…) for DECORATOR_APPLICATION and DECORATOR_* sites, positioned at the decorator line; and the class\'s TYPE hash for METACLASS_CREATION, positioned at the class declaration.' },
   { language: 'python', table: 'call_edges', note: 'A `boundary_lib` edge may point at a builtin (callee_provenance builtin, callee_label `builtin:NAME`) or at an unstaged import path (callee_provenance external) — neither has a methods row.' },
   { language: 'java', table: 'call_edges', note: 'A `boundary_lib` edge with callee_provenance external names a method of an ancestor type no staged IR declares (callee_label `external:<type>.<name>`, no methods row). A site whose receiver is declared as such a type is multi_inferred even with one client override: the platform method itself, and the platform\'s own subclasses, are the other possible targets. Stage the library to replace the label with the real method.' },

graph/pipeline/run-souffle.sh

@@ -201,6 +201,7 @@ write_program(){
   _SU=()
   t+="#include \"$LANG_ARG/souffle/decls_base.dl\"$nl#include \"$LANG_ARG/souffle/decls_all.dl\"$nl"
   map_rels_su "$TPL/client-ir.map" || return 1
+  if [ -f "$TPL/client-extra.map" ]; then map_rels_su "$TPL/client-extra.map" || return 1; fi
   map_rels_su "$TPL/lib.map" sig || return 1
   for r in $LIB_BODY; do _SU+=("$r"); done
   _SU+=(jdk_max_depth lib_max_depth taint_gating dispatch_cap dispatch_closed_world)
@@ -254,7 +255,8 @@ write_program(){
 # lost on either side shows up as a difference. Every .input and .output line must be exactly
 # the expected one, each exactly once, with nothing extra. Exit status only: no pipe to lose.
 verify_program(){
-  awk -v cmap="$TPL/client-ir.map" -v lmap="$TPL/lib.map" -v man="$DL/export_manifest.tsv" \
+  local xmap=""; [ -f "$TPL/client-extra.map" ] && xmap="$TPL/client-extra.map"
+  awk -v cmap="$TPL/client-ir.map" -v xmap="$xmap" -v lmap="$TPL/lib.map" -v man="$DL/export_manifest.tsv" \
       -v libsig=" $LIB_SIG " -v extra="$LIB_BODY jdk_max_depth lib_max_depth taint_gating dispatch_cap dispatch_closed_world" '
     function want(line) { if (!(line in need)) { need[line] = 1; n++ } }
     function inp(r) { want(".input " r "(IO=file, filename=\"" r ".facts\", delimiter=\"\\t\", rfc4180=true)") }
@@ -269,7 +271,7 @@ verify_program(){
       close(path)
     }
     BEGIN {
-      maprels(cmap, 0); maprels(lmap, 1)
+      maprels(cmap, 0); maprels(lmap, 1); if (xmap != "") maprels(xmap, 0)
       m = split(extra, e, " "); for (i = 1; i <= m; i++) if (e[i] != "") inp(e[i])
       while ((rc = (getline l < man)) > 0) {
         sub(/^\t+/, "", l); k = index(l, "\t"); p = l; f = ""
@@ -813,7 +815,7 @@ while IFS=$'\t' read -r rel csv; do
   if [ -f "$CLIENT/$csv.csv" ]; then awk -F'\t' 'NR==1{n=NF; next} NF==n{print; next} {bad++} END{if(bad>0) printf "  ! dropped %d malformed row(s) from %s\n", bad, FILENAME > "/dev/stderr"}'  "$CLIENT/$csv.csv" > "$FACTS/$rel.facts"
   else : > "$FACTS/$rel.facts"; fi
   CLIENT_INPUTS="$CLIENT_INPUTS$rel"$'\n'
-done < <(read_map "$TPL/client-ir.map")
+done < <(read_map "$TPL/client-ir.map"; if [ -f "$TPL/client-extra.map" ]; then read_map "$TPL/client-extra.map"; fi)
 
 # --- LIB: signature relations, concatenated across every module of every root ---
 # CACHED. This concatenation reads the ENTIRE library IR (the JDK alone is 2.0 GB in,

graph/python/engine/call-edge-generation/call_chain.dl

@@ -156,6 +156,31 @@ method_returns_method("client", m, r) :-
     expr_call_candidate(site, g),
     method_returns_method("client", g, r).
 
+// ── a call that hands back one of its arguments (py_returns_arg, resolution/builtins.dl) ──
+// Matched through the import that binds the callee, never by the bare name:
+// `functools.update_wrapper(...)` / `t.cast(...)` where the receiver is the name an `import`
+// bound, or `update_wrapper(...)` bound by `from functools import update_wrapper`. A local
+// function that happens to share the name does neither.
+py_passthrough_arg(site, pos) :-
+    call_name(site, fn), call_receiver_object(site, obj),
+    expr_binding("client", rb, ctx, obj), ctx != "STORE", binding_lookup("client", rb, rb2),
+    import_binding("client", rb2, i), import_decl("client", k, mod, _, i),
+    (k = "MODULE_IMPORT" ; k = "MODULE_IMPORT_ALIAS"),   // `import typing as t` is the alias kind
+    py_returns_arg(path, pos), cat(mod, cat(".", fn)) = path.
+py_passthrough_arg(site, pos) :-
+    call_callee_is_value(site), call_callee_expr(site, callee),
+    expr_binding("client", rb, ctx, callee), ctx != "STORE", binding_lookup("client", rb, rb2),
+    import_binding("client", rb2, i), import_decl("client", _, path, _, i),
+    py_returns_arg(path, pos).
+// the expression a value really is, through any number of such calls
+py_passthrough_root(e, e) :- method_return_value_expr("client", _, e).
+py_passthrough_root(e, x) :-
+    py_passthrough_root(e, c), call_of_expr(c, site), py_passthrough_arg(site, pos),
+    call_arg(site, pos, x).
+method_returns_method("client", m, r) :-
+    method_return_value_expr("client", m, e), py_passthrough_root(e, x), x != e,
+    expr_names_method("client", x, r).
+
 // ── decorator_hits_lib(SiteKey, LibMethodHash) ───────────────────────────────
 // `@abstractmethod` names `abc.abstractmethod`, which HAS Python source and is in the
 // staged stdlib IR — it is a library boundary, not a blind spot. A BARE decorator has no
@@ -320,6 +345,25 @@ iter_protocol_edge(src, caller, m) :-
     expr_type("client", src, t), iter_protocol_target(t, m),
     expr_ultimate_method("client", src, caller).
 
+// ── property_write_edge(WriteExprHash, CallerMethodHash, AccessorMethodHash) ──
+// The store and delete halves of the property protocol: `obj.x = v` calls x's setter,
+// `del obj.x` its deleter (type_property_accessor, resolution/attribute-lookup.dl). Same
+// shape as a read, keyed on the access's own name context.
+property_write_edge(e, caller, m) :-
+    expr_node("client", "ATTRIBUTE_ACCESS", _, n, e),
+    expr_name_context("client", "STORE", e),
+    expr_parent("client", e, "ATTRIBUTE_OBJECT", _, obj),
+    expr_type("client", obj, t),
+    type_property_accessor("client", t, n, "PROPERTY_SETTER", m),
+    expr_ultimate_method("client", e, caller).
+property_write_edge(e, caller, m) :-
+    expr_node("client", "ATTRIBUTE_ACCESS", _, n, e),
+    expr_name_context("client", "DEL", e),
+    expr_parent("client", e, "ATTRIBUTE_OBJECT", _, obj),
+    expr_type("client", obj, t),
+    type_property_accessor("client", t, n, "PROPERTY_DELETER", m),
+    expr_ultimate_method("client", e, caller).
+
 // ── property_read_edge(ReadExprHash, CallerMethodHash, GetterMethodHash) ─────
 property_read_edge(e, caller, getter) :-
     expr_node("client", "ATTRIBUTE_ACCESS", _, n, e),
@@ -668,6 +712,7 @@ subscript_protocol_edge(sub, caller, m) :-
 // from the graph rather than being re-tiered. A dropped edge is worse than a mislabelled
 // one, and the golden caught it.
 protocol_edge(e, d, caller, m) :- property_read_edge(e, caller, m), method_decl(_, d, _, _, _, m).
+protocol_edge(e, d, caller, m) :- property_write_edge(e, caller, m), method_decl(_, d, _, _, _, m).
 protocol_edge(e, d, caller, m) :- with_protocol_edge(e, caller, m), method_decl(_, d, _, _, _, m).
 protocol_edge(e, d, caller, m) :- iter_protocol_edge(e, caller, m), method_decl(_, d, _, _, _, m).
 protocol_edge(e, d, caller, m) :- subscript_protocol_edge(e, caller, m), method_decl(_, d, _, _, _, m).
@@ -731,6 +776,12 @@ call_chain_edge(e, caller, "-", getter, "client", cls, "PROPERTY_READ") :-
     property_read_edge(e, caller, getter), method_decl(_, d, _, _, _, getter),
     protocol_edge_class(e, d, cls).
 
+// PROPERTY WRITE — `obj.x = v` runs x's setter and `del obj.x` its deleter. Likewise no
+// call site, and its own kind so it is never counted as a written call.
+call_chain_edge(e, caller, "-", m, "client", cls, "PROPERTY_WRITE") :-
+    property_write_edge(e, caller, m), method_decl(_, d, _, _, _, m),
+    protocol_edge_class(e, d, cls).
+
 // CONTEXT MANAGER — the same shape: an edge with no call site, its own kind so it can
 // never be mistaken for a written call.
 call_chain_edge(cm, caller, "-", m, "client", cls, "CONTEXT_MANAGER") :-

graph/python/engine/expression-resolution/expr-type.dl

@@ -585,6 +585,15 @@ binding_declared_nominal(p, bind, d) :-
     binding_declared_ref(p, bind, r),
     type_ref_resolved(p, d, r),
     !type_is_structural(p, d).
+// `x: Optional[Order] = ...` / `x: Order | None = ...` IS an Order (or None) — the rule the
+// parameter, return and field annotations already have (resolution/annotations.dl,
+// "Optional[X] IS X"), which the local annotation alone was missing: a local written
+// with the commonest modern spelling stayed untyped while the same annotation on a
+// parameter resolved.
+binding_declared_nominal(p, bind, t) :-
+    type_ref_owner(p, bind, "BINDING", r),
+    type_ref(p, k, "VARIABLE_ANNOTATION", _, _, r), annotation_optional_kind(k),
+    type_ref_element(p, r, t), !type_is_structural(p, t).
 // The FK is resolved on only 14 of 479 variable annotations, so the name-based lookup
 // beside it is what carries this -- the same two clauses the parameter path uses, with
 // the same kind restriction. A SUBSCRIPT or a UNION names no single type and is served by